Using Permutation-Based Feature Importance for Improved Machine Learning Model Performance at Reduced Costs
Lopullinen julkaistu versio - 1.95 MB
https://creativecommons.org/licenses/by/4.0/
Pysyvä osoite
Kuvaus
© 2025 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
In Software Quality Assurance (SQA), predicting defect-prone software modules is essential for ensuring software reliability and consistency. This task is commonly achieved through Machine Learning (ML) techniques, but improving model performance typically incurs significant computational costs. These high computational costs and uncertain payoffs make most Software engineering researchers reluctant to optimize ML models. This creates a need for novel techniques that can achieve near-optimal performance of hyperparameter settings while maintaining the computational efficiency of default settings. To address this, we employed five ML models, Decision Tree, Ranger, Random Forest, Support Vector Machine, and k-nearest Neighbors, and optimized their parameters using the random search technique. Our experiments covered six diverse Software Fault Prediction (SFP) datasets, encompassing various software features, application domains, and defect patterns, to evaluate the approach’s generalizability and effectiveness. Moreover, the Permutation Feature Importance (PFI)-based model-agnostic method was employed to identify the top ten features most critical for model accuracy and efficiency. These selected features were used to retrain the ML models without hyperparameters (default settings) to determine whether similar performance could be achieved at low computational cost. The results show an average accuracy improvement of 77.39% and a 92.02% reduction in computational cost. The most important case attained a 99.25% accuracy improvement and a 96.77% cost reduction. Such results clearly show that PFI-based feature selection is capable of high performance at a fraction of computational cost, offering an efficient solution for software engineers to optimize ML models.
Emojulkaisu
ISBN
ISSN
2169-3536
Aihealue
Kausijulkaisu
IEEE access|13
OKM-julkaisutyyppi
A1 Alkuperäisartikkeli tieteellisessä aikakauslehdessä (vertaisarvioitu)
Avainsanat
Computational modelingFeature extractionAccuracyComputational efficiencyPredictive modelsOptimizationSupport vector machinesRandom forestsRadio frequencyDecision treesModel-agnostic techniquespermutation feature importance (PFI)software fault prediction (SFP)predictive accuracymachine learning (ML)computational costdefault settingshyperparameter