Industrial adoption of machine learning techniques for early identification of invalid bug reports
Laiq, Muhammad ; bin Ali, Nauman ; Börstler, Jürgen and Engström, Emelie LU
(2024)
In Empirical Software Engineering
29(5).
- Abstract
- Despite the accuracy of machine learning (ML) techniques in predicting invalid bug reports, as shown in earlier research, and the importance of early identification of invalid bug reports in software maintenance, the adoption of ML techniques for this task in industrial practice is yet to be investigated. In this study, we used a technology transfer model to guide the adoption of an ML technique at a company for the early identification of invalid bug reports. In the process, we also identify necessary conditions for adopting such techniques in practice. We followed a case study research approach with various design and analysis iterations for technology transfer activities. We collected data from bug repositories, through focus groups, a... (More)
- Despite the accuracy of machine learning (ML) techniques in predicting invalid bug reports, as shown in earlier research, and the importance of early identification of invalid bug reports in software maintenance, the adoption of ML techniques for this task in industrial practice is yet to be investigated. In this study, we used a technology transfer model to guide the adoption of an ML technique at a company for the early identification of invalid bug reports. In the process, we also identify necessary conditions for adopting such techniques in practice. We followed a case study research approach with various design and analysis iterations for technology transfer activities. We collected data from bug repositories, through focus groups, a questionnaire, and a presentation and feedback session with an expert. As expected, we found that an ML technique can identify invalid bug reports with acceptable accuracy at an early stage. However, the technique’s accuracy drops over time in its operational use due to changes in the product, the used technologies, or the development organization. Such changes may require retraining the ML model. During validation, practitioners highlighted the need to understand the ML technique’s predictions to trust the predictions. We found that a visual (using a state-of-the-art ML interpretation framework) and descriptive explanation of the prediction increases the trustability of the technique compared to just presenting the results of the validity predictions. We conclude that trustability, integration with the existing toolchain, and maintaining the techniques’ accuracy over time are critical for increasing the likelihood of adoption. (Less)
- Abstract (Swedish)
- Despite the accuracy of machine learning (ML) techniques in predict-
ing invalid bug reports, as shown in earlier research, and the importance of early
identification of invalid bug reports in software maintenance, the adoption of ML
techniques for this task in industrial practice is yet to be investigated. In this study,
we used a technology transfer model to guide the adoption of an ML technique at a
company for the early identification of invalid bug reports. In the process, we also
identify necessary conditions for adopting such techniques in practice. We followed
a case study research approach with various design and analysis iterations for tech-
nology transfer activities. We collected data from bug... (More) - Despite the accuracy of machine learning (ML) techniques in predict-
ing invalid bug reports, as shown in earlier research, and the importance of early
identification of invalid bug reports in software maintenance, the adoption of ML
techniques for this task in industrial practice is yet to be investigated. In this study,
we used a technology transfer model to guide the adoption of an ML technique at a
company for the early identification of invalid bug reports. In the process, we also
identify necessary conditions for adopting such techniques in practice. We followed
a case study research approach with various design and analysis iterations for tech-
nology transfer activities. We collected data from bug repositories, through focus
groups, a questionnaire, and a presentation and feedback session with an expert.
As expected, we found that an ML technique can identify invalid bug reports with
acceptable accuracy at an early stage. However, the technique’s accuracy drops
over time in its operational use due to changes in the product, the used technolo-
gies, or the development organization. Such changes may require retraining the
ML model. During validation, practitioners highlighted the need to understand
the ML technique’s predictions to trust the predictions. We found that a visual
(using a state-of-the-art ML interpretation framework) and descriptive explana-
tion of the prediction increases the trustability of the technique compared to just
presenting the results of the validity predictions. We conclude that trustability,
integration with the existing toolchain, and maintaining the techniques’ accuracy
over time are critical for increasing the likelihood of adoption. (Less)
Please use this url to cite or link to this publication:
https://lup.lub.lu.se/record/4ff7db63-79f9-4f62-a34e-269b6a3648ce
- author
- Laiq, Muhammad
; bin Ali, Nauman
; Börstler, Jürgen
and Engström, Emelie
LU
- organization
- publishing date
- 2024-09-05
- type
- Contribution to journal
- publication status
- published
- subject
- in
- Empirical Software Engineering
- volume
- 29
- issue
- 5
- article number
- 130
- publisher
- Springer
- external identifiers
-
- scopus:85200034314
- ISSN
- 1573-7616
- DOI
- 10.1007/s10664-024-10502-3
- language
- English
- LU publication?
- yes
- id
- 4ff7db63-79f9-4f62-a34e-269b6a3648ce
- date added to LUP
- 2024-05-21 09:28:50
- date last changed
- 2025-04-04 14:23:26
@article{4ff7db63-79f9-4f62-a34e-269b6a3648ce,
abstract = {{Despite the accuracy of machine learning (ML) techniques in predicting invalid bug reports, as shown in earlier research, and the importance of early identification of invalid bug reports in software maintenance, the adoption of ML techniques for this task in industrial practice is yet to be investigated. In this study, we used a technology transfer model to guide the adoption of an ML technique at a company for the early identification of invalid bug reports. In the process, we also identify necessary conditions for adopting such techniques in practice. We followed a case study research approach with various design and analysis iterations for technology transfer activities. We collected data from bug repositories, through focus groups, a questionnaire, and a presentation and feedback session with an expert. As expected, we found that an ML technique can identify invalid bug reports with acceptable accuracy at an early stage. However, the technique’s accuracy drops over time in its operational use due to changes in the product, the used technologies, or the development organization. Such changes may require retraining the ML model. During validation, practitioners highlighted the need to understand the ML technique’s predictions to trust the predictions. We found that a visual (using a state-of-the-art ML interpretation framework) and descriptive explanation of the prediction increases the trustability of the technique compared to just presenting the results of the validity predictions. We conclude that trustability, integration with the existing toolchain, and maintaining the techniques’ accuracy over time are critical for increasing the likelihood of adoption.}},
author = {{Laiq, Muhammad and bin Ali, Nauman and Börstler, Jürgen and Engström, Emelie}},
issn = {{1573-7616}},
language = {{eng}},
month = {{09}},
number = {{5}},
publisher = {{Springer}},
series = {{Empirical Software Engineering}},
title = {{Industrial adoption of machine learning techniques for early identification of invalid bug reports}},
url = {{http://dx.doi.org/10.1007/s10664-024-10502-3}},
doi = {{10.1007/s10664-024-10502-3}},
volume = {{29}},
year = {{2024}},
}