Skip to main content

Lund University Publications

LUND UNIVERSITY LIBRARIES

Threats to Validity in Software Engineering – hypocritical paper section or essential analysis?

Lago, Patricia ; Runeson, Per LU orcid ; Song, Qunying LU orcid and Verdecchia, Roberto (2024) 18th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM '24 p.314-324
Abstract
Background: In recent years, a discourse on how to systematically consider and report threats to validity started to gain momentum within the empirical software engineering community. Aims: With this study, we aim to systematically underpin the current state of threats to validity practices in software engineering research. Method: We conduct a literature review comprising 91 papers awarded with the ACM SIGSOFT Distinguished Paper Award at the ACM/IEEE International Conference on Software Engineering. Data is extracted and analyzed by considering six main facets of threats to validity, e.g., their explicit documentation, categorization, discussion of limitations, and trade-offs. Results: Results corroborate current critiques to the threats... (More)
Background: In recent years, a discourse on how to systematically consider and report threats to validity started to gain momentum within the empirical software engineering community. Aims: With this study, we aim to systematically underpin the current state of threats to validity practices in software engineering research. Method: We conduct a literature review comprising 91 papers awarded with the ACM SIGSOFT Distinguished Paper Award at the ACM/IEEE International Conference on Software Engineering. Data is extracted and analyzed by considering six main facets of threats to validity, e.g., their explicit documentation, categorization, discussion of limitations, and trade-offs. Results: Results corroborate current critiques to the threats management state of the art. Threats result to be seldom discussed in depth, and are mostly considered as an enforced afterthought rather than an active concern of the research design and execution. Conclusions: To improve the observed practice, we derived items to consider for researchers, reviewers and readers, and call for a community action to increase the understanding of knowledge creation in empirical software engineering research. (Less)
Please use this url to cite or link to this publication:
author
; ; and
organization
publishing date
type
Chapter in Book/Report/Conference proceeding
publication status
published
subject
keywords
Empirical studies, Limitations, Literature study, Threats to validity
host publication
Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement
pages
314 - 324
publisher
Association for Computing Machinery (ACM)
conference name
18th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM '24
conference location
Barcelona, Spain
conference dates
2024-10-24 - 2024-10-25
external identifiers
  • scopus:85210583700
ISBN
9798400710476
DOI
10.1145/3674805.3686691
project
WASP: Wallenberg AI, Autonomous Systems and Software Program at Lund University
language
English
LU publication?
yes
id
cdaa9e06-7308-42bc-aee4-6875a94f0023
date added to LUP
2024-10-25 11:30:22
date last changed
2025-04-04 14:42:06
@inproceedings{cdaa9e06-7308-42bc-aee4-6875a94f0023,
  abstract     = {{Background: In recent years, a discourse on how to systematically consider and report threats to validity started to gain momentum within the empirical software engineering community. Aims: With this study, we aim to systematically underpin the current state of threats to validity practices in software engineering research. Method: We conduct a literature review comprising 91 papers awarded with the ACM SIGSOFT Distinguished Paper Award at the ACM/IEEE International Conference on Software Engineering. Data is extracted and analyzed by considering six main facets of threats to validity, e.g., their explicit documentation, categorization, discussion of limitations, and trade-offs. Results: Results corroborate current critiques to the threats management state of the art. Threats result to be seldom discussed in depth, and are mostly considered as an enforced afterthought rather than an active concern of the research design and execution. Conclusions: To improve the observed practice, we derived items to consider for researchers, reviewers and readers, and call for a community action to increase the understanding of knowledge creation in empirical software engineering research.}},
  author       = {{Lago, Patricia and Runeson, Per and Song, Qunying and Verdecchia, Roberto}},
  booktitle    = {{Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement}},
  isbn         = {{9798400710476}},
  keywords     = {{Empirical studies; Limitations; Literature study; Threats to validity}},
  language     = {{eng}},
  month        = {{10}},
  pages        = {{314--324}},
  publisher    = {{Association for Computing Machinery (ACM)}},
  title        = {{Threats to Validity in Software Engineering – hypocritical paper section or essential analysis?}},
  url          = {{http://dx.doi.org/10.1145/3674805.3686691}},
  doi          = {{10.1145/3674805.3686691}},
  year         = {{2024}},
}