Skip to main content

Lund University Publications

LUND UNIVERSITY LIBRARIES

Threats to validity in software engineering research: A critical reflection

Verdecchia, Roberto ; Engström, Emelie LU orcid ; Lago, Patricia ; Runeson, Per LU orcid and Song, Qunying LU orcid (2023) In Information and Software Technology
Abstract
Context: In the contemporary body of software engineering literature, some recurrent shortcomings characterize how threats to validity (TTV) are considered in studies. Objective: With this position paper, we aim to open a discourse on the current use of TTV sections. The goal of our position is to jointly reflect and systematically improve how we, as a research community, consider TTV in our studies. Methods: Based on our personal experience as researchers, authors, reviewers, and editors, we critically reflect on the treatment of TTV in current empirical software engineering literature. Results: We discuss the key shortcomings of TTV consideration, including the failure to acknowledge different types of validity categorizations and the... (More)
Context: In the contemporary body of software engineering literature, some recurrent shortcomings characterize how threats to validity (TTV) are considered in studies. Objective: With this position paper, we aim to open a discourse on the current use of TTV sections. The goal of our position is to jointly reflect and systematically improve how we, as a research community, consider TTV in our studies. Methods: Based on our personal experience as researchers, authors, reviewers, and editors, we critically reflect on the treatment of TTV in current empirical software engineering literature. Results: We discuss the key shortcomings of TTV consideration, including the failure to acknowledge different types of validity categorizations and the tendency to treat threats just as an afterthought. For each identified problem, we propose a vision for an improved state, intending to catalyze thoughtful engagement and improvements the way our community addresses TTV. Conclusion: We posit there is an urgent need to reconsider how we approach, document, and evaluate TTV in software engineering research. (Less)
Please use this url to cite or link to this publication:
author
; ; ; and
organization
publishing date
type
Contribution to journal
publication status
published
subject
keywords
Threats to validity, Software engineering, Empirical research
in
Information and Software Technology
article number
107329
pages
4 pages
publisher
Elsevier
external identifiers
  • scopus:85170643531
ISSN
0950-5849
DOI
10.1016/j.infsof.2023.107329
project
Software testing of autonomous systems
language
English
LU publication?
yes
id
ffc559a2-ea52-4bac-a52c-d007ab2363de
date added to LUP
2023-09-20 14:21:25
date last changed
2024-06-13 15:00:02
@article{ffc559a2-ea52-4bac-a52c-d007ab2363de,
  abstract     = {{Context: In the contemporary body of software engineering literature, some recurrent shortcomings characterize how threats to validity (TTV) are considered in studies. Objective: With this position paper, we aim to open a discourse on the current use of TTV sections. The goal of our position is to jointly reflect and systematically improve how we, as a research community, consider TTV in our studies. Methods: Based on our personal experience as researchers, authors, reviewers, and editors, we critically reflect on the treatment of TTV in current empirical software engineering literature. Results: We discuss the key shortcomings of TTV consideration, including the failure to acknowledge different types of validity categorizations and the tendency to treat threats just as an afterthought. For each identified problem, we propose a vision for an improved state, intending to catalyze thoughtful engagement and improvements the way our community addresses TTV. Conclusion: We posit there is an urgent need to reconsider how we approach, document, and evaluate TTV in software engineering research.}},
  author       = {{Verdecchia, Roberto and Engström, Emelie and Lago, Patricia and Runeson, Per and Song, Qunying}},
  issn         = {{0950-5849}},
  keywords     = {{Threats to validity; Software engineering; Empirical research}},
  language     = {{eng}},
  publisher    = {{Elsevier}},
  series       = {{Information and Software Technology}},
  title        = {{Threats to validity in software engineering research: A critical reflection}},
  url          = {{http://dx.doi.org/10.1016/j.infsof.2023.107329}},
  doi          = {{10.1016/j.infsof.2023.107329}},
  year         = {{2023}},
}