Advanced

Trends in the quality of human-intensive software engineering experiments – A quasi-experiment.

Kitchenham, Barbara; Sjøberg, Dag; Dybå, Tore; Brereton, Pearl; Budgen, David; Höst, Martin LU and Runeson, Per LU (2013) In IEEE Transactions on Software Engineering 39(7). p.1002-1017
Abstract
Context: Several text books and articles published between 2000 and 2002 have attempted to introduce experimental design and statistical methods to software engineers undertaking empirical studies. Objective: This paper investigates whether there has been an increase in the quality of human-centric experimental and quasi-experimental journal papers over the time period 1993 to 2010. Method: 70 experimental and quasi-experimental papers published in four general software engineering journals in the years 1992-2002 and 2006-2010 were each assessed for quality by three empirical software engineering researchers using two quality assessment methods (a questionnaire based method and a subjective overall assessment). Regression analysis was used... (More)
Context: Several text books and articles published between 2000 and 2002 have attempted to introduce experimental design and statistical methods to software engineers undertaking empirical studies. Objective: This paper investigates whether there has been an increase in the quality of human-centric experimental and quasi-experimental journal papers over the time period 1993 to 2010. Method: 70 experimental and quasi-experimental papers published in four general software engineering journals in the years 1992-2002 and 2006-2010 were each assessed for quality by three empirical software engineering researchers using two quality assessment methods (a questionnaire based method and a subjective overall assessment). Regression analysis was used to assess the relationship between paper quality and the year of publication, publication date group (before 2003 and after 2005), source journal, average co-author experience, citation of statistical text books and articles, and paper length. The results were validated both by removing papers for which the quality score appeared unreliable and using an alternative quality measure. Results: Paper quality was significantly associated with year, citing general statistical texts and paper length (p<0.05). Paper length did not reach significance when quality was measured using an overall subjective assessment. Conclusions: The quality of experimental and quasi-experimental software engineering papers appears to have improved gradually since 1993. (Less)
Please use this url to cite or link to this publication:
author
organization
publishing date
type
Contribution to journal
publication status
published
subject
in
IEEE Transactions on Software Engineering
volume
39
issue
7
pages
1002 - 1017
publisher
IEEE--Institute of Electrical and Electronics Engineers Inc.
external identifiers
  • wos:000321125200007
  • scopus:84880239200
ISSN
0098-5589
DOI
10.1109/TSE.2012.76
language
English
LU publication?
yes
id
fd1cf933-58a7-43fc-9329-2013cfa1aba2 (old id 3242798)
date added to LUP
2012-12-18 12:15:06
date last changed
2019-03-12 02:27:39
@article{fd1cf933-58a7-43fc-9329-2013cfa1aba2,
  abstract     = {Context: Several text books and articles published between 2000 and 2002 have attempted to introduce experimental design and statistical methods to software engineers undertaking empirical studies. Objective: This paper investigates whether there has been an increase in the quality of human-centric experimental and quasi-experimental journal papers over the time period 1993 to 2010. Method: 70 experimental and quasi-experimental papers published in four general software engineering journals in the years 1992-2002 and 2006-2010 were each assessed for quality by three empirical software engineering researchers using two quality assessment methods (a questionnaire based method and a subjective overall assessment). Regression analysis was used to assess the relationship between paper quality and the year of publication, publication date group (before 2003 and after 2005), source journal, average co-author experience, citation of statistical text books and articles, and paper length. The results were validated both by removing papers for which the quality score appeared unreliable and using an alternative quality measure. Results: Paper quality was significantly associated with year, citing general statistical texts and paper length (p&lt;0.05). Paper length did not reach significance when quality was measured using an overall subjective assessment. Conclusions: The quality of experimental and quasi-experimental software engineering papers appears to have improved gradually since 1993.},
  author       = {Kitchenham, Barbara and Sjøberg, Dag and Dybå, Tore and Brereton, Pearl and Budgen, David and Höst, Martin and Runeson, Per},
  issn         = {0098-5589},
  language     = {eng},
  number       = {7},
  pages        = {1002--1017},
  publisher    = {IEEE--Institute of Electrical and Electronics Engineers Inc.},
  series       = {IEEE Transactions on Software Engineering},
  title        = {Trends in the quality of human-intensive software engineering experiments – A quasi-experiment.},
  url          = {http://dx.doi.org/10.1109/TSE.2012.76},
  volume       = {39},
  year         = {2013},
}