Advanced

On rapid releases and software testing: a case study and a semi-systematic literature review

Mäntylä, Mika V.; Adams, Bram; Khomh, Foutse and Engström, Emelie LU (2015) In Empirical Software Engineering 20(5). p.1384-1425
Abstract
Large open and closed source organizations like Google, Facebook and Mozilla are migrating their products towards rapid releases. While this allows faster time-to-market and user f eedback, it also implies less time for

testing and bug fixing. Since initial research results indeed show that rapid releases fix proportionally less reported bugs than traditional releases, this paper investigates the changes in sof tware testing effort after moving to rapid releases in the context of a case study on Mozilla Firef ox, and performs a semi-sy stematic literature rev iew. The case study analyzes the results of 312,502 execution runs of the 1,547 mostly manual system-level test cases of Mozilla Firef ox f rom 2006 to 2012 (5 major... (More)
Large open and closed source organizations like Google, Facebook and Mozilla are migrating their products towards rapid releases. While this allows faster time-to-market and user f eedback, it also implies less time for

testing and bug fixing. Since initial research results indeed show that rapid releases fix proportionally less reported bugs than traditional releases, this paper investigates the changes in sof tware testing effort after moving to rapid releases in the context of a case study on Mozilla Firef ox, and performs a semi-sy stematic literature rev iew. The case study analyzes the results of 312,502 execution runs of the 1,547 mostly manual system-level test cases of Mozilla Firef ox f rom 2006 to 2012 (5 major traditional and 9 major rapid releases), and triangulates our findings with a Mozilla QA engineer. We f ind that rapid releases hav e a narrower test scope that enables a deeper investigation of the f eatures and regressions with the highest risk. Furthermore, rapid releases make testing more continuous and have proportionally smaller spikes before the main release. However, rapid releases make it more difficult to build a large testing community , and they decrease test suite diversity and make testing more deadline oriented. In addition, our semi-sy stematic literature rev iew presents the benefits, problems and enablers of rapid releases from 24 papers found using systematic search queries and a similar amount of papers found through other means. The literature review shows that rapid releases are a prevalent industrial practice that are utilized even in some highly critical domains of software engineering, and that rapid releases originated from several software development methodologies such as agile, open source, lean and internet-speed software development. However, empirical studies proving evidence of the claimed advantages and disadvantages of rapid releases are scarce. (Less)
Please use this url to cite or link to this publication:
author
organization
publishing date
type
Contribution to journal
publication status
published
subject
in
Empirical Software Engineering
volume
20
issue
5
pages
1384 - 1425
publisher
Springer
external identifiers
  • wos:000360671500007
  • scopus:84940719377
ISSN
1573-7616
DOI
10.1007/s10664-014-9338-4
language
English
LU publication?
yes
id
cca60536-41ea-45d8-9964-4419ab0a03c9 (old id 7989322)
date added to LUP
2015-09-24 09:42:39
date last changed
2017-10-08 03:05:35
@article{cca60536-41ea-45d8-9964-4419ab0a03c9,
  abstract     = {Large open and closed source organizations like Google, Facebook and Mozilla are migrating their products towards rapid releases. While this allows faster time-to-market and user f eedback, it also implies less time for<br/><br>
testing and bug fixing. Since initial research results indeed show that rapid releases fix proportionally less reported bugs than traditional releases, this paper investigates the changes in sof tware testing effort after moving to rapid releases in the context of a case study on Mozilla Firef ox, and performs a semi-sy stematic literature rev iew. The case study analyzes the results of 312,502 execution runs of the 1,547 mostly manual system-level test cases of Mozilla Firef ox f rom 2006 to 2012 (5 major traditional and 9 major rapid releases), and triangulates our findings with a Mozilla QA engineer. We f ind that rapid releases hav e a narrower test scope that enables a deeper investigation of the f eatures and regressions with the highest risk. Furthermore, rapid releases make testing more continuous and have proportionally smaller spikes before the main release. However, rapid releases make it more difficult to build a large testing community , and they decrease test suite diversity and make testing more deadline oriented. In addition, our semi-sy stematic literature rev iew presents the benefits, problems and enablers of rapid releases from 24 papers found using systematic search queries and a similar amount of papers found through other means. The literature review shows that rapid releases are a prevalent industrial practice that are utilized even in some highly critical domains of software engineering, and that rapid releases originated from several software development methodologies such as agile, open source, lean and internet-speed software development. However, empirical studies proving evidence of the claimed advantages and disadvantages of rapid releases are scarce.},
  author       = {Mäntylä, Mika V. and Adams, Bram and Khomh, Foutse and Engström, Emelie},
  issn         = {1573-7616},
  language     = {eng},
  number       = {5},
  pages        = {1384--1425},
  publisher    = {Springer},
  series       = {Empirical Software Engineering},
  title        = {On rapid releases and software testing: a case study and a semi-systematic literature review},
  url          = {http://dx.doi.org/10.1007/s10664-014-9338-4},
  volume       = {20},
  year         = {2015},
}