Controlled experimentation in continuous experimentation : Knowledge and challenges
Auer, Florian ; Ros, Rasmus LU ; Kaltenbrunner, Lukas ; Runeson, Per LU
and Felderer, Michael
(2021)
In Information and Software Technology
134.
- Abstract
Context: Continuous experimentation and A/B testing is an established industry practice that has been researched for more than 10 years. Our aim is to synthesize the conducted research. Objective: We wanted to find the core constituents of a framework for continuous experimentation and the solutions that are applied within the field. Finally, we were interested in the challenges and benefits reported of continuous experimentation. Methods: We applied forward snowballing on a known set of papers and identified a total of 128 relevant papers. Based on this set of papers we performed two qualitative narrative syntheses and a thematic synthesis to answer the research questions. Results: The framework constituents for continuous... (More)
Context: Continuous experimentation and A/B testing is an established industry practice that has been researched for more than 10 years. Our aim is to synthesize the conducted research. Objective: We wanted to find the core constituents of a framework for continuous experimentation and the solutions that are applied within the field. Finally, we were interested in the challenges and benefits reported of continuous experimentation. Methods: We applied forward snowballing on a known set of papers and identified a total of 128 relevant papers. Based on this set of papers we performed two qualitative narrative syntheses and a thematic synthesis to answer the research questions. Results: The framework constituents for continuous experimentation include experimentation processes as well as supportive technical and organizational infrastructure. The solutions found in the literature were synthesized to nine themes, e.g. experiment design, automated experiments, or metric specification. Concerning the challenges of continuous experimentation, the analysis identified cultural, organizational, business, technical, statistical, ethical, and domain-specific challenges. Further, the study concludes that the benefits of experimentation are mostly implicit in the studies. Conclusion: The research on continuous experimentation has yielded a large body of knowledge on experimentation. The synthesis of published research presented within include recommended infrastructure and experimentation process models, guidelines to mitigate the identified challenges, and what problems the various published solutions solve.
(Less)
- author
- Auer, Florian
; Ros, Rasmus
LU
; Kaltenbrunner, Lukas
; Runeson, Per
LU
and Felderer, Michael
- organization
- publishing date
- 2021
- type
- Contribution to journal
- publication status
- published
- subject
- keywords
- A/B testing, Continuous experimentation, Online controlled experiments, Systematic literature review
- in
- Information and Software Technology
- volume
- 134
- article number
- 106551
- publisher
- Elsevier
- external identifiers
-
- scopus:85101342705
- ISSN
- 0950-5849
- DOI
- 10.1016/j.infsof.2021.106551
- project
- WASP Software Engineering Cluster
- Continuous Experimentation and Optimization
- language
- English
- LU publication?
- yes
- id
- 2bd2b828-9ad0-4cb9-9dae-aec2a7ea7cce
- date added to LUP
- 2021-03-08 11:28:22
- date last changed
- 2024-05-02 04:26:26
@article{2bd2b828-9ad0-4cb9-9dae-aec2a7ea7cce,
abstract = {{<p>Context: Continuous experimentation and A/B testing is an established industry practice that has been researched for more than 10 years. Our aim is to synthesize the conducted research. Objective: We wanted to find the core constituents of a framework for continuous experimentation and the solutions that are applied within the field. Finally, we were interested in the challenges and benefits reported of continuous experimentation. Methods: We applied forward snowballing on a known set of papers and identified a total of 128 relevant papers. Based on this set of papers we performed two qualitative narrative syntheses and a thematic synthesis to answer the research questions. Results: The framework constituents for continuous experimentation include experimentation processes as well as supportive technical and organizational infrastructure. The solutions found in the literature were synthesized to nine themes, e.g. experiment design, automated experiments, or metric specification. Concerning the challenges of continuous experimentation, the analysis identified cultural, organizational, business, technical, statistical, ethical, and domain-specific challenges. Further, the study concludes that the benefits of experimentation are mostly implicit in the studies. Conclusion: The research on continuous experimentation has yielded a large body of knowledge on experimentation. The synthesis of published research presented within include recommended infrastructure and experimentation process models, guidelines to mitigate the identified challenges, and what problems the various published solutions solve.</p>}},
author = {{Auer, Florian and Ros, Rasmus and Kaltenbrunner, Lukas and Runeson, Per and Felderer, Michael}},
issn = {{0950-5849}},
keywords = {{A/B testing; Continuous experimentation; Online controlled experiments; Systematic literature review}},
language = {{eng}},
publisher = {{Elsevier}},
series = {{Information and Software Technology}},
title = {{Controlled experimentation in continuous experimentation : Knowledge and challenges}},
url = {{http://dx.doi.org/10.1016/j.infsof.2021.106551}},
doi = {{10.1016/j.infsof.2021.106551}},
volume = {{134}},
year = {{2021}},
}