Continuous Experimentation and A/B Testing: A Mapping Study
Ros, Rasmus LU and Runeson, Per LU
(2018)
RCoSE 2018 : 4th International Workshop on Rapid Continuous Software Engineering
p.35-41
- Abstract
- Background. Continuous experimentation (CE) has recently emerged as an established industry practice and as a research subject. Our aim is to study the application of CE and A/B testing in various industrial contexts. Objective. We wanted to investigate whether CE is used in different sectors of industry, by how it is reported in academic studies. We also wanted to explore the main topics researched to give an overview of the subject and discuss future research directions. Method. We performed a systematic mapping study of the published literature and included 62 papers, using a combination of database search and snowballing. Results. Most reported software experiments are done online and with software delivered as a service, although... (More)
- Background. Continuous experimentation (CE) has recently emerged as an established industry practice and as a research subject. Our aim is to study the application of CE and A/B testing in various industrial contexts. Objective. We wanted to investigate whether CE is used in different sectors of industry, by how it is reported in academic studies. We also wanted to explore the main topics researched to give an overview of the subject and discuss future research directions. Method. We performed a systematic mapping study of the published literature and included 62 papers, using a combination of database search and snowballing. Results. Most reported software experiments are done online and with software delivered as a service, although varied exemptions exist for e.g., financial software and games. The most frequently researched topics are challenges to conduct experiments and statistical methods for software experiments. Conclusions. The software engineering research on CE is still in its infancy. There are future research opportunities in evaluation research of technical topics and investigations of ethical experimentation. We conclude that the included studies show that A/B testing is applicable to varied software and organisations. (Less)
Please use this url to cite or link to this publication:
https://lup.lub.lu.se/record/f88910dc-16fd-433e-ada2-7482f09d33d0
- author
- Ros, Rasmus
LU
and Runeson, Per
LU
- organization
- publishing date
- 2018-03-16
- type
- Chapter in Book/Report/Conference proceeding
- publication status
- published
- subject
- host publication
- RCoSE'18 Proceedings of the 4th International Workshop on Rapid Continuous Software Engineering
- pages
- 7 pages
- publisher
- Association for Computing Machinery (ACM)
- conference name
- RCoSE 2018 : 4th International Workshop on Rapid Continuous Software Engineering
- conference location
- Gothenburg, Sweden
- conference dates
- 2018-05-28 - 2018-05-28
- external identifiers
-
- scopus:85051206831
- DOI
- 10.1145/3194760.3194766
- project
- Continuous Experimentation and Optimization
- language
- English
- LU publication?
- yes
- id
- f88910dc-16fd-433e-ada2-7482f09d33d0
- date added to LUP
- 2018-03-15 13:19:37
- date last changed
- 2022-05-03 01:53:31
@inproceedings{f88910dc-16fd-433e-ada2-7482f09d33d0,
abstract = {{Background. Continuous experimentation (CE) has recently emerged as an established industry practice and as a research subject. Our aim is to study the application of CE and A/B testing in various industrial contexts. Objective. We wanted to investigate whether CE is used in different sectors of industry, by how it is reported in academic studies. We also wanted to explore the main topics researched to give an overview of the subject and discuss future research directions. Method. We performed a systematic mapping study of the published literature and included 62 papers, using a combination of database search and snowballing. Results. Most reported software experiments are done online and with software delivered as a service, although varied exemptions exist for e.g., financial software and games. The most frequently researched topics are challenges to conduct experiments and statistical methods for software experiments. Conclusions. The software engineering research on CE is still in its infancy. There are future research opportunities in evaluation research of technical topics and investigations of ethical experimentation. We conclude that the included studies show that A/B testing is applicable to varied software and organisations.}},
author = {{Ros, Rasmus and Runeson, Per}},
booktitle = {{RCoSE'18 Proceedings of the 4th International Workshop on Rapid Continuous Software Engineering}},
language = {{eng}},
month = {{03}},
pages = {{35--41}},
publisher = {{Association for Computing Machinery (ACM)}},
title = {{Continuous Experimentation and A/B Testing: A Mapping Study}},
url = {{http://dx.doi.org/10.1145/3194760.3194766}},
doi = {{10.1145/3194760.3194766}},
year = {{2018}},
}