Advanced

Continuous Experimentation and A/B Testing: A Mapping Study

Ros, Rasmus LU and Runeson, Per LU (2018) RCoSE 2018 : 4th International Workshop on Rapid Continuous Software Engineering p.35-41
Abstract
Background. Continuous experimentation (CE) has recently emerged as an established industry practice and as a research subject. Our aim is to study the application of CE and A/B testing in various industrial contexts. Objective. We wanted to investigate whether CE is used in different sectors of industry, by how it is reported in academic studies. We also wanted to explore the main topics researched to give an overview of the subject and discuss future research directions. Method. We performed a systematic mapping study of the published literature and included 62 papers, using a combination of database search and snowballing. Results. Most reported software experiments are done online and with software delivered as a service, although... (More)
Background. Continuous experimentation (CE) has recently emerged as an established industry practice and as a research subject. Our aim is to study the application of CE and A/B testing in various industrial contexts. Objective. We wanted to investigate whether CE is used in different sectors of industry, by how it is reported in academic studies. We also wanted to explore the main topics researched to give an overview of the subject and discuss future research directions. Method. We performed a systematic mapping study of the published literature and included 62 papers, using a combination of database search and snowballing. Results. Most reported software experiments are done online and with software delivered as a service, although varied exemptions exist for e.g., financial software and games. The most frequently researched topics are challenges to conduct experiments and statistical methods for software experiments. Conclusions. The software engineering research on CE is still in its infancy. There are future research opportunities in evaluation research of technical topics and investigations of ethical experimentation. We conclude that the included studies show that A/B testing is applicable to varied software and organisations. (Less)
Please use this url to cite or link to this publication:
author
organization
publishing date
type
Chapter in Book/Report/Conference proceeding
publication status
published
subject
host publication
RCoSE'18 Proceedings of the 4th International Workshop on Rapid Continuous Software Engineering
pages
7 pages
publisher
ACM
conference name
RCoSE 2018 : 4th International Workshop on Rapid Continuous Software Engineering
conference location
Gothenburg, Sweden
conference dates
2018-05-28 - 2018-05-28
external identifiers
  • scopus:85051206831
DOI
10.1145/3194760.3194766
language
English
LU publication?
yes
id
f88910dc-16fd-433e-ada2-7482f09d33d0
date added to LUP
2018-03-15 13:19:37
date last changed
2018-12-09 04:37:47
@inproceedings{f88910dc-16fd-433e-ada2-7482f09d33d0,
  abstract     = {Background. Continuous experimentation (CE) has recently emerged as an established industry practice and as a research subject. Our aim is to study the application of CE and A/B testing in various industrial contexts. Objective. We wanted to investigate whether CE is used in different sectors of industry, by how it is reported in academic studies. We also wanted to explore the main topics researched to give an overview of the subject and discuss future research directions. Method. We performed a systematic mapping study of the published literature and included 62 papers, using a combination of database search and snowballing. Results. Most reported software experiments are done online and with software delivered as a service, although varied exemptions exist for e.g., financial software and games. The most frequently researched topics are challenges to conduct experiments and statistical methods for software experiments. Conclusions. The software engineering research on CE is still in its infancy. There are future research opportunities in evaluation research of technical topics and investigations of ethical experimentation. We conclude that the included studies show that A/B testing is applicable to varied software and organisations.},
  author       = {Ros, Rasmus and Runeson, Per},
  language     = {eng},
  location     = {Gothenburg, Sweden},
  month        = {03},
  pages        = {35--41},
  publisher    = {ACM},
  title        = {Continuous Experimentation and A/B Testing: A Mapping Study},
  url          = {http://dx.doi.org/10.1145/3194760.3194766},
  year         = {2018},
}