A theory of factors affecting continuous experimentation (FACE)
Ros, Rasmus LU ; Bjarnason, Elizabeth LU
and Runeson, Per
LU
(2024)
In Empirical Software Engineering
29(1).
- Abstract
Context: Continuous experimentation (CE) is used by many companies with internet-facing products to improve their business models and software solutions based on user data. Some companies deliberately adopt a systematic experiment-driven approach to software development while some companies use CE in a more ad-hoc fashion. Objective: The goal of this study is to identify factors for success in CE that explain the variations in the utility and efficacy of CE between different companies. Method: We conducted a multi-case study of 12 companies involved with CE and performed 27 interviews with practitioners at these companies. Based on that empirical data, we then built a theory of factors at play in CE. Results: We introduce a theory of... (More)
Context: Continuous experimentation (CE) is used by many companies with internet-facing products to improve their business models and software solutions based on user data. Some companies deliberately adopt a systematic experiment-driven approach to software development while some companies use CE in a more ad-hoc fashion. Objective: The goal of this study is to identify factors for success in CE that explain the variations in the utility and efficacy of CE between different companies. Method: We conducted a multi-case study of 12 companies involved with CE and performed 27 interviews with practitioners at these companies. Based on that empirical data, we then built a theory of factors at play in CE. Results: We introduce a theory of Factors Affecting Continuous Experimentation (FACE). The theory includes three factors, namely 1) processes and infrastructure for CE, 2) the user problem complexity of the product offering, and 3) incentive structures for CE. The theory explains how these factors affect the effectiveness of CE and its ability to achieve problem-solution and product-market fit. Conclusions: Our theory may inspire practitioners to assess an organisation’s potential for adopting CE and to identify factors that pose challenges in gaining value from CE practices. Our results also provide a basis for defining practitioner guidelines and a starting point for further research on how contextual factors affect CE and how these may be mitigated.
(Less)
- author
- Ros, Rasmus
LU
; Bjarnason, Elizabeth
LU
and Runeson, Per
LU
- organization
- publishing date
- 2024-02
- type
- Contribution to journal
- publication status
- published
- subject
- keywords
- A/B testing, Continuous experimentation, Data-driven development, Empirical research, Multi-case study, Theory building
- in
- Empirical Software Engineering
- volume
- 29
- issue
- 1
- article number
- 21
- publisher
- Springer
- external identifiers
-
- scopus:85179585225
- ISSN
- 1382-3256
- DOI
- 10.1007/s10664-023-10358-z
- language
- English
- LU publication?
- yes
- additional info
- Funding Information: This work was funded by the Wallenberg Artificial Intelligence, Autonomous Systems and Software Program (WASP), which in turn is funded by Knut and Alice Wallenberg Foundation. The authors have no other financial or non-financial interests, or connection to the involved case companies, to disclose Publisher Copyright: © 2023, The Author(s).
- id
- de8df270-95bf-40e0-a208-0cbeca5228f6
- date added to LUP
- 2023-12-20 14:09:40
- date last changed
- 2023-12-20 18:51:27
@article{de8df270-95bf-40e0-a208-0cbeca5228f6,
abstract = {{<p>Context: Continuous experimentation (CE) is used by many companies with internet-facing products to improve their business models and software solutions based on user data. Some companies deliberately adopt a systematic experiment-driven approach to software development while some companies use CE in a more ad-hoc fashion. Objective: The goal of this study is to identify factors for success in CE that explain the variations in the utility and efficacy of CE between different companies. Method: We conducted a multi-case study of 12 companies involved with CE and performed 27 interviews with practitioners at these companies. Based on that empirical data, we then built a theory of factors at play in CE. Results: We introduce a theory of Factors Affecting Continuous Experimentation (FACE). The theory includes three factors, namely 1) processes and infrastructure for CE, 2) the user problem complexity of the product offering, and 3) incentive structures for CE. The theory explains how these factors affect the effectiveness of CE and its ability to achieve problem-solution and product-market fit. Conclusions: Our theory may inspire practitioners to assess an organisation’s potential for adopting CE and to identify factors that pose challenges in gaining value from CE practices. Our results also provide a basis for defining practitioner guidelines and a starting point for further research on how contextual factors affect CE and how these may be mitigated.</p>}},
author = {{Ros, Rasmus and Bjarnason, Elizabeth and Runeson, Per}},
issn = {{1382-3256}},
keywords = {{A/B testing; Continuous experimentation; Data-driven development; Empirical research; Multi-case study; Theory building}},
language = {{eng}},
number = {{1}},
publisher = {{Springer}},
series = {{Empirical Software Engineering}},
title = {{A theory of factors affecting continuous experimentation (FACE)}},
url = {{http://dx.doi.org/10.1007/s10664-023-10358-z}},
doi = {{10.1007/s10664-023-10358-z}},
volume = {{29}},
year = {{2024}},
}