Skip to main content

Lund University Publications

LUND UNIVERSITY LIBRARIES

Data-limited Continuous Experimentation (dlCE) : a literature review

Chren, Stanislav ; Fagerholm, Fabian ; Bjarnason, Elizabeth LU orcid ; Linåker, Johan LU orcid ; Rafi, Saima ; Lehtelä, Bettina ; Runeson, Per LU orcid and Kauppinen, Marjo (2025) 15th International Conference on Software Business, ICSOB 2024 In Lecture Notes in Business Information Processing 539. p.324-339
Abstract

Continuous experimentation (CE) is a software development approach where product decisions are data-driven. Large global internet-facing companies such as Microsoft, Google, and Facebook apply the practice by leveraging their massive user bases to obtain high statistical significance in experimentation results. However, companies with smaller user bases, such as small- and medium-sized enterprises (SMEs) and early-stage software startups, struggle to adopt CE due to limitations in user data. Our goal is to increase understanding of situations in CE where data limitations occur, as observed in the research literature. We investigate data limitations and challenges related to them, including their characteristics, and solutions and... (More)

Continuous experimentation (CE) is a software development approach where product decisions are data-driven. Large global internet-facing companies such as Microsoft, Google, and Facebook apply the practice by leveraging their massive user bases to obtain high statistical significance in experimentation results. However, companies with smaller user bases, such as small- and medium-sized enterprises (SMEs) and early-stage software startups, struggle to adopt CE due to limitations in user data. Our goal is to increase understanding of situations in CE where data limitations occur, as observed in the research literature. We investigate data limitations and challenges related to them, including their characteristics, and solutions and practices that may address these. We conducted a rapid review of CE papers from a previous systematic literature review, and analysed these to identify scenarios that exhibit data limitations in CE. We present a framework that illustrates different dimensions of data-limited CE (DLC) and connects scenarios to challenges and their potential future solutions. Most challenges and potential solutions that we found are related to the amount of data. We also note examples of other limitations related to, e.g., evaluation metrics, that provide interesting avenues for further research.

(Less)
Please use this url to cite or link to this publication:
author
; ; ; ; ; ; and
organization
publishing date
type
Chapter in Book/Report/Conference proceeding
publication status
published
subject
keywords
Continuous experimentation, Software engineering, User data
host publication
Software business : 15th International Conference, ICSOB 2024, Utrecht, The Netherlands, November 18–20, 2024, Proceedings - 15th International Conference, ICSOB 2024, Utrecht, The Netherlands, November 18–20, 2024, Proceedings
series title
Lecture Notes in Business Information Processing
editor
Papatheocharous, Efi ; Farshidi, Siamak ; Jansen, Slinger and Hyrynsalmi, Sonja
volume
539
pages
16 pages
conference name
15th International Conference on Software Business, ICSOB 2024
conference location
Utrecht, Netherlands
conference dates
2024-11-18 - 2024-11-20
external identifiers
  • scopus:105001396970
ISSN
1865-1348
1865-1356
ISBN
978-3-031-85849-9
978-3-031-85848-2
DOI
10.1007/978-3-031-85849-9_26
language
English
LU publication?
yes
id
10fa257e-0c74-4abf-8f37-a2ac074f5972
date added to LUP
2025-04-10 08:23:18
date last changed
2025-06-05 14:55:08
@inproceedings{10fa257e-0c74-4abf-8f37-a2ac074f5972,
  abstract     = {{<p>Continuous experimentation (CE) is a software development approach where product decisions are data-driven. Large global internet-facing companies such as Microsoft, Google, and Facebook apply the practice by leveraging their massive user bases to obtain high statistical significance in experimentation results. However, companies with smaller user bases, such as small- and medium-sized enterprises (SMEs) and early-stage software startups, struggle to adopt CE due to limitations in user data. Our goal is to increase understanding of situations in CE where data limitations occur, as observed in the research literature. We investigate data limitations and challenges related to them, including their characteristics, and solutions and practices that may address these. We conducted a rapid review of CE papers from a previous systematic literature review, and analysed these to identify scenarios that exhibit data limitations in CE. We present a framework that illustrates different dimensions of data-limited CE (DLC) and connects scenarios to challenges and their potential future solutions. Most challenges and potential solutions that we found are related to the amount of data. We also note examples of other limitations related to, e.g., evaluation metrics, that provide interesting avenues for further research.</p>}},
  author       = {{Chren, Stanislav and Fagerholm, Fabian and Bjarnason, Elizabeth and Linåker, Johan and Rafi, Saima and Lehtelä, Bettina and Runeson, Per and Kauppinen, Marjo}},
  booktitle    = {{Software business : 15th International Conference, ICSOB 2024, Utrecht, The Netherlands, November 18–20, 2024, Proceedings}},
  editor       = {{Papatheocharous, Efi and Farshidi, Siamak and Jansen, Slinger and Hyrynsalmi, Sonja}},
  isbn         = {{978-3-031-85849-9}},
  issn         = {{1865-1348}},
  keywords     = {{Continuous experimentation; Software engineering; User data}},
  language     = {{eng}},
  month        = {{03}},
  pages        = {{324--339}},
  series       = {{Lecture Notes in Business Information Processing}},
  title        = {{Data-limited Continuous Experimentation (dlCE) : a literature review}},
  url          = {{http://dx.doi.org/10.1007/978-3-031-85849-9_26}},
  doi          = {{10.1007/978-3-031-85849-9_26}},
  volume       = {{539}},
  year         = {{2025}},
}