A/B Testing in the Small
Paulsson, Amalia ; Runeson, Per LU
and Ros, Rasmus
LU
(2022)
23:rd International Conference on Product-Focused Software Process Improvement, PROFES
In Lecture Notes in Computer Science
13709.
p.449-463
- Abstract
- Previous research on A/B testing and continuous experimentation has mostly focused on large-scale settings with company-external customers. However, work efficiency and satisfaction for company coworkers may be highly related to internal tools and their fit-for-use. In this study, we therefore explore A/B testing for online services exclusively used by company co-workers. We A/B tested two versions of Customer Admin, a tool that helps 34 500 IKEA co-workers to interact with customer data. The study comprised i) stakeholder interviews to understand objectives and phrase the experimentation goals, and ii) A/B test execution where data was collected and processed from approximately 350 users for 33 days. While the user base is relatively... (More)
- Previous research on A/B testing and continuous experimentation has mostly focused on large-scale settings with company-external customers. However, work efficiency and satisfaction for company coworkers may be highly related to internal tools and their fit-for-use. In this study, we therefore explore A/B testing for online services exclusively used by company co-workers. We A/B tested two versions of Customer Admin, a tool that helps 34 500 IKEA co-workers to interact with customer data. The study comprised i) stakeholder interviews to understand objectives and phrase the experimentation goals, and ii) A/B test execution where data was collected and processed from approximately 350 users for 33 days. While the user base is relatively large for this internal system, the primary metric data collected was too scarce to allow distinction between the two versions. However, secondary metrics and a user questionnaire suggest that the users are more efficient in the new menu design and that the users prefer it to the old. We conclude that A/B testing requires lots of data, making it less feasible for internal users, also for large, global organizations. Thus, we propose combining quantitative and qualitative evaluation of internal tools and propose further research on how to adapt A/B testing for smaller-scale settings. (Less)
Please use this url to cite or link to this publication:
https://lup.lub.lu.se/record/b9cda931-3414-4a24-ac5c-4bcd1c09f178
- author
- Paulsson, Amalia
; Runeson, Per
LU
and Ros, Rasmus
LU
- organization
- publishing date
- 2022-11-22
- type
- Chapter in Book/Report/Conference proceeding
- publication status
- published
- subject
- host publication
- International Conference on Product-Focused Software Process Improvement
- series title
- Lecture Notes in Computer Science
- editor
- Taibi, Davide ; Kuhrmann, Marco and Abrahamsson, Pekka
- volume
- 13709
- pages
- 15 pages
- publisher
- Springer
- conference name
- 23:rd International Conference on Product-Focused Software Process Improvement, PROFES
- conference location
- Jyväskylä, Finland
- conference dates
- 2022-11-21 - 2022-11-23
- external identifiers
-
- scopus:85142673584
- ISSN
- 1611-3349
- 0302-9743
- ISBN
- 978-3-031-21388-5
- 978-3-031-21387-8
- DOI
- 10.1007/978-3-031-21388-5_31
- project
- Continuous Experimentation and Optimization
- language
- English
- LU publication?
- yes
- id
- b9cda931-3414-4a24-ac5c-4bcd1c09f178
- date added to LUP
- 2022-10-16 20:04:42
- date last changed
- 2024-06-13 10:03:32
@inproceedings{b9cda931-3414-4a24-ac5c-4bcd1c09f178,
abstract = {{Previous research on A/B testing and continuous experimentation has mostly focused on large-scale settings with company-external customers. However, work efficiency and satisfaction for company coworkers may be highly related to internal tools and their fit-for-use. In this study, we therefore explore A/B testing for online services exclusively used by company co-workers. We A/B tested two versions of Customer Admin, a tool that helps 34 500 IKEA co-workers to interact with customer data. The study comprised i) stakeholder interviews to understand objectives and phrase the experimentation goals, and ii) A/B test execution where data was collected and processed from approximately 350 users for 33 days. While the user base is relatively large for this internal system, the primary metric data collected was too scarce to allow distinction between the two versions. However, secondary metrics and a user questionnaire suggest that the users are more efficient in the new menu design and that the users prefer it to the old. We conclude that A/B testing requires lots of data, making it less feasible for internal users, also for large, global organizations. Thus, we propose combining quantitative and qualitative evaluation of internal tools and propose further research on how to adapt A/B testing for smaller-scale settings.}},
author = {{Paulsson, Amalia and Runeson, Per and Ros, Rasmus}},
booktitle = {{International Conference on Product-Focused Software Process Improvement}},
editor = {{Taibi, Davide and Kuhrmann, Marco and Abrahamsson, Pekka}},
isbn = {{978-3-031-21388-5}},
issn = {{1611-3349}},
language = {{eng}},
month = {{11}},
pages = {{449--463}},
publisher = {{Springer}},
series = {{Lecture Notes in Computer Science}},
title = {{A/B Testing in the Small}},
url = {{http://dx.doi.org/10.1007/978-3-031-21388-5_31}},
doi = {{10.1007/978-3-031-21388-5_31}},
volume = {{13709}},
year = {{2022}},
}