Navigating Information Overload Caused by Automated Testing – A Clustering Approach in Multi-Branch Development
Erman, Nicklas ; Tufvesson, Vanja ; Borg, Markus LU ; Ardö, Anders LU and Runeson, Per LU
(2015)
International Conference on Software Testing, Verification and Validation, 2015
- Abstract
- Background. Test automation is a widely used technique to increase the efficiency of software testing. However, executing more test cases increases the effort required to analyze test results. At Qlik, automated tests run nightly for up to 20 development branches, each containing thousands of test cases, resulting in information overload. Aim. We therefore develop a tool that supports the analysis of test results. Method. We create NIOCAT, a tool that clusters similar test case failures, to help the analyst identify underlying causes. To evaluate the tool, experiments on manually created subsets of failed test cases representing different use cases are conducted, and a focus group meeting is held with test analysts at Qlik. Results. The... (More)
- Background. Test automation is a widely used technique to increase the efficiency of software testing. However, executing more test cases increases the effort required to analyze test results. At Qlik, automated tests run nightly for up to 20 development branches, each containing thousands of test cases, resulting in information overload. Aim. We therefore develop a tool that supports the analysis of test results. Method. We create NIOCAT, a tool that clusters similar test case failures, to help the analyst identify underlying causes. To evaluate the tool, experiments on manually created subsets of failed test cases representing different use cases are conducted, and a focus group meeting is held with test analysts at Qlik. Results. The case study shows that NIOCAT creates accurate clusters, in line with analyses performed by human analysts. Further, the potential time-savings of our approach is confirmed by the participants in the focus group. Conclusions. NIOCAT provides a feasible complement to current automated testing practices at Qlik by
reducing information overload. (Less)
Please use this url to cite or link to this publication:
https://lup.lub.lu.se/record/5152972
- author
- Erman, Nicklas
; Tufvesson, Vanja
; Borg, Markus
LU
; Ardö, Anders
LU
and Runeson, Per
LU
- organization
- publishing date
- 2015
- type
- Chapter in Book/Report/Conference proceeding
- publication status
- published
- subject
- keywords
- software testing, test automation, test result analysis, clustering, case study
- host publication
- 2015 IEEE 8th International Conference on Software Testing, Verification and Validation, ICST 2015 - Proceedings
- article number
- 7102596
- publisher
- IEEE - Institute of Electrical and Electronics Engineers Inc.
- conference name
- International Conference on Software Testing, Verification and Validation, 2015
- conference location
- Graz, Austria
- conference dates
- 2015-04-13 - 2015-04-17
- external identifiers
-
- scopus:84935081504
- ISBN
- 9781479971251
- DOI
- 10.1109/ICST.2015.7102596
- project
- Embedded Applications Software Engineering
- language
- English
- LU publication?
- yes
- id
- d7639445-d325-47a5-9e62-3bc2d8317919 (old id 5152972)
- date added to LUP
- 2016-04-04 13:28:32
- date last changed
- 2023-04-06 13:33:04
@inproceedings{d7639445-d325-47a5-9e62-3bc2d8317919,
abstract = {{Background. Test automation is a widely used technique to increase the efficiency of software testing. However, executing more test cases increases the effort required to analyze test results. At Qlik, automated tests run nightly for up to 20 development branches, each containing thousands of test cases, resulting in information overload. Aim. We therefore develop a tool that supports the analysis of test results. Method. We create NIOCAT, a tool that clusters similar test case failures, to help the analyst identify underlying causes. To evaluate the tool, experiments on manually created subsets of failed test cases representing different use cases are conducted, and a focus group meeting is held with test analysts at Qlik. Results. The case study shows that NIOCAT creates accurate clusters, in line with analyses performed by human analysts. Further, the potential time-savings of our approach is confirmed by the participants in the focus group. Conclusions. NIOCAT provides a feasible complement to current automated testing practices at Qlik by<br/><br>
reducing information overload.}},
author = {{Erman, Nicklas and Tufvesson, Vanja and Borg, Markus and Ardö, Anders and Runeson, Per}},
booktitle = {{2015 IEEE 8th International Conference on Software Testing, Verification and Validation, ICST 2015 - Proceedings}},
isbn = {{9781479971251}},
keywords = {{software testing; test automation; test result analysis; clustering; case study}},
language = {{eng}},
publisher = {{IEEE - Institute of Electrical and Electronics Engineers Inc.}},
title = {{Navigating Information Overload Caused by Automated Testing – A Clustering Approach in Multi-Branch Development}},
url = {{https://lup.lub.lu.se/search/files/6967491/5265771.pdf}},
doi = {{10.1109/ICST.2015.7102596}},
year = {{2015}},
}