Advanced

Navigating Information Overload Caused by Automated Testing – A Clustering Approach in Multi-Branch Development

Erman, Nicklas; Tufvesson, Vanja; Borg, Markus LU ; Ardö, Anders LU and Runeson, Per LU (2015) International Conference on Software Testing, Verification and Validation, 2015 In 2015 IEEE 8th International Conference on Software Testing, Verification and Validation, ICST 2015 - Proceedings
Abstract
Background. Test automation is a widely used technique to increase the efficiency of software testing. However, executing more test cases increases the effort required to analyze test results. At Qlik, automated tests run nightly for up to 20 development branches, each containing thousands of test cases, resulting in information overload. Aim. We therefore develop a tool that supports the analysis of test results. Method. We create NIOCAT, a tool that clusters similar test case failures, to help the analyst identify underlying causes. To evaluate the tool, experiments on manually created subsets of failed test cases representing different use cases are conducted, and a focus group meeting is held with test analysts at Qlik. Results. The... (More)
Background. Test automation is a widely used technique to increase the efficiency of software testing. However, executing more test cases increases the effort required to analyze test results. At Qlik, automated tests run nightly for up to 20 development branches, each containing thousands of test cases, resulting in information overload. Aim. We therefore develop a tool that supports the analysis of test results. Method. We create NIOCAT, a tool that clusters similar test case failures, to help the analyst identify underlying causes. To evaluate the tool, experiments on manually created subsets of failed test cases representing different use cases are conducted, and a focus group meeting is held with test analysts at Qlik. Results. The case study shows that NIOCAT creates accurate clusters, in line with analyses performed by human analysts. Further, the potential time-savings of our approach is confirmed by the participants in the focus group. Conclusions. NIOCAT provides a feasible complement to current automated testing practices at Qlik by

reducing information overload. (Less)
Please use this url to cite or link to this publication:
author
organization
publishing date
type
Chapter in Book/Report/Conference proceeding
publication status
published
subject
keywords
software testing, test automation, test result analysis, clustering, case study
in
2015 IEEE 8th International Conference on Software Testing, Verification and Validation, ICST 2015 - Proceedings
publisher
IEEE--Institute of Electrical and Electronics Engineers Inc.
conference name
International Conference on Software Testing, Verification and Validation, 2015
external identifiers
  • Scopus:84935081504
ISBN
9781479971251
DOI
10.1109/ICST.2015.7102596
project
EASE
language
English
LU publication?
yes
id
d7639445-d325-47a5-9e62-3bc2d8317919 (old id 5152972)
date added to LUP
2015-03-30 11:02:33
date last changed
2016-10-13 05:05:38
@misc{d7639445-d325-47a5-9e62-3bc2d8317919,
  abstract     = {Background. Test automation is a widely used technique to increase the efficiency of software testing. However, executing more test cases increases the effort required to analyze test results. At Qlik, automated tests run nightly for up to 20 development branches, each containing thousands of test cases, resulting in information overload. Aim. We therefore develop a tool that supports the analysis of test results. Method. We create NIOCAT, a tool that clusters similar test case failures, to help the analyst identify underlying causes. To evaluate the tool, experiments on manually created subsets of failed test cases representing different use cases are conducted, and a focus group meeting is held with test analysts at Qlik. Results. The case study shows that NIOCAT creates accurate clusters, in line with analyses performed by human analysts. Further, the potential time-savings of our approach is confirmed by the participants in the focus group. Conclusions. NIOCAT provides a feasible complement to current automated testing practices at Qlik by<br/><br>
reducing information overload.},
  author       = {Erman, Nicklas and Tufvesson, Vanja and Borg, Markus and Ardö, Anders and Runeson, Per},
  isbn         = {9781479971251},
  keyword      = {software testing,test automation,test result analysis,clustering,case study},
  language     = {eng},
  publisher    = {ARRAY(0xb548668)},
  series       = {2015 IEEE 8th International Conference on Software Testing, Verification and Validation, ICST 2015 - Proceedings},
  title        = {Navigating Information Overload Caused by Automated Testing – A Clustering Approach in Multi-Branch Development},
  url          = {http://dx.doi.org/10.1109/ICST.2015.7102596},
  year         = {2015},
}