Advanced

Using simulation for assessing the real impact of test coverage on defect coverage

Briand, Lionel and Pfahl, Dietmar LU (1999) ISSRE'99 - 10th International Symposium on Software Reliability Engineering In Proceedings of the 1999 10th International Symposium on Software Reliability Engineering, ISSRE'99 p.148-157
Abstract
The use of test coverage measures (e.g., block coverage) to control the software test process has become an increasingly common practice. This is justified by the assumption that higher test coverage helps achieve higher defect coverage and therefore improves software quality. In practice, data often shows that defect coverage and test coverage grow over time, as additional testing is performed. However, it is unclear whether this phenomenon of concurrent growth can be attributed to a causal dependent, or if it is coincidental, simply due to the cumulative nature of both measures. Answering such a question is important as it determines whether a given test coverage measure should be monitored for quality control and used to drive testing.... (More)
The use of test coverage measures (e.g., block coverage) to control the software test process has become an increasingly common practice. This is justified by the assumption that higher test coverage helps achieve higher defect coverage and therefore improves software quality. In practice, data often shows that defect coverage and test coverage grow over time, as additional testing is performed. However, it is unclear whether this phenomenon of concurrent growth can be attributed to a causal dependent, or if it is coincidental, simply due to the cumulative nature of both measures. Answering such a question is important as it determines whether a given test coverage measure should be monitored for quality control and used to drive testing. Although this is no general answer to the problem above, we propose a procedure to investigate whether any test coverage criterion has a genuine additional impact on defect coverage when compared to the impact of just running additional test cases. This procedure is applicable in typical testing conditions where the software is tested once, according to a given strategy, and where coverage measures are collected as well as defect data. We then test the procedure on published data and compare our results with the original findings. The study outcomes do not support the assumption of a causal dependency between test coverage and defect coverage, a result for which several plausible explanations are provided. (Less)
Please use this url to cite or link to this publication:
author
publishing date
type
Chapter in Book/Report/Conference proceeding
publication status
published
subject
keywords
Software tests, Computer simulation, Computer software selection and evaluation, Software engineering
in
Proceedings of the 1999 10th International Symposium on Software Reliability Engineering, ISSRE'99
pages
148 - 157
publisher
IEEE--Institute of Electrical and Electronics Engineers Inc.
conference name
ISSRE'99 - 10th International Symposium on Software Reliability Engineering
external identifiers
  • scopus:0033329421
ISSN
1071-9458
DOI
10.1109/ISSRE.1999.809319
language
English
LU publication?
no
id
d898641b-9c9f-4480-9f17-be2f447d461c (old id 1662433)
date added to LUP
2010-08-30 13:27:43
date last changed
2017-08-06 04:38:41
@inproceedings{d898641b-9c9f-4480-9f17-be2f447d461c,
  abstract     = {The use of test coverage measures (e.g., block coverage) to control the software test process has become an increasingly common practice. This is justified by the assumption that higher test coverage helps achieve higher defect coverage and therefore improves software quality. In practice, data often shows that defect coverage and test coverage grow over time, as additional testing is performed. However, it is unclear whether this phenomenon of concurrent growth can be attributed to a causal dependent, or if it is coincidental, simply due to the cumulative nature of both measures. Answering such a question is important as it determines whether a given test coverage measure should be monitored for quality control and used to drive testing. Although this is no general answer to the problem above, we propose a procedure to investigate whether any test coverage criterion has a genuine additional impact on defect coverage when compared to the impact of just running additional test cases. This procedure is applicable in typical testing conditions where the software is tested once, according to a given strategy, and where coverage measures are collected as well as defect data. We then test the procedure on published data and compare our results with the original findings. The study outcomes do not support the assumption of a causal dependency between test coverage and defect coverage, a result for which several plausible explanations are provided.},
  author       = {Briand, Lionel and Pfahl, Dietmar},
  booktitle    = {Proceedings of the 1999 10th International Symposium on Software Reliability Engineering, ISSRE'99},
  issn         = {1071-9458},
  keyword      = {Software tests,Computer simulation,Computer software selection and evaluation,Software engineering},
  language     = {eng},
  pages        = {148--157},
  publisher    = {IEEE--Institute of Electrical and Electronics Engineers Inc.},
  title        = {Using simulation for assessing the real impact of test coverage on defect coverage},
  url          = {http://dx.doi.org/10.1109/ISSRE.1999.809319},
  year         = {1999},
}