Advanced

Using simulation for assessing the real impact of test-coverage on defect-coverage

Briand, Lionel and Pfahl, Dietmar LU (2000) In IEEE Transactions on Reliability 49(1). p.60-70
Abstract
The use of test-coverage measures (e.g., block-coverage to control the software test process has become an increasingly common practice. This is justified by the assumption that higher test-coverage helps achieve higher defect-coverage and therefore improves software quality. In practice, data often show that defect-coverage and test-coverage grow over time, as additional testing is performed. However, it is unclear whether this phenomenon of concurrent growth can be attributed to a causal dependency, or if it is coincidental, simply due to the cumulative nature of both measures. Answering such a question is important as it determines whether a given test-coverage measure should be monitored for quality control and used to drive testing.... (More)
The use of test-coverage measures (e.g., block-coverage to control the software test process has become an increasingly common practice. This is justified by the assumption that higher test-coverage helps achieve higher defect-coverage and therefore improves software quality. In practice, data often show that defect-coverage and test-coverage grow over time, as additional testing is performed. However, it is unclear whether this phenomenon of concurrent growth can be attributed to a causal dependency, or if it is coincidental, simply due to the cumulative nature of both measures. Answering such a question is important as it determines whether a given test-coverage measure should be monitored for quality control and used to drive testing. Although it is no general answer to this problem, a procedure is proposed to investigate whether any test-coverage criterion has a genuine additional impact on defect-coverage when compared to the impact of just running additional test cases. This procedure applies in typical testing conditions where the software is tested once, according to a given strategy, coverage measures are collected as well as defect data. This procedure is tested on published data, and the results are compared with the original findings. The study outcomes do not support the assumption of a causal dependency between test-coverage and defect-coverage, a result for which several plausible explanations are provided. (Less)
Please use this url to cite or link to this publication:
author
publishing date
type
Contribution to journal
publication status
published
subject
keywords
monte carlo simulation, defect-coverage, software test, test-coverage, branch, test intensity
in
IEEE Transactions on Reliability
volume
49
issue
1
pages
60 - 70
publisher
Institute of Electrical and Electronics Engineers Inc.
external identifiers
  • Scopus:0034156363
ISSN
0018-9529
DOI
10.1109/24.855537
language
English
LU publication?
no
id
09b49599-7eb0-4def-8248-fb19a3abbbe0 (old id 1662020)
date added to LUP
2010-08-30 13:41:32
date last changed
2016-10-13 04:35:47
@misc{09b49599-7eb0-4def-8248-fb19a3abbbe0,
  abstract     = {The use of test-coverage measures (e.g., block-coverage to control the software test process has become an increasingly common practice. This is justified by the assumption that higher test-coverage helps achieve higher defect-coverage and therefore improves software quality. In practice, data often show that defect-coverage and test-coverage grow over time, as additional testing is performed. However, it is unclear whether this phenomenon of concurrent growth can be attributed to a causal dependency, or if it is coincidental, simply due to the cumulative nature of both measures. Answering such a question is important as it determines whether a given test-coverage measure should be monitored for quality control and used to drive testing. Although it is no general answer to this problem, a procedure is proposed to investigate whether any test-coverage criterion has a genuine additional impact on defect-coverage when compared to the impact of just running additional test cases. This procedure applies in typical testing conditions where the software is tested once, according to a given strategy, coverage measures are collected as well as defect data. This procedure is tested on published data, and the results are compared with the original findings. The study outcomes do not support the assumption of a causal dependency between test-coverage and defect-coverage, a result for which several plausible explanations are provided.},
  author       = {Briand, Lionel and Pfahl, Dietmar},
  issn         = {0018-9529},
  keyword      = {monte carlo simulation,defect-coverage,software test,test-coverage,branch,test intensity},
  language     = {eng},
  number       = {1},
  pages        = {60--70},
  publisher    = {ARRAY(0x8099850)},
  series       = {IEEE Transactions on Reliability},
  title        = {Using simulation for assessing the real impact of test-coverage on defect-coverage},
  url          = {http://dx.doi.org/10.1109/24.855537},
  volume       = {49},
  year         = {2000},
}