Advanced

Are test cases needed? Replicated comparison between exploratory and test-case-based software testing

Itkonen, Juha and Mäntylä, Mika LU (2014) In Empirical Software Engineering 19(2). p.303-342
Abstract
Manual software testing is a widely practiced verification and validation method that is unlikely to fade away despite the advances in test automation. In the domain of manual testing, many practitioners advocate exploratory testing (ET), i.e., creative, experience-based testing without predesigned test cases, and they claim that it is more efficient than testing with detailed test cases. This paper reports a replicated experiment comparing effectiveness, efficiency, and perceived differences between ET and test-case-based testing (TCT) using 51 students as subjects, who performed manual functional testing on the jEdit text editor. Our results confirm the findings of the original study: 1) there is no difference in the defect detection... (More)
Manual software testing is a widely practiced verification and validation method that is unlikely to fade away despite the advances in test automation. In the domain of manual testing, many practitioners advocate exploratory testing (ET), i.e., creative, experience-based testing without predesigned test cases, and they claim that it is more efficient than testing with detailed test cases. This paper reports a replicated experiment comparing effectiveness, efficiency, and perceived differences between ET and test-case-based testing (TCT) using 51 students as subjects, who performed manual functional testing on the jEdit text editor. Our results confirm the findings of the original study: 1) there is no difference in the defect detection effectiveness between ET and TCT, 2) ET is more efficient by requiring less design effort, and 3) TCT produces more false-positive defect reports than ET. Based on the small differences in the experimental design, we also put forward a hypothesis that the effectiveness of the TCT approach would suffer more than ET from time pressure. We also found that both approaches had distinctive issues: in TCT, the problems were related to correct abstraction levels of test cases, and the problems in ET were related to test design and logging of the test execution and results. Finally, we recognize that TCT has other benefits over ET in managing and controlling testing in large organizations. (Less)
Please use this url to cite or link to this publication:
author
organization
publishing date
type
Contribution to journal
publication status
published
subject
keywords
Software testing, Manual testing, Test cases, Exploratory testing, Experiment, Effectiveness, Efficiency
in
Empirical Software Engineering
volume
19
issue
2
pages
303 - 342
publisher
Springer
external identifiers
  • wos:000333013100003
  • scopus:84896390487
ISSN
1573-7616
DOI
10.1007/s10664-013-9266-8
language
English
LU publication?
yes
id
53bc6297-d609-49ba-84f0-7a64816d47fe (old id 4410918)
date added to LUP
2014-04-29 11:00:58
date last changed
2017-05-21 03:04:26
@article{53bc6297-d609-49ba-84f0-7a64816d47fe,
  abstract     = {Manual software testing is a widely practiced verification and validation method that is unlikely to fade away despite the advances in test automation. In the domain of manual testing, many practitioners advocate exploratory testing (ET), i.e., creative, experience-based testing without predesigned test cases, and they claim that it is more efficient than testing with detailed test cases. This paper reports a replicated experiment comparing effectiveness, efficiency, and perceived differences between ET and test-case-based testing (TCT) using 51 students as subjects, who performed manual functional testing on the jEdit text editor. Our results confirm the findings of the original study: 1) there is no difference in the defect detection effectiveness between ET and TCT, 2) ET is more efficient by requiring less design effort, and 3) TCT produces more false-positive defect reports than ET. Based on the small differences in the experimental design, we also put forward a hypothesis that the effectiveness of the TCT approach would suffer more than ET from time pressure. We also found that both approaches had distinctive issues: in TCT, the problems were related to correct abstraction levels of test cases, and the problems in ET were related to test design and logging of the test execution and results. Finally, we recognize that TCT has other benefits over ET in managing and controlling testing in large organizations.},
  author       = {Itkonen, Juha and Mäntylä, Mika},
  issn         = {1573-7616},
  keyword      = {Software testing,Manual testing,Test cases,Exploratory testing,Experiment,Effectiveness,Efficiency},
  language     = {eng},
  number       = {2},
  pages        = {303--342},
  publisher    = {Springer},
  series       = {Empirical Software Engineering},
  title        = {Are test cases needed? Replicated comparison between exploratory and test-case-based software testing},
  url          = {http://dx.doi.org/10.1007/s10664-013-9266-8},
  volume       = {19},
  year         = {2014},
}