Advanced

Evaluation of Traceability Recovery in Context: A Taxonomy for Information Retrieval Tools

Borg, Markus LU ; Runeson, Per LU and Brodén, Lina (2012) 16th International Conference on Evaluation & Assessment in Software Engineering In 16th International Conference on Evaluation & Assessment in Software Engineering (EASE 2012) p.111-120
Abstract
Background: Development of complex, software intensive systems generates large amounts of information. Several researchers have developed tools implementing information retrieval (IR) approaches to suggest traceability links among artifacts. Aim: We explore the consequences of the fact that a majority of the evaluations of such tools have been focused on benchmarking of mere tool output. Method: To illustrate this issue, we have adapted a framework of general IR evaluations to a context taxonomy specifically for IR-based traceability recovery. Furthermore, we evaluate a previously proposed experimental framework by conducting a study using two publicly available tools on two datasets originating from development of embedded software... (More)
Background: Development of complex, software intensive systems generates large amounts of information. Several researchers have developed tools implementing information retrieval (IR) approaches to suggest traceability links among artifacts. Aim: We explore the consequences of the fact that a majority of the evaluations of such tools have been focused on benchmarking of mere tool output. Method: To illustrate this issue, we have adapted a framework of general IR evaluations to a context taxonomy specifically for IR-based traceability recovery. Furthermore, we evaluate a previously proposed experimental framework by conducting a study using two publicly available tools on two datasets originating from development of embedded software systems. Results: Our study shows that even though both datasets contain software artifacts from embedded development, the characteristics of the two datasets differ considerably, and consequently the traceability outcomes. Conclusions: To enable replications and secondary studies, we suggest that datasets should be thoroughly characterized in future studies on traceability recovery, especially when they can not be disclosed. Also, while we conclude that the experimental framework provides useful support, we argue that our proposed context taxonomy is a useful complement. Finally, we discuss how empirical evidence of the feasibility of IR-based traceability recovery can be strengthened in future research. (Less)
Please use this url to cite or link to this publication:
author
organization
publishing date
type
Chapter in Book/Report/Conference proceeding
publication status
published
subject
keywords
empirical software engineering, traceability recovery, information retrieval, experiment
in
16th International Conference on Evaluation & Assessment in Software Engineering (EASE 2012)
editor
Baldassarre, Teresa; Genero, Marcela; Mendes, Emilia and Piattini, Mario
pages
111 - 120
publisher
IEEE--Institute of Electrical and Electronics Engineers Inc.
conference name
16th International Conference on Evaluation & Assessment in Software Engineering
external identifiers
  • Scopus:84865504745
ISBN
978-1-84919-541-6
DOI
10.1049/ic.2012.0014
project
EASE
language
English
LU publication?
yes
id
96edc99f-810b-4356-80e5-3423e37223ec (old id 2540494)
date added to LUP
2012-05-15 10:21:02
date last changed
2016-10-13 04:48:15
@misc{96edc99f-810b-4356-80e5-3423e37223ec,
  abstract     = {Background: Development of complex, software intensive systems generates large amounts of information. Several researchers have developed tools implementing information retrieval (IR) approaches to suggest traceability links among artifacts. Aim: We explore the consequences of the fact that a majority of the evaluations of such tools have been focused on benchmarking of mere tool output. Method: To illustrate this issue, we have adapted a framework of general IR evaluations to a context taxonomy specifically for IR-based traceability recovery. Furthermore, we evaluate a previously proposed experimental framework by conducting a study using two publicly available tools on two datasets originating from development of embedded software systems. Results: Our study shows that even though both datasets contain software artifacts from embedded development, the characteristics of the two datasets differ considerably, and consequently the traceability outcomes. Conclusions: To enable replications and secondary studies, we suggest that datasets should be thoroughly characterized in future studies on traceability recovery, especially when they can not be disclosed. Also, while we conclude that the experimental framework provides useful support, we argue that our proposed context taxonomy is a useful complement. Finally, we discuss how empirical evidence of the feasibility of IR-based traceability recovery can be strengthened in future research.},
  author       = {Borg, Markus and Runeson, Per and Brodén, Lina},
  editor       = {Baldassarre, Teresa and Genero, Marcela and Mendes, Emilia and Piattini, Mario},
  isbn         = {978-1-84919-541-6},
  keyword      = {empirical software engineering,traceability recovery,information retrieval,experiment},
  language     = {eng},
  pages        = {111--120},
  publisher    = {ARRAY(0x98dc270)},
  series       = {16th International Conference on Evaluation & Assessment in Software Engineering (EASE 2012)},
  title        = {Evaluation of Traceability Recovery in Context: A Taxonomy for Information Retrieval Tools},
  url          = {http://dx.doi.org/10.1049/ic.2012.0014},
  year         = {2012},
}