Advanced

Testing Highly Complex System of Systems: An Industrial Case Study

Ali, Nauman Bin; Petersen, Kai and Mäntylä, Mika LU (2012) 6th International Symposium on Empirical Software Engineering and Measurement (ESEM) In [Host publication title missing] p.211-220
Abstract
Context: Systems of systems (SoS) are highly complex and are integrated on multiple levels (unit, component, system, system of systems). Many of the characteristics of SoS (such as operational and managerial independence, integration of system into system of systems, SoS comprised of complex systems) make their development and testing challenging.

Contribution: This paper provides an understanding of SoS testing in large-scale industry settings with respect to challenges and how to address them.

Method: The research method used is case study research. As data collection methods we used interviews, documentation, and fault slippage data.

Results: We identified challenges related to SoS with respect to fault... (More)
Context: Systems of systems (SoS) are highly complex and are integrated on multiple levels (unit, component, system, system of systems). Many of the characteristics of SoS (such as operational and managerial independence, integration of system into system of systems, SoS comprised of complex systems) make their development and testing challenging.

Contribution: This paper provides an understanding of SoS testing in large-scale industry settings with respect to challenges and how to address them.

Method: The research method used is case study research. As data collection methods we used interviews, documentation, and fault slippage data.

Results: We identified challenges related to SoS with respect to fault slippage, test turn-around time, and test maintainability. We also classified the testing challenges to general testing challenges, challenges amplified by SoS, and challenges that are SoS specific. Interestingly, the interviewees agreed on the challenges, even though we sampled them with diversity in mind, which meant that the number of interviews conducted was sufficient to answer our research questions. We also identified solution proposals to the challenges that were categorized under four classes of developer quality assurance, function test, testing in all levels, and requirements engineering and communication.

Conclusion: We conclude that although over half of the challenges we identified can be categorized as general testing challenges still SoS systems have their unique and amplified challenges stemming from SoS characteristics. Furthermore, it was found that interviews and fault slippage data indicated that different areas in the software process should be improved, which indicates that using only one of these methods would have led to an incomplete picture of the challenges in the case company. (Less)
Please use this url to cite or link to this publication:
author
organization
publishing date
type
Chapter in Book/Report/Conference proceeding
publication status
published
subject
in
[Host publication title missing]
pages
211 - 220
publisher
ACM
conference name
6th International Symposium on Empirical Software Engineering and Measurement (ESEM)
external identifiers
  • scopus:84867555453
ISBN
978-1-4503-1056-7
DOI
10.1145/2372251.2372290
language
English
LU publication?
yes
id
19ca685d-6992-4d51-a47c-c8c0308b39c4 (old id 3053007)
date added to LUP
2014-01-16 15:33:16
date last changed
2017-05-14 04:33:29
@inproceedings{19ca685d-6992-4d51-a47c-c8c0308b39c4,
  abstract     = {Context: Systems of systems (SoS) are highly complex and are integrated on multiple levels (unit, component, system, system of systems). Many of the characteristics of SoS (such as operational and managerial independence, integration of system into system of systems, SoS comprised of complex systems) make their development and testing challenging.<br/><br>
Contribution: This paper provides an understanding of SoS testing in large-scale industry settings with respect to challenges and how to address them.<br/><br>
Method: The research method used is case study research. As data collection methods we used interviews, documentation, and fault slippage data.<br/><br>
Results: We identified challenges related to SoS with respect to fault slippage, test turn-around time, and test maintainability. We also classified the testing challenges to general testing challenges, challenges amplified by SoS, and challenges that are SoS specific. Interestingly, the interviewees agreed on the challenges, even though we sampled them with diversity in mind, which meant that the number of interviews conducted was sufficient to answer our research questions. We also identified solution proposals to the challenges that were categorized under four classes of developer quality assurance, function test, testing in all levels, and requirements engineering and communication.<br/><br>
Conclusion: We conclude that although over half of the challenges we identified can be categorized as general testing challenges still SoS systems have their unique and amplified challenges stemming from SoS characteristics. Furthermore, it was found that interviews and fault slippage data indicated that different areas in the software process should be improved, which indicates that using only one of these methods would have led to an incomplete picture of the challenges in the case company.},
  author       = {Ali, Nauman Bin and Petersen, Kai and Mäntylä, Mika},
  booktitle    = {[Host publication title missing]},
  isbn         = {978-1-4503-1056-7},
  language     = {eng},
  pages        = {211--220},
  publisher    = {ACM},
  title        = {Testing Highly Complex System of Systems: An Industrial Case Study},
  url          = {http://dx.doi.org/10.1145/2372251.2372290},
  year         = {2012},
}