Digital Twins Are Not Monozygotic - Cross-Replicating ADAS Testing in Two Industry-Grade Automotive Simulators
(2021) 14th IEEE International Conference on Software Testing, Verification and Validation, ICST 2021 p.383-393- Abstract
The increasing levels of software- and data-intensive driving automation call for an evolution of automotive soft-ware testing. As a recommended practice of the Verification and Validation (VV) process of ISO/PAS 21448, a candidate standard for safety of the intended functionality for road vehicles, simulation-based testing has the potential to reduce both risks and costs. There is a growing body of research on devising test automation techniques using simulators for Advanced Driver-Assistance Systems (ADAS). However, how similar are the results if the same test scenarios are executed in different simulators? We conduct a replication study of applying a Search-Based Software Testing (SBST) solution to a real-world ADAS (PeVi, a... (More)
The increasing levels of software- and data-intensive driving automation call for an evolution of automotive soft-ware testing. As a recommended practice of the Verification and Validation (VV) process of ISO/PAS 21448, a candidate standard for safety of the intended functionality for road vehicles, simulation-based testing has the potential to reduce both risks and costs. There is a growing body of research on devising test automation techniques using simulators for Advanced Driver-Assistance Systems (ADAS). However, how similar are the results if the same test scenarios are executed in different simulators? We conduct a replication study of applying a Search-Based Software Testing (SBST) solution to a real-world ADAS (PeVi, a pedestrian vision detection system) using two different commercial simulators, namely, TASS/Siemens PreScan and ESI Pro-SiVIC. Based on a minimalistic scene, we compare critical test scenarios generated using our SBST solution in these two simulators. We show that SBST can be used to effectively generate critical test scenarios in both simulators, and the test results obtained from the two simulators can reveal several weaknesses of the ADAS under test. However, executing the same test scenarios in the two simulators leads to notable differences in the details of the test outputs, in particular, related to (1) safety violations revealed by tests, and (2) dynamics of cars and pedestrians. Based on our findings, we recommend future VV plans to include multiple simulators to support robust simulation-based testing and to base test objectives on measures that are less dependant on the internals of the simulators.
(Less)
- author
- Borg, Markus LU ; Abdessalem, Raja Ben ; Nejati, Shiva ; Jegeden, Francois Xavier and Shin, Donghwan
- organization
- publishing date
- 2021-04
- type
- Chapter in Book/Report/Conference proceeding
- publication status
- published
- subject
- keywords
- advanced driver-assistance systems, automotive simulators, replication, search-based software testing
- host publication
- Proceedings - 2021 IEEE 14th International Conference on Software Testing, Verification and Validation, ICST 2021
- article number
- 9438594
- pages
- 11 pages
- publisher
- IEEE - Institute of Electrical and Electronics Engineers Inc.
- conference name
- 14th IEEE International Conference on Software Testing, Verification and Validation, ICST 2021
- conference location
- Virtual, Porto de Galinhas, Brazil
- conference dates
- 2021-04-12 - 2021-04-16
- external identifiers
-
- scopus:85101260467
- ISBN
- 9781728168364
- 978-1-7281-6837-1
- DOI
- 10.1109/ICST49551.2021.00050
- language
- English
- LU publication?
- yes
- id
- 091333a1-30bb-4d6a-9fb3-c0d6cdd55b7a
- date added to LUP
- 2022-02-22 16:51:44
- date last changed
- 2024-07-11 13:31:52
@inproceedings{091333a1-30bb-4d6a-9fb3-c0d6cdd55b7a,
abstract = {{<p>The increasing levels of software- and data-intensive driving automation call for an evolution of automotive soft-ware testing. As a recommended practice of the Verification and Validation (VV) process of ISO/PAS 21448, a candidate standard for safety of the intended functionality for road vehicles, simulation-based testing has the potential to reduce both risks and costs. There is a growing body of research on devising test automation techniques using simulators for Advanced Driver-Assistance Systems (ADAS). However, how similar are the results if the same test scenarios are executed in different simulators? We conduct a replication study of applying a Search-Based Software Testing (SBST) solution to a real-world ADAS (PeVi, a pedestrian vision detection system) using two different commercial simulators, namely, TASS/Siemens PreScan and ESI Pro-SiVIC. Based on a minimalistic scene, we compare critical test scenarios generated using our SBST solution in these two simulators. We show that SBST can be used to effectively generate critical test scenarios in both simulators, and the test results obtained from the two simulators can reveal several weaknesses of the ADAS under test. However, executing the same test scenarios in the two simulators leads to notable differences in the details of the test outputs, in particular, related to (1) safety violations revealed by tests, and (2) dynamics of cars and pedestrians. Based on our findings, we recommend future VV plans to include multiple simulators to support robust simulation-based testing and to base test objectives on measures that are less dependant on the internals of the simulators.</p>}},
author = {{Borg, Markus and Abdessalem, Raja Ben and Nejati, Shiva and Jegeden, Francois Xavier and Shin, Donghwan}},
booktitle = {{Proceedings - 2021 IEEE 14th International Conference on Software Testing, Verification and Validation, ICST 2021}},
isbn = {{9781728168364}},
keywords = {{advanced driver-assistance systems; automotive simulators; replication; search-based software testing}},
language = {{eng}},
pages = {{383--393}},
publisher = {{IEEE - Institute of Electrical and Electronics Engineers Inc.}},
title = {{Digital Twins Are Not Monozygotic - Cross-Replicating ADAS Testing in Two Industry-Grade Automotive Simulators}},
url = {{http://dx.doi.org/10.1109/ICST49551.2021.00050}},
doi = {{10.1109/ICST49551.2021.00050}},
year = {{2021}},
}