Testing self-adaptive software with probabilistic guarantees on performance metrics
Mandrioli, Claudio LU
and Maggio, Martina
LU
(2020)
28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020
In ESEC/FSE 2020 - Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering
p.1002-1014
- Abstract
This paper discusses the problem of testing the performance of the adaptation layer in a self-adaptive system. The problem is notoriously hard, due to the high degree of uncertainty and variability inherent in an adaptive software application. In particular, providing any type of formal guarantee for this problem is extremely difficult. In this paper we propose the use of a rigorous probabilistic approach to overcome the mentioned difficulties and provide probabilistic guarantees on the software performance. We describe the set up needed for the application of a probabilistic approach. We then discuss the traditional tools from statistics that could be applied to analyse the results, highlighting their limitations and motivating why... (More)
This paper discusses the problem of testing the performance of the adaptation layer in a self-adaptive system. The problem is notoriously hard, due to the high degree of uncertainty and variability inherent in an adaptive software application. In particular, providing any type of formal guarantee for this problem is extremely difficult. In this paper we propose the use of a rigorous probabilistic approach to overcome the mentioned difficulties and provide probabilistic guarantees on the software performance. We describe the set up needed for the application of a probabilistic approach. We then discuss the traditional tools from statistics that could be applied to analyse the results, highlighting their limitations and motivating why they are unsuitable for the given problem. We propose the use of a novel tool - the scenario theory - to overcome said limitations. We conclude the paper with a thorough empirical evaluation of the proposed approach, using two adaptive software applications: the Tele-Assistance Service and the Self-Adaptive Video Encoder. With the first, we empirically expose the trade-off between data collection and confidence in the testing campaign. With the second, we demonstrate how to compare different adaptation strategies.
(Less)
- author
- Mandrioli, Claudio
LU
and Maggio, Martina
LU
- organization
- publishing date
- 2020
- type
- Chapter in Book/Report/Conference proceeding
- publication status
- published
- subject
- keywords
- Autonomous Systems, Self-Adaptive Software, Testing
- host publication
- ESEC/FSE 2020 - Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- series title
- ESEC/FSE 2020 - Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering
- editor
- Devanbu, Prem ; Cohen, Myra and Zimmermann, Thomas
- pages
- 13 pages
- publisher
- Association for Computing Machinery (ACM)
- conference name
- 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020
- conference location
- Virtual, Online, United States
- conference dates
- 2020-11-08 - 2020-11-13
- external identifiers
-
- scopus:85097138944
- ISBN
- 9781450370431
- DOI
- 10.1145/3368089.3409685
- project
- Testing Autonomous Control-Based Software Systems
- Testing of Self-Adaptive Software Systems
- Towards Adaptively Morphing Embedded Systems
- language
- English
- LU publication?
- yes
- id
- 70ea9c7d-9db7-46aa-9d24-7c617492d977
- alternative location
- http://admorph.eu/wp-content/uploads/2020/10/fse20_preprint.pdf
- date added to LUP
- 2020-12-15 13:29:29
- date last changed
- 2022-05-12 08:33:56
@inproceedings{70ea9c7d-9db7-46aa-9d24-7c617492d977,
abstract = {{<p>This paper discusses the problem of testing the performance of the adaptation layer in a self-adaptive system. The problem is notoriously hard, due to the high degree of uncertainty and variability inherent in an adaptive software application. In particular, providing any type of formal guarantee for this problem is extremely difficult. In this paper we propose the use of a rigorous probabilistic approach to overcome the mentioned difficulties and provide probabilistic guarantees on the software performance. We describe the set up needed for the application of a probabilistic approach. We then discuss the traditional tools from statistics that could be applied to analyse the results, highlighting their limitations and motivating why they are unsuitable for the given problem. We propose the use of a novel tool - the scenario theory - to overcome said limitations. We conclude the paper with a thorough empirical evaluation of the proposed approach, using two adaptive software applications: the Tele-Assistance Service and the Self-Adaptive Video Encoder. With the first, we empirically expose the trade-off between data collection and confidence in the testing campaign. With the second, we demonstrate how to compare different adaptation strategies.</p>}},
author = {{Mandrioli, Claudio and Maggio, Martina}},
booktitle = {{ESEC/FSE 2020 - Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering}},
editor = {{Devanbu, Prem and Cohen, Myra and Zimmermann, Thomas}},
isbn = {{9781450370431}},
keywords = {{Autonomous Systems; Self-Adaptive Software; Testing}},
language = {{eng}},
pages = {{1002--1014}},
publisher = {{Association for Computing Machinery (ACM)}},
series = {{ESEC/FSE 2020 - Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering}},
title = {{Testing self-adaptive software with probabilistic guarantees on performance metrics}},
url = {{http://dx.doi.org/10.1145/3368089.3409685}},
doi = {{10.1145/3368089.3409685}},
year = {{2020}},
}