Testing Self-Adaptive Software with Probabilistic Guarantees on Performance Metrics : Extended and Comparative Results
(2022) In IEEE Transactions on Software Engineering 48(9). p.3554-3572- Abstract
This paper discusses methods to test the performance of the adaptation layer in a self-adaptive system. The problem is notoriously hard, due to the high degree of uncertainty and variability inherent in an adaptive software application. In particular, providing any type of formal guarantee for this problem is extremely difficult. In this paper we propose the use of a rigorous probabilistic approach to overcome the mentioned difficulties and provide probabilistic guarantees on the software performance. We describe the set up needed for the application of a probabilistic approach. We then discuss the traditional tools from statistics that could be applied to analyse the results, highlighting their limitations and motivating why they are... (More)
This paper discusses methods to test the performance of the adaptation layer in a self-adaptive system. The problem is notoriously hard, due to the high degree of uncertainty and variability inherent in an adaptive software application. In particular, providing any type of formal guarantee for this problem is extremely difficult. In this paper we propose the use of a rigorous probabilistic approach to overcome the mentioned difficulties and provide probabilistic guarantees on the software performance. We describe the set up needed for the application of a probabilistic approach. We then discuss the traditional tools from statistics that could be applied to analyse the results, highlighting their limitations and motivating why they are unsuitable for the given problem. We propose the use of a novel tool - the Scenario Theory - to overcome said limitations. We conclude the paper with a thorough empirical evaluation of the proposed approach, using three adaptive software applications: the Tele-Assistance Service, the Self-Adaptive Video Encoder, and the Traffic Reconfiguration via Adaptive Participatory Planning. With the first, we empirically expose the trade-off between data collection and confidence in the testing campaign. With the second, we demonstrate how to compare different adaptation strategies. With the third, we discuss the role of the randomisation in the selection of test inputs. In the evaluation, we apply the scenario theory and also classical statistical tools: Monte Carlo and Extreme Value Theory. We provide a complete evaluation and a thorough comparison of the confidence and guarantees that can be given with all the approaches.
(Less)
- author
- Mandrioli, Claudio LU and Maggio, Martina LU
- organization
- publishing date
- 2022
- type
- Contribution to journal
- publication status
- published
- subject
- keywords
- Autonomous Systems, Monte Carlo methods, Probabilistic logic, Self-Adaptive Software, Servers, Software, Testing, Tools, Uncertainty
- in
- IEEE Transactions on Software Engineering
- volume
- 48
- issue
- 9
- pages
- 3554 - 3572
- publisher
- IEEE - Institute of Electrical and Electronics Engineers Inc.
- external identifiers
-
- scopus:85112591299
- ISSN
- 0098-5589
- DOI
- 10.1109/TSE.2021.3101130
- language
- English
- LU publication?
- yes
- id
- 21a0d183-c1f8-4803-8ea0-03e9bdede961
- date added to LUP
- 2021-09-10 10:13:10
- date last changed
- 2022-10-31 14:57:31
@article{21a0d183-c1f8-4803-8ea0-03e9bdede961, abstract = {{<p>This paper discusses methods to test the performance of the adaptation layer in a self-adaptive system. The problem is notoriously hard, due to the high degree of uncertainty and variability inherent in an adaptive software application. In particular, providing any type of formal guarantee for this problem is extremely difficult. In this paper we propose the use of a rigorous probabilistic approach to overcome the mentioned difficulties and provide probabilistic guarantees on the software performance. We describe the set up needed for the application of a probabilistic approach. We then discuss the traditional tools from statistics that could be applied to analyse the results, highlighting their limitations and motivating why they are unsuitable for the given problem. We propose the use of a novel tool - the Scenario Theory - to overcome said limitations. We conclude the paper with a thorough empirical evaluation of the proposed approach, using three adaptive software applications: the Tele-Assistance Service, the Self-Adaptive Video Encoder, and the Traffic Reconfiguration via Adaptive Participatory Planning. With the first, we empirically expose the trade-off between data collection and confidence in the testing campaign. With the second, we demonstrate how to compare different adaptation strategies. With the third, we discuss the role of the randomisation in the selection of test inputs. In the evaluation, we apply the scenario theory and also classical statistical tools: Monte Carlo and Extreme Value Theory. We provide a complete evaluation and a thorough comparison of the confidence and guarantees that can be given with all the approaches.</p>}}, author = {{Mandrioli, Claudio and Maggio, Martina}}, issn = {{0098-5589}}, keywords = {{Autonomous Systems; Monte Carlo methods; Probabilistic logic; Self-Adaptive Software; Servers; Software; Testing; Tools; Uncertainty}}, language = {{eng}}, number = {{9}}, pages = {{3554--3572}}, publisher = {{IEEE - Institute of Electrical and Electronics Engineers Inc.}}, series = {{IEEE Transactions on Software Engineering}}, title = {{Testing Self-Adaptive Software with Probabilistic Guarantees on Performance Metrics : Extended and Comparative Results}}, url = {{http://dx.doi.org/10.1109/TSE.2021.3101130}}, doi = {{10.1109/TSE.2021.3101130}}, volume = {{48}}, year = {{2022}}, }