Advanced

Testing self-adaptive software with probabilistic guarantees on performance metrics

Mandrioli, Claudio LU and Maggio, Martina LU (2020) 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020 In ESEC/FSE 2020 - Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering p.1002-1014
Abstract

This paper discusses the problem of testing the performance of the adaptation layer in a self-adaptive system. The problem is notoriously hard, due to the high degree of uncertainty and variability inherent in an adaptive software application. In particular, providing any type of formal guarantee for this problem is extremely difficult. In this paper we propose the use of a rigorous probabilistic approach to overcome the mentioned difficulties and provide probabilistic guarantees on the software performance. We describe the set up needed for the application of a probabilistic approach. We then discuss the traditional tools from statistics that could be applied to analyse the results, highlighting their limitations and motivating why... (More)

This paper discusses the problem of testing the performance of the adaptation layer in a self-adaptive system. The problem is notoriously hard, due to the high degree of uncertainty and variability inherent in an adaptive software application. In particular, providing any type of formal guarantee for this problem is extremely difficult. In this paper we propose the use of a rigorous probabilistic approach to overcome the mentioned difficulties and provide probabilistic guarantees on the software performance. We describe the set up needed for the application of a probabilistic approach. We then discuss the traditional tools from statistics that could be applied to analyse the results, highlighting their limitations and motivating why they are unsuitable for the given problem. We propose the use of a novel tool - the scenario theory - to overcome said limitations. We conclude the paper with a thorough empirical evaluation of the proposed approach, using two adaptive software applications: the Tele-Assistance Service and the Self-Adaptive Video Encoder. With the first, we empirically expose the trade-off between data collection and confidence in the testing campaign. With the second, we demonstrate how to compare different adaptation strategies.

(Less)
Please use this url to cite or link to this publication:
author
and
organization
publishing date
type
Chapter in Book/Report/Conference proceeding
publication status
published
subject
keywords
Autonomous Systems, Self-Adaptive Software, Testing
host publication
ESEC/FSE 2020 - Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering
series title
ESEC/FSE 2020 - Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering
editor
Devanbu, Prem ; Cohen, Myra and Zimmermann, Thomas
pages
13 pages
publisher
Association for Computing Machinery (ACM)
conference name
28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020
conference location
Virtual, Online, United States
conference dates
2020-11-08 - 2020-11-13
external identifiers
  • scopus:85097138944
ISBN
9781450370431
DOI
10.1145/3368089.3409685
project
Testing Autonomous Control-Based Software Systems
Testing of Self-Adaptive Software Systems
Towards Adaptively Morphing Embedded Systems
language
English
LU publication?
yes
id
70ea9c7d-9db7-46aa-9d24-7c617492d977
alternative location
http://admorph.eu/wp-content/uploads/2020/10/fse20_preprint.pdf
date added to LUP
2020-12-15 13:29:29
date last changed
2021-04-13 01:20:43
@inproceedings{70ea9c7d-9db7-46aa-9d24-7c617492d977,
  abstract     = {<p>This paper discusses the problem of testing the performance of the adaptation layer in a self-adaptive system. The problem is notoriously hard, due to the high degree of uncertainty and variability inherent in an adaptive software application. In particular, providing any type of formal guarantee for this problem is extremely difficult. In this paper we propose the use of a rigorous probabilistic approach to overcome the mentioned difficulties and provide probabilistic guarantees on the software performance. We describe the set up needed for the application of a probabilistic approach. We then discuss the traditional tools from statistics that could be applied to analyse the results, highlighting their limitations and motivating why they are unsuitable for the given problem. We propose the use of a novel tool - the scenario theory - to overcome said limitations. We conclude the paper with a thorough empirical evaluation of the proposed approach, using two adaptive software applications: the Tele-Assistance Service and the Self-Adaptive Video Encoder. With the first, we empirically expose the trade-off between data collection and confidence in the testing campaign. With the second, we demonstrate how to compare different adaptation strategies.</p>},
  author       = {Mandrioli, Claudio and Maggio, Martina},
  booktitle    = {ESEC/FSE 2020 - Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering},
  editor       = {Devanbu, Prem and Cohen, Myra and Zimmermann, Thomas},
  isbn         = {9781450370431},
  language     = {eng},
  pages        = {1002--1014},
  publisher    = {Association for Computing Machinery (ACM)},
  series       = {ESEC/FSE 2020 - Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering},
  title        = {Testing self-adaptive software with probabilistic guarantees on performance metrics},
  url          = {http://dx.doi.org/10.1145/3368089.3409685},
  doi          = {10.1145/3368089.3409685},
  year         = {2020},
}