Skip to main content

Lund University Publications

LUND UNIVERSITY LIBRARIES

Testing Self-Adaptive Software with Probabilistic Guarantees on Performance Metrics : Extended and Comparative Results

Mandrioli, Claudio LU orcid and Maggio, Martina LU (2022) In IEEE Transactions on Software Engineering 48(9). p.3554-3572
Abstract

This paper discusses methods to test the performance of the adaptation layer in a self-adaptive system. The problem is notoriously hard, due to the high degree of uncertainty and variability inherent in an adaptive software application. In particular, providing any type of formal guarantee for this problem is extremely difficult. In this paper we propose the use of a rigorous probabilistic approach to overcome the mentioned difficulties and provide probabilistic guarantees on the software performance. We describe the set up needed for the application of a probabilistic approach. We then discuss the traditional tools from statistics that could be applied to analyse the results, highlighting their limitations and motivating why they are... (More)

This paper discusses methods to test the performance of the adaptation layer in a self-adaptive system. The problem is notoriously hard, due to the high degree of uncertainty and variability inherent in an adaptive software application. In particular, providing any type of formal guarantee for this problem is extremely difficult. In this paper we propose the use of a rigorous probabilistic approach to overcome the mentioned difficulties and provide probabilistic guarantees on the software performance. We describe the set up needed for the application of a probabilistic approach. We then discuss the traditional tools from statistics that could be applied to analyse the results, highlighting their limitations and motivating why they are unsuitable for the given problem. We propose the use of a novel tool - the Scenario Theory - to overcome said limitations. We conclude the paper with a thorough empirical evaluation of the proposed approach, using three adaptive software applications: the Tele-Assistance Service, the Self-Adaptive Video Encoder, and the Traffic Reconfiguration via Adaptive Participatory Planning. With the first, we empirically expose the trade-off between data collection and confidence in the testing campaign. With the second, we demonstrate how to compare different adaptation strategies. With the third, we discuss the role of the randomisation in the selection of test inputs. In the evaluation, we apply the scenario theory and also classical statistical tools: Monte Carlo and Extreme Value Theory. We provide a complete evaluation and a thorough comparison of the confidence and guarantees that can be given with all the approaches.

(Less)
Please use this url to cite or link to this publication:
author
and
organization
publishing date
type
Contribution to journal
publication status
published
subject
keywords
Autonomous Systems, Monte Carlo methods, Probabilistic logic, Self-Adaptive Software, Servers, Software, Testing, Tools, Uncertainty
in
IEEE Transactions on Software Engineering
volume
48
issue
9
pages
3554 - 3572
publisher
IEEE - Institute of Electrical and Electronics Engineers Inc.
external identifiers
  • scopus:85112591299
ISSN
0098-5589
DOI
10.1109/TSE.2021.3101130
language
English
LU publication?
yes
id
21a0d183-c1f8-4803-8ea0-03e9bdede961
date added to LUP
2021-09-10 10:13:10
date last changed
2022-10-31 14:57:31
@article{21a0d183-c1f8-4803-8ea0-03e9bdede961,
  abstract     = {{<p>This paper discusses methods to test the performance of the adaptation layer in a self-adaptive system. The problem is notoriously hard, due to the high degree of uncertainty and variability inherent in an adaptive software application. In particular, providing any type of formal guarantee for this problem is extremely difficult. In this paper we propose the use of a rigorous probabilistic approach to overcome the mentioned difficulties and provide probabilistic guarantees on the software performance. We describe the set up needed for the application of a probabilistic approach. We then discuss the traditional tools from statistics that could be applied to analyse the results, highlighting their limitations and motivating why they are unsuitable for the given problem. We propose the use of a novel tool - the Scenario Theory - to overcome said limitations. We conclude the paper with a thorough empirical evaluation of the proposed approach, using three adaptive software applications: the Tele-Assistance Service, the Self-Adaptive Video Encoder, and the Traffic Reconfiguration via Adaptive Participatory Planning. With the first, we empirically expose the trade-off between data collection and confidence in the testing campaign. With the second, we demonstrate how to compare different adaptation strategies. With the third, we discuss the role of the randomisation in the selection of test inputs. In the evaluation, we apply the scenario theory and also classical statistical tools: Monte Carlo and Extreme Value Theory. We provide a complete evaluation and a thorough comparison of the confidence and guarantees that can be given with all the approaches.</p>}},
  author       = {{Mandrioli, Claudio and Maggio, Martina}},
  issn         = {{0098-5589}},
  keywords     = {{Autonomous Systems; Monte Carlo methods; Probabilistic logic; Self-Adaptive Software; Servers; Software; Testing; Tools; Uncertainty}},
  language     = {{eng}},
  number       = {{9}},
  pages        = {{3554--3572}},
  publisher    = {{IEEE - Institute of Electrical and Electronics Engineers Inc.}},
  series       = {{IEEE Transactions on Software Engineering}},
  title        = {{Testing Self-Adaptive Software with Probabilistic Guarantees on Performance Metrics : Extended and Comparative Results}},
  url          = {{http://dx.doi.org/10.1109/TSE.2021.3101130}},
  doi          = {{10.1109/TSE.2021.3101130}},
  volume       = {{48}},
  year         = {{2022}},
}