Advanced

DoKnowMe: Towards a Domain Knowledgedriven Methodology for Performance Evaluation

Li, Zheng LU ; O'Brien, Liam and Kihl, Maria LU (2016) In SIGMETRICS Performance Evaluation Review 43(4). p.23-32
Abstract
Software engineering considers performance evaluation to be one of the key portions of software quality assurance. Unfortunately, there seems to be a lack of standard methodologies for performance evaluation even in the scope of experimental computer science. Inspired by the concept of “instantiation” in object-oriented programming, we distinguish the generic performance evaluation logic from the distributed and ad-hoc relevant studies, and develop an abstract evaluation methodology (by analogy of “class”) we name Domain Knowledge-driven Methodology (DoKnowMe). By replacing five predefined domain-specific knowledge artefacts, DoKnowMe can be instantiated into specific methodologies (by analogy of “object”) to guide evaluators in... (More)
Software engineering considers performance evaluation to be one of the key portions of software quality assurance. Unfortunately, there seems to be a lack of standard methodologies for performance evaluation even in the scope of experimental computer science. Inspired by the concept of “instantiation” in object-oriented programming, we distinguish the generic performance evaluation logic from the distributed and ad-hoc relevant studies, and develop an abstract evaluation methodology (by analogy of “class”) we name Domain Knowledge-driven Methodology (DoKnowMe). By replacing five predefined domain-specific knowledge artefacts, DoKnowMe can be instantiated into specific methodologies (by analogy of “object”) to guide evaluators in performance evaluation of different software and even computing systems. We also propose a generic validation framework with four indicators (i.e. usefulness, feasibility, effectiveness and repeatability), and use it to validate DoKnowMe in the Cloud services evaluation domain. Given the positive and promising validation result, we plan to integrate more common evaluation strategies to improve DoKnowMe and further focus on the performance evaluation of Cloud autoscaler systems. (Less)
Please use this url to cite or link to this publication:
author
organization
publishing date
type
Contribution to journal
publication status
published
subject
in
SIGMETRICS Performance Evaluation Review
volume
43
issue
4
pages
10 pages
publisher
Association for Computing Machinery (ACM)
external identifiers
  • scopus:85009754937
ISSN
0163-5999
DOI
10.1145/2897356.2897360
language
English
LU publication?
yes
id
6abda950-c068-42ca-986a-af562cdd5a58
date added to LUP
2016-05-06 02:37:09
date last changed
2017-02-05 04:49:51
@article{6abda950-c068-42ca-986a-af562cdd5a58,
  abstract     = {Software engineering considers performance evaluation to be one of the key portions of software quality assurance. Unfortunately, there seems to be a lack of standard methodologies for performance evaluation even in the scope of experimental computer science. Inspired by the concept of “instantiation” in object-oriented programming, we distinguish the generic performance evaluation logic from the distributed and ad-hoc relevant studies, and develop an abstract evaluation methodology (by analogy of “class”) we name Domain Knowledge-driven Methodology (DoKnowMe). By replacing five predefined domain-specific knowledge artefacts, DoKnowMe can be instantiated into specific methodologies (by analogy of “object”) to guide evaluators in performance evaluation of different software and even computing systems. We also propose a generic validation framework with four indicators (i.e. usefulness, feasibility, effectiveness and repeatability), and use it to validate DoKnowMe in the Cloud services evaluation domain. Given the positive and promising validation result, we plan to integrate more common evaluation strategies to improve DoKnowMe and further focus on the performance evaluation of Cloud autoscaler systems.},
  author       = {Li, Zheng and O'Brien, Liam and Kihl, Maria},
  issn         = {0163-5999},
  language     = {eng},
  number       = {4},
  pages        = {23--32},
  publisher    = {Association for Computing Machinery (ACM)},
  series       = {SIGMETRICS Performance Evaluation Review},
  title        = {DoKnowMe: Towards a Domain Knowledgedriven Methodology for Performance Evaluation},
  url          = {http://dx.doi.org/10.1145/2897356.2897360 },
  volume       = {43},
  year         = {2016},
}