Advanced

Ensemble based unsupervised anomaly detection

Pieta Theofanous, Alexander LU and Alstersjö, Erik LU (2017) EITM01 20162
Department of Electrical and Information Technology
Abstract
A methodology as well as a suggested solution to the problem of unsupervised anomaly detection for contextual anomalies is presented. Using a combination of statistical and clustering approaches, an ensemble of algorithms provide automatic anomaly detection in an Application-to-person networking environment which can be scaled to different domains using hierarchical time series data.

The aim of this thesis is to further advance the field of anomaly detection and to provide conclusions with regards to the usability, maintainability and trustworthiness of unsupervised anomaly detection frameworks. Applications in the domain of unsupervised anomaly detection are hard to evaluate, thus methods as well as future work, which can be used to... (More)
A methodology as well as a suggested solution to the problem of unsupervised anomaly detection for contextual anomalies is presented. Using a combination of statistical and clustering approaches, an ensemble of algorithms provide automatic anomaly detection in an Application-to-person networking environment which can be scaled to different domains using hierarchical time series data.

The aim of this thesis is to further advance the field of anomaly detection and to provide conclusions with regards to the usability, maintainability and trustworthiness of unsupervised anomaly detection frameworks. Applications in the domain of unsupervised anomaly detection are hard to evaluate, thus methods as well as future work, which can be used to further create unmitigated assertions about any data set, is investigated.

An introduction to the concepts underlying anomaly detection as well as an implementation of the concepts are presented. Principles of machine learning are applied using static thresholds and assumptions about the data set being monitored. No active learning or dynamic adjustments of the anomaly detection framework is applied with the drawback of limiting the resulting classification but still providing clear and robust insights into the analyzed data.



It is shown that purely statistical or naive probabilistic assumptions about any data monitored is inconclusive in producing a fair estimation of anomalies. For a setting where the utility of an anomaly detection framework are not adamant to the survival of a monitoring system, the proposed solution works adequately. Since the results have not been validated, no conclusions can be drawn with regards to recall and precision metrics. (Less)
Please use this url to cite or link to this publication:
author
Pieta Theofanous, Alexander LU and Alstersjö, Erik LU
supervisor
organization
alternative title
Sammansatta och oövervakade system för upptäckande av oregelbundenheter
course
EITM01 20162
year
type
H2 - Master's Degree (Two Years)
subject
keywords
unsupervised, ensemble based, computer engineering, Anomaly detection
report number
LU/LTH-EIT 2017-567
language
English
id
8904850
date added to LUP
2017-06-15 13:03:13
date last changed
2017-06-15 13:03:13
@misc{8904850,
  abstract     = {A methodology as well as a suggested solution to the problem of unsupervised anomaly detection for contextual anomalies is presented. Using a combination of statistical and clustering approaches, an ensemble of algorithms provide automatic anomaly detection in an Application-to-person networking environment which can be scaled to different domains using hierarchical time series data. 

The aim of this thesis is to further advance the field of anomaly detection and to provide conclusions with regards to the usability, maintainability and trustworthiness of unsupervised anomaly detection frameworks. Applications in the domain of unsupervised anomaly detection are hard to evaluate, thus methods as well as future work, which can be used to further create unmitigated assertions about any data set, is investigated. 

An introduction to the concepts underlying anomaly detection as well as an implementation of the concepts are presented. Principles of machine learning are applied using static thresholds and assumptions about the data set being monitored. No active learning or dynamic adjustments of the anomaly detection framework is applied with the drawback of limiting the resulting classification but still providing clear and robust insights into the analyzed data. 



It is shown that purely statistical or naive probabilistic assumptions about any data monitored is inconclusive in producing a fair estimation of anomalies. For a setting where the utility of an anomaly detection framework are not adamant to the survival of a monitoring system, the proposed solution works adequately. Since the results have not been validated, no conclusions can be drawn with regards to recall and precision metrics.},
  author       = {Pieta Theofanous, Alexander and Alstersjö, Erik},
  keyword      = {unsupervised,ensemble based,computer engineering,Anomaly detection},
  language     = {eng},
  note         = {Student Paper},
  title        = {Ensemble based unsupervised anomaly detection},
  year         = {2017},
}