Advanced

Low Bias Local Intrinsic Dimension Estimation from Expected Simplex Skewness

Johnsson, Kerstin LU ; Soneson, Charlotte and Fontes, Magnus LU (2015) In IEEE Transactions on Pattern Analysis and Machine Intelligence 37(1). p.196-202
Abstract
In exploratory high-dimensional data analysis, local intrinsic dimension estimation can sometimes be used in order to discriminate between data sets sampled from different low-dimensional structures. Global intrinsic dimension estimators can in many cases be adapted to local estimation, but this leads to problems with high negative bias or high variance. We introduce a method that exploits the curse/blessing of dimensionality and produces local intrinsic dimension estimators that have very low bias, even in cases where the intrinsic dimension is higher than the number of data points, in combination with relatively low variance. We show that our estimators have a very good ability to classify local data sets by their dimension compared to... (More)
In exploratory high-dimensional data analysis, local intrinsic dimension estimation can sometimes be used in order to discriminate between data sets sampled from different low-dimensional structures. Global intrinsic dimension estimators can in many cases be adapted to local estimation, but this leads to problems with high negative bias or high variance. We introduce a method that exploits the curse/blessing of dimensionality and produces local intrinsic dimension estimators that have very low bias, even in cases where the intrinsic dimension is higher than the number of data points, in combination with relatively low variance. We show that our estimators have a very good ability to classify local data sets by their dimension compared to other local intrinsic dimension estimators; furthermore we provide examples showing the usefulness of local intrinsic dimension estimation in general and our method in particular for stratification of real data sets. (Less)
Please use this url to cite or link to this publication:
author
organization
publishing date
type
Contribution to journal
publication status
published
subject
keywords
Eigenvalues and eigenfunctions, Calibration, Estimation, Dimension, Manifolds
in
IEEE Transactions on Pattern Analysis and Machine Intelligence
volume
37
issue
1
pages
196 - 202
publisher
IEEE--Institute of Electrical and Electronics Engineers Inc.
external identifiers
  • wos:000346970600017
  • scopus:84916930372
ISSN
1939-3539
DOI
10.1109/TPAMI.2014.2343220
language
English
LU publication?
yes
id
4c17cb4c-ff2d-48ef-b2f5-c0930c6aa865 (old id 4622255)
date added to LUP
2014-11-20 12:37:50
date last changed
2017-01-01 04:02:59
@article{4c17cb4c-ff2d-48ef-b2f5-c0930c6aa865,
  abstract     = {In exploratory high-dimensional data analysis, local intrinsic dimension estimation can sometimes be used in order to discriminate between data sets sampled from different low-dimensional structures. Global intrinsic dimension estimators can in many cases be adapted to local estimation, but this leads to problems with high negative bias or high variance. We introduce a method that exploits the curse/blessing of dimensionality and produces local intrinsic dimension estimators that have very low bias, even in cases where the intrinsic dimension is higher than the number of data points, in combination with relatively low variance. We show that our estimators have a very good ability to classify local data sets by their dimension compared to other local intrinsic dimension estimators; furthermore we provide examples showing the usefulness of local intrinsic dimension estimation in general and our method in particular for stratification of real data sets.},
  author       = {Johnsson, Kerstin and Soneson, Charlotte and Fontes, Magnus},
  issn         = {1939-3539},
  keyword      = {Eigenvalues and eigenfunctions,Calibration,Estimation,Dimension,Manifolds},
  language     = {eng},
  number       = {1},
  pages        = {196--202},
  publisher    = {IEEE--Institute of Electrical and Electronics Engineers Inc.},
  series       = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  title        = {Low Bias Local Intrinsic Dimension Estimation from Expected Simplex Skewness},
  url          = {http://dx.doi.org/10.1109/TPAMI.2014.2343220},
  volume       = {37},
  year         = {2015},
}