Advanced

Hierarchies of Self-Organizing Maps for Action Recognition

Buonamente, Miriam LU ; Dindo, Haris and Johnsson, Magnus LU (2016) In Cognitive Systems Research
Abstract
We propose a hierarchical neural architecture able to recognise observed human actions. Each layer in the architecture represents increasingly complex human activity features. The first layer consists of a SOM which performs dimensionality reduction and clustering of the feature space. It represents the dynamics of the stream of posture frames in action sequences as activity trajectories over time. The second layer in the hierarchy consists of another SOM which clusters the activity trajectories of the first-layer SOM and learns to represent action prototypes. The third - and last - layer of the hierarchy consists of a neural network that learns to label action prototypes of the second-layer SOM and is independent - to certain extent - of... (More)
We propose a hierarchical neural architecture able to recognise observed human actions. Each layer in the architecture represents increasingly complex human activity features. The first layer consists of a SOM which performs dimensionality reduction and clustering of the feature space. It represents the dynamics of the stream of posture frames in action sequences as activity trajectories over time. The second layer in the hierarchy consists of another SOM which clusters the activity trajectories of the first-layer SOM and learns to represent action prototypes. The third - and last - layer of the hierarchy consists of a neural network that learns to label action prototypes of the second-layer SOM and is independent - to certain extent - of the camera’s angle and relative distance to the actor. The experiments were carried out with encouraging results with action movies taken from the INRIA 4D repository. In terms of representational accuracy, measured as the recognition rate over the training set, the architecture exhibits 100% accuracy indicating that actions with overlapping patterns of activity can be correctly discriminated. On the other hand, the architecture exhibits 53% recognition rate when presented with the same actions interpreted and performed by a different actor. Experiments on actions captured from different view points revealed a robustness of our system to camera rotation. Indeed, recognition accuracy was comparable to the single viewpoint case. To further assess the performance of the system we have also devised a behavioral experiments in which humans were asked to recognize the same set of actions, captured from different points of view. Results form such a behavioral study let us argue that our architecture is a good candidate as cognitive model of human action recognition, as architectural results are comparable to those observed in humans. (Less)
Please use this url to cite or link to this publication:
author
organization
publishing date
type
Contribution to journal
publication status
published
subject
keywords
Self-Organizing Map, Neural Network, Action Recognition, Hierarchical models, Intention Understanding
in
Cognitive Systems Research
publisher
Elsevier
external identifiers
  • scopus:84960503556
  • wos:000372430400004
ISSN
1389-0417
DOI
10.1016/j.cogsys.2015.12.009
project
What you see is what you do (WYSIWYD)
language
English
LU publication?
yes
id
418e37c2-b0ac-4652-801c-7aac17f4f1c1 (old id 8567645)
alternative location
http://www.sciencedirect.com/science/article/pii/S138904171600005X
date added to LUP
2016-01-26 12:51:04
date last changed
2017-07-09 03:14:26
@article{418e37c2-b0ac-4652-801c-7aac17f4f1c1,
  abstract     = {We propose a hierarchical neural architecture able to recognise observed human actions. Each layer in the architecture represents increasingly complex human activity features. The first layer consists of a SOM which performs dimensionality reduction and clustering of the feature space. It represents the dynamics of the stream of posture frames in action sequences as activity trajectories over time. The second layer in the hierarchy consists of another SOM which clusters the activity trajectories of the first-layer SOM and learns to represent action prototypes. The third - and last - layer of the hierarchy consists of a neural network that learns to label action prototypes of the second-layer SOM and is independent - to certain extent - of the camera’s angle and relative distance to the actor. The experiments were carried out with encouraging results with action movies taken from the INRIA 4D repository. In terms of representational accuracy, measured as the recognition rate over the training set, the architecture exhibits 100% accuracy indicating that actions with overlapping patterns of activity can be correctly discriminated. On the other hand, the architecture exhibits 53% recognition rate when presented with the same actions interpreted and performed by a different actor. Experiments on actions captured from different view points revealed a robustness of our system to camera rotation. Indeed, recognition accuracy was comparable to the single viewpoint case. To further assess the performance of the system we have also devised a behavioral experiments in which humans were asked to recognize the same set of actions, captured from different points of view. Results form such a behavioral study let us argue that our architecture is a good candidate as cognitive model of human action recognition, as architectural results are comparable to those observed in humans.},
  author       = {Buonamente, Miriam and Dindo, Haris and Johnsson, Magnus},
  issn         = {1389-0417},
  keyword      = {Self-Organizing Map,Neural Network,Action Recognition,Hierarchical models,Intention Understanding},
  language     = {eng},
  publisher    = {Elsevier},
  series       = {Cognitive Systems Research},
  title        = {Hierarchies of Self-Organizing Maps for Action Recognition},
  url          = {http://dx.doi.org/10.1016/j.cogsys.2015.12.009},
  year         = {2016},
}