Skip to main content

LUP Student Papers

LUND UNIVERSITY LIBRARIES

MISK-Moves

Elovson Grey, Bo LU (2023) MAML05 20231
Certec - Rehabilitation Engineering and Design
Abstract
Move-to-play is a musical instrument for persons with both cognitive and physical impairments, who would have trouble playing traditional instruments. Everyone, no matter their abilities, are given the chance to play and control music by moving their own body. This project is part of the MISK project, which is a collaboration between Certec, Furuboda folkhögskola and Eldorado resurscenter. To make this possible, a machine learning model for the detection of people and their limbs, pose estimation, is used. This enables the translation of body movement into data used for controlling music software, which is made as another part of the MISK project. It runs under Max 8 by Cycling '74. To make it easy and affordable for other creators to... (More)
Move-to-play is a musical instrument for persons with both cognitive and physical impairments, who would have trouble playing traditional instruments. Everyone, no matter their abilities, are given the chance to play and control music by moving their own body. This project is part of the MISK project, which is a collaboration between Certec, Furuboda folkhögskola and Eldorado resurscenter. To make this possible, a machine learning model for the detection of people and their limbs, pose estimation, is used. This enables the translation of body movement into data used for controlling music software, which is made as another part of the MISK project. It runs under Max 8 by Cycling '74. To make it easy and affordable for other creators to recreate the project, relatively cheap hardware has been used, a NVIDIA Jetson Nano and a Raspberry Pi V2 camera. It is a system-on-module, which compact size and low power consumption makes placement even in small compartments easy. The machine learning model used is ResNet-18 pose, which proved to have adequately sufficient precision and performance to track people and their limbs in real time. It has not mattered whether a person has been standing or sitting, playing the instrument has worked equally well in both cases. Move-to-play has some latency however, up to about 150 ms. Users will have to be selective about what or how the instrument should control the music, since different people have different sensitivity to the connection between their own actions, latency and the effect of their actions on the music. (Less)
Please use this url to cite or link to this publication:
author
Elovson Grey, Bo LU
supervisor
organization
alternative title
Using your body as a musical instrument
course
MAML05 20231
year
type
M2 - Bachelor Degree
subject
keywords
MISK, machine learning, ResNet-18, NVIDIA Jetson Nano, system-on-module, Certec, Furuboda KompetensCenter, Eldorado resurcenter, pose estimation, object detection, object tracking, Raspberry Pi V2 camera, Max 8
language
English
id
9129111
date added to LUP
2023-06-22 08:43:04
date last changed
2023-06-22 08:43:04
@misc{9129111,
  abstract     = {{Move-to-play is a musical instrument for persons with both cognitive and physical impairments, who would have trouble playing traditional instruments. Everyone, no matter their abilities, are given the chance to play and control music by moving their own body. This project is part of the MISK project, which is a collaboration between Certec, Furuboda folkhögskola and Eldorado resurscenter. To make this possible, a machine learning model for the detection of people and their limbs, pose estimation, is used. This enables the translation of body movement into data used for controlling music software, which is made as another part of the MISK project. It runs under Max 8 by Cycling '74. To make it easy and affordable for other creators to recreate the project, relatively cheap hardware has been used, a NVIDIA Jetson Nano and a Raspberry Pi V2 camera. It is a system-on-module, which compact size and low power consumption makes placement even in small compartments easy. The machine learning model used is ResNet-18 pose, which proved to have adequately sufficient precision and performance to track people and their limbs in real time. It has not mattered whether a person has been standing or sitting, playing the instrument has worked equally well in both cases. Move-to-play has some latency however, up to about 150 ms. Users will have to be selective about what or how the instrument should control the music, since different people have different sensitivity to the connection between their own actions, latency and the effect of their actions on the music.}},
  author       = {{Elovson Grey, Bo}},
  language     = {{eng}},
  note         = {{Student Paper}},
  title        = {{MISK-Moves}},
  year         = {{2023}},
}