Advanced

Approaches to multimodal interfaces in modern mobile phones

Williamsson, Marcus (2009)
Certec - Rehabilitation Engineering and Design
Abstract (Swedish)
Multimodal user interfaces or interfaces that involve the user with several senses could greatly improve usability and accessibility. A mobile phone contains several hardware technologies such as accelerometers, touch screens and vibrators that can be used to support a multimodal user interface, but these capabilities are seldom used.

In this thesis multimodal interaction and how two-dimensional graphical and textual information in a modern mobile phone can be illustrated and enhanced in an understandable way using a multimodal user interface is explored. The thesis consists of three parts, the construction of a multimodal design concept, development of a software prototype using a Samsung Omnia and an evaluation of the concept and... (More)
Multimodal user interfaces or interfaces that involve the user with several senses could greatly improve usability and accessibility. A mobile phone contains several hardware technologies such as accelerometers, touch screens and vibrators that can be used to support a multimodal user interface, but these capabilities are seldom used.

In this thesis multimodal interaction and how two-dimensional graphical and textual information in a modern mobile phone can be illustrated and enhanced in an understandable way using a multimodal user interface is explored. The thesis consists of three parts, the construction of a multimodal design concept, development of a software prototype using a Samsung Omnia and an evaluation of the concept and the software prototype.

“Moving objects with physical attributes” was chosen as design concept and from the concept three ideas, the event handler, the image viewer and the weighted icon were derived.

The event handler is based on the concept of moving objects inside a container. When the phone acting as the container is moved or tilted the objects bounce against the sides and each other providing the user with sound and vibration feedback. This lets the user handle and identify new messages and other events without looking at the screen. The application also has a visual part that makes it easy to handle the events.

The image viewer allows the user to pan for images with the phone, as images move and bounce against the wall they age and after a while they float away making room for new images. This fun image browser illustrates the immersive qualities of multimodal interfaces.

The weighted icon is a drag and drop icon that provides the user with vibration and sound feedback based on its weight and drag velocity. It can be used to illustrate amounts and compare objects.

Several different aspects of the three ideas were tested and the experiences summarized. Among other things it was concluded that physics can work as glue between different senses and that a clear and simple design is important. (Less)
Please use this url to cite or link to this publication:
author
Williamsson, Marcus
supervisor
organization
year
type
H2 - Master's Degree (Two Years)
subject
keywords
Multimodal, mobile phones, interfaces, haptics, sound
report number
2:2009
language
English
id
1513865
date added to LUP
2010-03-04 13:59:51
date last changed
2010-03-04 13:59:51
@misc{1513865,
  abstract     = {Multimodal user interfaces or interfaces that involve the user with several senses could greatly improve usability and accessibility.   A mobile phone contains several hardware technologies such as accelerometers, touch screens and vibrators that can be used to support a multimodal user interface, but these capabilities are seldom used. 

In this thesis multimodal interaction and how two-dimensional graphical and textual information in a modern mobile phone can be illustrated and enhanced in an understandable way using a multimodal user interface is explored.  The thesis consists of three parts, the construction of a multimodal design concept, development of a software prototype using a Samsung Omnia and an evaluation of the concept and the software prototype. 

“Moving objects with physical attributes” was chosen as design concept and from the concept three ideas, the event handler, the image viewer and the weighted icon were derived. 

The event handler is based on the concept of moving objects inside a container. When the phone acting as the container is moved or tilted the objects bounce against the sides and each other providing the user with sound and vibration feedback. This lets the user handle and identify new messages and other events without looking at the screen. The application also has a visual part that makes it easy to handle the events.

The image viewer allows the user to pan for images with the phone, as images move and bounce against the wall they age and after a while they float away making room for new images. This fun image browser illustrates the immersive qualities of multimodal interfaces.

The weighted icon is a drag and drop icon that provides the user with vibration and sound feedback based on its weight and drag velocity. It can be used to illustrate amounts and compare objects.

Several different aspects of the three ideas were tested and the experiences summarized.  Among other things it was concluded that physics can work as glue between different senses and that a clear and simple design is important.},
  author       = {Williamsson, Marcus},
  keyword      = {Multimodal,mobile phones,interfaces,haptics,sound

},
  language     = {eng},
  note         = {Student Paper},
  title        = {Approaches to multimodal interfaces in modern mobile phones},
  year         = {2009},
}