The MEI robot: Towards using motherese to develop multimodal emotional intelligence

Angelica Lim, Hiroshi G. Okuno

Research output: Contribution to journalArticle

16 Citations (Scopus)

Abstract

We introduce the first steps in a developmental robot called MEI (multimodal emotional intelligence), a robot that can understand and express emotions in voice, gesture and gait using a controller trained only on voice. Whereas it is known that humans can perceive affect in voice, movement, music and even as little as point light displays, it is not clear how humans develop this skill. Is it innate? If not, how does this emotional intelligence develop in infants? The MEI robot develops these skills through vocal input and perceptual mapping of vocal features to other modalities. We base MEI's development on the idea that motherese is used as a way to associate dynamic vocal contours to facial emotion from an early age. MEI uses these dynamic contours to both understand and express multimodal emotions using a unified model called SIRE (Speed, Intensity, irRegularity, and Extent). Offline experiments with MEI support its cross-modal generalization ability: a model trained with voice data can recognize happiness, sadness, and fear in a completely different modality - human gait. User evaluations of the MEI robot speaking, gesturing and walking show that it can reliably express multimodal happiness and sadness using only the voice-trained model as a basis.

Original languageEnglish
Article number6798757
Pages (from-to)126-138
Number of pages13
JournalIEEE Transactions on Autonomous Mental Development
Volume6
Issue number2
DOIs
Publication statusPublished - 2014
Externally publishedYes

Fingerprint

Robots
Display devices
Controllers
Experiments

Keywords

  • Cross-modal recognition
  • emotion recognition
  • gait
  • gaussian mixture
  • gesture
  • motherese
  • SIRE
  • voice

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software

Cite this

The MEI robot : Towards using motherese to develop multimodal emotional intelligence. / Lim, Angelica; Okuno, Hiroshi G.

In: IEEE Transactions on Autonomous Mental Development, Vol. 6, No. 2, 6798757, 2014, p. 126-138.

Research output: Contribution to journalArticle

@article{1e444dc810474b0d941c036885426a67,
title = "The MEI robot: Towards using motherese to develop multimodal emotional intelligence",
abstract = "We introduce the first steps in a developmental robot called MEI (multimodal emotional intelligence), a robot that can understand and express emotions in voice, gesture and gait using a controller trained only on voice. Whereas it is known that humans can perceive affect in voice, movement, music and even as little as point light displays, it is not clear how humans develop this skill. Is it innate? If not, how does this emotional intelligence develop in infants? The MEI robot develops these skills through vocal input and perceptual mapping of vocal features to other modalities. We base MEI's development on the idea that motherese is used as a way to associate dynamic vocal contours to facial emotion from an early age. MEI uses these dynamic contours to both understand and express multimodal emotions using a unified model called SIRE (Speed, Intensity, irRegularity, and Extent). Offline experiments with MEI support its cross-modal generalization ability: a model trained with voice data can recognize happiness, sadness, and fear in a completely different modality - human gait. User evaluations of the MEI robot speaking, gesturing and walking show that it can reliably express multimodal happiness and sadness using only the voice-trained model as a basis.",
keywords = "Cross-modal recognition, emotion recognition, gait, gaussian mixture, gesture, motherese, SIRE, voice",
author = "Angelica Lim and Okuno, {Hiroshi G.}",
year = "2014",
doi = "10.1109/TAMD.2014.2317513",
language = "English",
volume = "6",
pages = "126--138",
journal = "IEEE Transactions on Cognitive and Developmental Systems",
issn = "2379-8920",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "2",

}

TY - JOUR

T1 - The MEI robot

T2 - Towards using motherese to develop multimodal emotional intelligence

AU - Lim, Angelica

AU - Okuno, Hiroshi G.

PY - 2014

Y1 - 2014

N2 - We introduce the first steps in a developmental robot called MEI (multimodal emotional intelligence), a robot that can understand and express emotions in voice, gesture and gait using a controller trained only on voice. Whereas it is known that humans can perceive affect in voice, movement, music and even as little as point light displays, it is not clear how humans develop this skill. Is it innate? If not, how does this emotional intelligence develop in infants? The MEI robot develops these skills through vocal input and perceptual mapping of vocal features to other modalities. We base MEI's development on the idea that motherese is used as a way to associate dynamic vocal contours to facial emotion from an early age. MEI uses these dynamic contours to both understand and express multimodal emotions using a unified model called SIRE (Speed, Intensity, irRegularity, and Extent). Offline experiments with MEI support its cross-modal generalization ability: a model trained with voice data can recognize happiness, sadness, and fear in a completely different modality - human gait. User evaluations of the MEI robot speaking, gesturing and walking show that it can reliably express multimodal happiness and sadness using only the voice-trained model as a basis.

AB - We introduce the first steps in a developmental robot called MEI (multimodal emotional intelligence), a robot that can understand and express emotions in voice, gesture and gait using a controller trained only on voice. Whereas it is known that humans can perceive affect in voice, movement, music and even as little as point light displays, it is not clear how humans develop this skill. Is it innate? If not, how does this emotional intelligence develop in infants? The MEI robot develops these skills through vocal input and perceptual mapping of vocal features to other modalities. We base MEI's development on the idea that motherese is used as a way to associate dynamic vocal contours to facial emotion from an early age. MEI uses these dynamic contours to both understand and express multimodal emotions using a unified model called SIRE (Speed, Intensity, irRegularity, and Extent). Offline experiments with MEI support its cross-modal generalization ability: a model trained with voice data can recognize happiness, sadness, and fear in a completely different modality - human gait. User evaluations of the MEI robot speaking, gesturing and walking show that it can reliably express multimodal happiness and sadness using only the voice-trained model as a basis.

KW - Cross-modal recognition

KW - emotion recognition

KW - gait

KW - gaussian mixture

KW - gesture

KW - motherese

KW - SIRE

KW - voice

UR - http://www.scopus.com/inward/record.url?scp=84903213220&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84903213220&partnerID=8YFLogxK

U2 - 10.1109/TAMD.2014.2317513

DO - 10.1109/TAMD.2014.2317513

M3 - Article

AN - SCOPUS:84903213220

VL - 6

SP - 126

EP - 138

JO - IEEE Transactions on Cognitive and Developmental Systems

JF - IEEE Transactions on Cognitive and Developmental Systems

SN - 2379-8920

IS - 2

M1 - 6798757

ER -