THE HINGE between input and output: Understanding the multimodal input fusion results in an agent-based multimodal presentation system

Yong Sun; Helmut Prendinger; Yu Shi; Fang Chen; Vera Chung; Mitsuru Ishizuka

doi:10.1145/1358628.1358878

THE HINGE between input and output: Understanding the multimodal input fusion results in an agent-based multimodal presentation system

Yong Sun^*, Helmut Prendinger, Yu Shi, Fang Chen, Vera Chung, Mitsuru Ishizuka

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

5 Citations (Scopus)

Abstract

A multimodal interface provides multiple modalities for input and output, such as speech, eye gaze and facial expression. With the recent progresses in multimodal interfaces, various approaches about multimodal input fusion and output generation have been proposed. However, less attention has been paid to how to integrate them together in a multimodal input and output system. This paper proposes an approach, termed as THE HINGE, in providing agent-based multimodal presentations in accordance with multimodal input fusion results. The analysis of experiment result shows the proposed approach enhances the flexibility of the system while maintains its stability.

Original language	English
Title of host publication	Conference on Human Factors in Computing Systems - Proceedings
Pages	3483-3488
Number of pages	6
DOIs	https://doi.org/10.1145/1358628.1358878
Publication status	Published - 2008
Externally published	Yes
Event	28th Annual CHI Conference on Human Factors in Computing Systems - Florence Duration: 2008 Apr 5 → 2008 Apr 10

Other

Other	28th Annual CHI Conference on Human Factors in Computing Systems
City	Florence
Period	08/4/5 → 08/4/10

Keywords

Discourse representation
Input understanding
Multimodal input fusion
Multimodal interfaces

ASJC Scopus subject areas

Human-Computer Interaction
Computer Graphics and Computer-Aided Design
Software

Access to Document

10.1145/1358628.1358878

Cite this

THE HINGE between input and output : Understanding the multimodal input fusion results in an agent-based multimodal presentation system. / Sun, Yong; Prendinger, Helmut; Shi, Yu; Chen, Fang; Chung, Vera; Ishizuka, Mitsuru.

Conference on Human Factors in Computing Systems - Proceedings. 2008. p. 3483-3488.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Sun, Y, Prendinger, H, Shi, Y, Chen, F, Chung, V & Ishizuka, M 2008, THE HINGE between input and output: Understanding the multimodal input fusion results in an agent-based multimodal presentation system. in Conference on Human Factors in Computing Systems - Proceedings. pp. 3483-3488, 28th Annual CHI Conference on Human Factors in Computing Systems, Florence, 08/4/5. https://doi.org/10.1145/1358628.1358878

@inproceedings{e4f0632c52e440bb8c1a45a87404c75d,

title = "THE HINGE between input and output: Understanding the multimodal input fusion results in an agent-based multimodal presentation system",

abstract = "A multimodal interface provides multiple modalities for input and output, such as speech, eye gaze and facial expression. With the recent progresses in multimodal interfaces, various approaches about multimodal input fusion and output generation have been proposed. However, less attention has been paid to how to integrate them together in a multimodal input and output system. This paper proposes an approach, termed as THE HINGE, in providing agent-based multimodal presentations in accordance with multimodal input fusion results. The analysis of experiment result shows the proposed approach enhances the flexibility of the system while maintains its stability.",

keywords = "Discourse representation, Input understanding, Multimodal input fusion, Multimodal interfaces",

author = "Yong Sun and Helmut Prendinger and Yu Shi and Fang Chen and Vera Chung and Mitsuru Ishizuka",

year = "2008",

doi = "10.1145/1358628.1358878",

language = "English",

pages = "3483--3488",

booktitle = "Conference on Human Factors in Computing Systems - Proceedings",

note = "28th Annual CHI Conference on Human Factors in Computing Systems ; Conference date: 05-04-2008 Through 10-04-2008",

}

TY - GEN

T1 - THE HINGE between input and output

T2 - 28th Annual CHI Conference on Human Factors in Computing Systems

AU - Sun, Yong

AU - Prendinger, Helmut

AU - Shi, Yu

AU - Chen, Fang

AU - Chung, Vera

AU - Ishizuka, Mitsuru

PY - 2008

Y1 - 2008

N2 - A multimodal interface provides multiple modalities for input and output, such as speech, eye gaze and facial expression. With the recent progresses in multimodal interfaces, various approaches about multimodal input fusion and output generation have been proposed. However, less attention has been paid to how to integrate them together in a multimodal input and output system. This paper proposes an approach, termed as THE HINGE, in providing agent-based multimodal presentations in accordance with multimodal input fusion results. The analysis of experiment result shows the proposed approach enhances the flexibility of the system while maintains its stability.

AB - A multimodal interface provides multiple modalities for input and output, such as speech, eye gaze and facial expression. With the recent progresses in multimodal interfaces, various approaches about multimodal input fusion and output generation have been proposed. However, less attention has been paid to how to integrate them together in a multimodal input and output system. This paper proposes an approach, termed as THE HINGE, in providing agent-based multimodal presentations in accordance with multimodal input fusion results. The analysis of experiment result shows the proposed approach enhances the flexibility of the system while maintains its stability.

KW - Discourse representation

KW - Input understanding

KW - Multimodal input fusion

KW - Multimodal interfaces

UR - http://www.scopus.com/inward/record.url?scp=57049088566&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=57049088566&partnerID=8YFLogxK

U2 - 10.1145/1358628.1358878

DO - 10.1145/1358628.1358878

M3 - Conference contribution

AN - SCOPUS:57049088566

SP - 3483

EP - 3488

BT - Conference on Human Factors in Computing Systems - Proceedings

Y2 - 5 April 2008 through 10 April 2008

ER -

THE HINGE between input and output: Understanding the multimodal input fusion results in an agent-based multimodal presentation system

Abstract

Other

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this