Gesture, speech, and gaze cues for discourse segmentation

Francis Quek, David McNeill, Robert Bryll, Cemil Kirbas, Hasan Arslan, Karl E. McCullough, Nobuhiro Furuyama, Rashid Ansari

Research output: Chapter in Book/Report/Conference proceedingConference contribution

28 Citations (Scopus)

Abstract

Psycholinguistic evidence has established the complementary nature of the verbal and non-verbal aspects of human expression. We present our findings in the detection of these cues in interaction. We use the psycholinguistic device known as the 'catchment' as the locus of integration of gesture, speech and gaze components. We videotape conversation elicitation experiments in which subjects convey complex spatial plans to an interlocutor using a calibrated three-camera setup. We extract the gestural motion of both hands, gaze direction, and voiced units in the discourse and compare these with transcripts generated by expert microanalysis of the video. Our results show the complementary nature of these communicative modalities. Where there is ambiguity in the structure of one modality (such as in haplologies or owing to noise in the audio signal), other modalities provide evidence for correct segmentation.

Original languageEnglish
Title of host publicationProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
PublisherIEEE
Pages247-254
Number of pages8
Volume2
Publication statusPublished - 2000
Externally publishedYes
EventCVPR '2000: IEEE Conference on Computer Vision and Pattern Recognition - Hilton Head Island, SC, USA
Duration: 2000 Jun 132000 Jun 15

Other

OtherCVPR '2000: IEEE Conference on Computer Vision and Pattern Recognition
CityHilton Head Island, SC, USA
Period00/6/1300/6/15

Fingerprint

Microanalysis
Catchments
Cameras
Experiments

ASJC Scopus subject areas

  • Computer Vision and Pattern Recognition
  • Software
  • Control and Systems Engineering
  • Electrical and Electronic Engineering

Cite this

Quek, F., McNeill, D., Bryll, R., Kirbas, C., Arslan, H., McCullough, K. E., ... Ansari, R. (2000). Gesture, speech, and gaze cues for discourse segmentation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Vol. 2, pp. 247-254). IEEE.

Gesture, speech, and gaze cues for discourse segmentation. / Quek, Francis; McNeill, David; Bryll, Robert; Kirbas, Cemil; Arslan, Hasan; McCullough, Karl E.; Furuyama, Nobuhiro; Ansari, Rashid.

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Vol. 2 IEEE, 2000. p. 247-254.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Quek, F, McNeill, D, Bryll, R, Kirbas, C, Arslan, H, McCullough, KE, Furuyama, N & Ansari, R 2000, Gesture, speech, and gaze cues for discourse segmentation. in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. vol. 2, IEEE, pp. 247-254, CVPR '2000: IEEE Conference on Computer Vision and Pattern Recognition, Hilton Head Island, SC, USA, 00/6/13.
Quek F, McNeill D, Bryll R, Kirbas C, Arslan H, McCullough KE et al. Gesture, speech, and gaze cues for discourse segmentation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Vol. 2. IEEE. 2000. p. 247-254
Quek, Francis ; McNeill, David ; Bryll, Robert ; Kirbas, Cemil ; Arslan, Hasan ; McCullough, Karl E. ; Furuyama, Nobuhiro ; Ansari, Rashid. / Gesture, speech, and gaze cues for discourse segmentation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Vol. 2 IEEE, 2000. pp. 247-254
@inproceedings{33ac905f9de74f5ba1a406793456d22d,
title = "Gesture, speech, and gaze cues for discourse segmentation",
abstract = "Psycholinguistic evidence has established the complementary nature of the verbal and non-verbal aspects of human expression. We present our findings in the detection of these cues in interaction. We use the psycholinguistic device known as the 'catchment' as the locus of integration of gesture, speech and gaze components. We videotape conversation elicitation experiments in which subjects convey complex spatial plans to an interlocutor using a calibrated three-camera setup. We extract the gestural motion of both hands, gaze direction, and voiced units in the discourse and compare these with transcripts generated by expert microanalysis of the video. Our results show the complementary nature of these communicative modalities. Where there is ambiguity in the structure of one modality (such as in haplologies or owing to noise in the audio signal), other modalities provide evidence for correct segmentation.",
author = "Francis Quek and David McNeill and Robert Bryll and Cemil Kirbas and Hasan Arslan and McCullough, {Karl E.} and Nobuhiro Furuyama and Rashid Ansari",
year = "2000",
language = "English",
volume = "2",
pages = "247--254",
booktitle = "Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition",
publisher = "IEEE",

}

TY - GEN

T1 - Gesture, speech, and gaze cues for discourse segmentation

AU - Quek, Francis

AU - McNeill, David

AU - Bryll, Robert

AU - Kirbas, Cemil

AU - Arslan, Hasan

AU - McCullough, Karl E.

AU - Furuyama, Nobuhiro

AU - Ansari, Rashid

PY - 2000

Y1 - 2000

N2 - Psycholinguistic evidence has established the complementary nature of the verbal and non-verbal aspects of human expression. We present our findings in the detection of these cues in interaction. We use the psycholinguistic device known as the 'catchment' as the locus of integration of gesture, speech and gaze components. We videotape conversation elicitation experiments in which subjects convey complex spatial plans to an interlocutor using a calibrated three-camera setup. We extract the gestural motion of both hands, gaze direction, and voiced units in the discourse and compare these with transcripts generated by expert microanalysis of the video. Our results show the complementary nature of these communicative modalities. Where there is ambiguity in the structure of one modality (such as in haplologies or owing to noise in the audio signal), other modalities provide evidence for correct segmentation.

AB - Psycholinguistic evidence has established the complementary nature of the verbal and non-verbal aspects of human expression. We present our findings in the detection of these cues in interaction. We use the psycholinguistic device known as the 'catchment' as the locus of integration of gesture, speech and gaze components. We videotape conversation elicitation experiments in which subjects convey complex spatial plans to an interlocutor using a calibrated three-camera setup. We extract the gestural motion of both hands, gaze direction, and voiced units in the discourse and compare these with transcripts generated by expert microanalysis of the video. Our results show the complementary nature of these communicative modalities. Where there is ambiguity in the structure of one modality (such as in haplologies or owing to noise in the audio signal), other modalities provide evidence for correct segmentation.

UR - http://www.scopus.com/inward/record.url?scp=0033698728&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0033698728&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:0033698728

VL - 2

SP - 247

EP - 254

BT - Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

PB - IEEE

ER -