Analysis of Multimodal Features for Speaking Proficiency Scoring in an Interview Dialogue

研究成果

1 被引用数 (Scopus)

抄録

This paper analyzes the effectiveness of different modalities in automated speaking proficiency scoring in an online dialogue task of non-native speakers. Conversational competence of a language learner can be assessed through the use of multimodal behaviors such as speech content, prosody, and visual cues. Although lexical and acoustic features have been widely studied, there has been no study on the usage of visual features, such as facial expressions and eye gaze. To build an automated speaking proficiency scoring system using multi-modal features, we first constructed an online video interview dataset of 210 Japanese English-learners with annotations of their speaking proficiency. We then examined two approaches for incorporating visual features and compared the effectiveness of each modality. Results show the end-to-end approach with deep neural networks achieves a higher correlation with human scoring than one with handcrafted features. Modalities are effective in the order of lexical, acoustic, and visual features.

本文言語English
ホスト出版物のタイトル2021 IEEE Spoken Language Technology Workshop, SLT 2021 - Proceedings
出版社Institute of Electrical and Electronics Engineers Inc.
ページ629-635
ページ数7
ISBN(電子版)9781728170664
DOI
出版ステータスPublished - 2021 1 19
イベント2021 IEEE Spoken Language Technology Workshop, SLT 2021 - Virtual, Shenzhen, China
継続期間: 2021 1 192021 1 22

出版物シリーズ

名前2021 IEEE Spoken Language Technology Workshop, SLT 2021 - Proceedings

Conference

Conference2021 IEEE Spoken Language Technology Workshop, SLT 2021
国/地域China
CityVirtual, Shenzhen
Period21/1/1921/1/22

ASJC Scopus subject areas

  • 言語学および言語
  • 言語および言語学
  • 人工知能
  • コンピュータ サイエンスの応用
  • コンピュータ ビジョンおよびパターン認識
  • ハードウェアとアーキテクチャ

フィンガープリント

「Analysis of Multimodal Features for Speaking Proficiency Scoring in an Interview Dialogue」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル