Multi-modal feature fusion for better understanding of human personality traits in social human–robot interaction

Zhihao Shen*, Armagan Elibol, Nak Young Chong


研究成果: Article査読

4 被引用数 (Scopus)


Since the dynamic nature of human–robot interaction becomes increasingly prevalent in our daily life, there is a great demand for enabling the robot to better understand human personality traits and inspiring humans to be more engaged in the interaction with the robot. Therefore, in this work, as we design the paradigm of human–robot interaction as close to the real situation as possible, the following three main problems are addressed: (1) fusion of visual and audio features of human interaction modalities, (2) integration of variable length feature vectors, and (3) compensation of shaky camera motion caused by movements of the robot's communicative gesture. Specifically, the three most important visual features of humans including head motion, gaze, and body motion were extracted from a camera mounted on the robot performing verbal and body gestures during the interaction. Then, our system was geared to fuse the aforementioned visual features and different types of vocal features, such as voice pitch, voice energy, and Mel-Frequency Cepstral Coefficient, dealing with variable length multiple feature vectors. Lastly, considering unknown patterns and sequential characteristics of human communicative behavior, we proposed a multi-layer Hidden Markov Model that improved the classification accuracy of personality traits and offered notable advantages of fusing the multiple features. The results were thoroughly analyzed and supported by psychological studies. The proposed multi-modal fusion approach is expected to deepen the communicative competence of social robots interacting with humans from different cultures and backgrounds.

ジャーナルRobotics and Autonomous Systems
出版ステータスPublished - 2021 12月

ASJC Scopus subject areas

  • 制御およびシステム工学
  • ソフトウェア
  • 数学 (全般)
  • コンピュータ サイエンスの応用


「Multi-modal feature fusion for better understanding of human personality traits in social human–robot interaction」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。