Attention-Based Multi-Hypothesis Fusion for Speech Summarization

Takatomo Kano, Atsunori Ogawa, Marc Delcroix, Shinji Watanabe

研究成果: Conference contribution

1 被引用数 (Scopus)

抄録

Speech summarization, which generates a text summary from speech, can be achieved by combining automatic speech recognition (ASR) and text summarization (TS). With this cascade approach, we can exploit state-of-the-art models and large training datasets for both subtasks, i.e., Transformer for ASR and Bidirectional Encoder Representations from Transformers (BERT) for TS. However, ASR errors directly affect the quality of the output summary in the cascade approach. We propose a cascade speech summarization model that is robust to ASR errors and that exploits multiple hypotheses generated by ASR to attenuate the effect of ASR errors on the summary. We investigate several schemes to combine ASR hypotheses. First, we propose using the sum of sub-word embedding vectors weighted by their posterior values provided by an ASR system as an input to a BERT-based TS system. Then, we introduce a more general scheme that uses an attention-based fusion module added to a pre-trained BERT module to align and combine several ASR hypotheses. Finally, we perform speech summarization experiments on the How2 dataset and a newly assembled TED-based dataset that we will release with this paper11https://github.com/nttcslab-sp-admin/TEDSummary. These experiments show that retraining the BERT-based TS system with these schemes can improve summarization performance and that the attention-based fusion module is particularly effective.

本文言語English
ホスト出版物のタイトル2021 IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2021 - Proceedings
出版社Institute of Electrical and Electronics Engineers Inc.
ページ487-494
ページ数8
ISBN(電子版)9781665437394
DOI
出版ステータスPublished - 2021
外部発表はい
イベント2021 IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2021 - Cartagena, Colombia
継続期間: 2021 12月 132021 12月 17

出版物シリーズ

名前2021 IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2021 - Proceedings

Conference

Conference2021 IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2021
国/地域Colombia
CityCartagena
Period21/12/1321/12/17

ASJC Scopus subject areas

  • コンピュータ ビジョンおよびパターン認識
  • 信号処理
  • 言語学および言語

フィンガープリント

「Attention-Based Multi-Hypothesis Fusion for Speech Summarization」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル