Joint speaker diarization and speech recognition based on region proposal networks

Zili Huang*, Marc Delcroix, Leibny Paola Garcia, Shinji Watanabe, Desh Raj, Sanjeev Khudanpur

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

1 Citation (Scopus)


Speaker diarization, the process of partitioning an input audio stream into homogeneous segments according to the speaker identity, is an important task for speech processing. The standard clustering-based diarization pipeline (1) segments the whole utterance into small chunks, (2) extracts speaker embedding for each chunk, and (3) groups the chunks into clusters, where each cluster represents one speaker. It has two major disadvantages: first, it contains several individually optimized modules in the pipeline, and second, it cannot handle overlapping speech. To address these issues, we proposed region proposal network-based speaker diarization (RPNSD) (Huang et al., 2020). In this paper, we perform a detailed study of the RPNSD system, and make two important contributions. First, we report its diarization performance on additional datasets and empirically investigate the impact of different system settings. Second, we integrate an automatic speech recognition (ASR) component into the RPNSD system and propose a new framework called RPN-JOINT that simultaneously performs diarization and ASR. Our experiments reveal that (1) the RPNSD system can consistently achieve diarization results that are competitive with state-of-the-art methods, and (2) the RPN-JOINT system offers several advantages over the conventional cascade of diarization and ASR systems.

Original languageEnglish
Article number101316
JournalComputer Speech and Language
Publication statusPublished - 2022 Mar
Externally publishedYes


  • Faster R-CNN
  • Multi-speaker speech recognition
  • Region proposal network
  • Speaker diarization

ASJC Scopus subject areas

  • Software
  • Theoretical Computer Science
  • Human-Computer Interaction


Dive into the research topics of 'Joint speaker diarization and speech recognition based on region proposal networks'. Together they form a unique fingerprint.

Cite this