Can Humans Correct Errors From System? Investigating Error Tendencies in Speaker Identification Using Crowdsourcing

Yuta Ide, Susumu Saito, Teppei Nakano, Tetsuji Ogawa

Research output: Contribution to journalConference articlepeer-review

Abstract

An attempt was made to clarify the effectiveness of crowdsourcing on reducing errors in automatic speaker identification (ASID). It is possible to efficiently reduce errors by manually revalidating the unreliable results given by ASID systems. Ideally, errors should be corrected appropriately, and correct answers should not be miscorrected. In addition, a low false acceptance rate is desirable in authentication, but a high false rejection rate should be avoided from a usability viewpoint. It, however, is not certain that humans can achieve such an ideal SID, and in the case of crowdsourcing, the existence of malicious workers cannot be ignored. This study, therefore, investigates whether manual verification of error-prone inputs by crowd workers can reduce ASID errors and whether the resulting corrections are ideal. Experimental investigations on Amazon Mechanical Turk, in which 426 qualified workers identified 256 speech pairs from VoxCeleb data, demonstrated that crowdsourced verification can significantly reduce the number of false acceptances without increasing the number of false rejections compared to the results from the ASID system.

Original languageEnglish
Pages (from-to)5100-5104
Number of pages5
JournalProceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH
Volume2022-September
DOIs
Publication statusPublished - 2022
Event23rd Annual Conference of the International Speech Communication Association, INTERSPEECH 2022 - Incheon, Korea, Republic of
Duration: 2022 Sep 182022 Sep 22

Keywords

  • Amazon Mechanical Turk
  • crowdsourcing
  • human-assisted pattern recognition
  • speaker identification

ASJC Scopus subject areas

  • Language and Linguistics
  • Human-Computer Interaction
  • Signal Processing
  • Software
  • Modelling and Simulation

Fingerprint

Dive into the research topics of 'Can Humans Correct Errors From System? Investigating Error Tendencies in Speaker Identification Using Crowdsourcing'. Together they form a unique fingerprint.

Cite this