Development of audio-visual speech corpus toward speaker-independent Japanese LVCSR

Kazuto Ukai; Satoshi Tamura; Satoru Hayamizu

doi:10.1109/ICSDA.2016.7918976

Development of audio-visual speech corpus toward speaker-independent Japanese LVCSR

Kazuto Ukai, Satoshi Tamura, Satoru Hayamizu

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

1 Citation (Scopus)

Abstract

In the speech recognition literature, building corpora for Large Vocabulary Continuous Speech Recognition (LVCSR) is quite important. In addition, in order to overcome performance decrease caused by noise, using visual information such as lip images is effective. In this paper, therefore, we focus on collecting speech and lip-image data for audio-visual LVCSR. Audio-visual speech data were obtained from 12 speakers, each who uttered ATR503 phonetically-balanced sentences. These data were recorded in acoustically and visually clean environments. Using the data, we conducted recognition experiments. Mel Frequency Cepstral Coefficients (MFCCs) and eigenlip features were obtained, and multi-stream Hidden Markov Models (HMMs) were built. We compared the performance in clean condition to those in noisy environments. It is found that visual information is able to compensate the performance. In addition, it turns out that we should improve visual speech recognition for high-performance audio-visual LVCSR.

Original language	English
Title of host publication	2016 Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	12-15
Number of pages	4
ISBN (Electronic)	9781509035168
DOIs	https://doi.org/10.1109/ICSDA.2016.7918976
Publication status	Published - 2017 May 3
Externally published	Yes
Event	19th Annual Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016 - Bali, Indonesia Duration: 2016 Oct 26 → 2016 Oct 28

Publication series

Name	2016 Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016

Other

Other	19th Annual Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016
Country/Territory	Indonesia
City	Bali
Period	16/10/26 → 16/10/28

Keywords

LVCSR
audio-visual speech recognition
lipreading
multi-stream HMM

ASJC Scopus subject areas

Information Systems
Signal Processing
Information Systems and Management
Linguistics and Language

Access to Document

10.1109/ICSDA.2016.7918976

Cite this

Ukai, K., Tamura, S., & Hayamizu, S. (2017). Development of audio-visual speech corpus toward speaker-independent Japanese LVCSR. In 2016 Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016 (pp. 12-15). [7918976] (2016 Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICSDA.2016.7918976

Development of audio-visual speech corpus toward speaker-independent Japanese LVCSR. / Ukai, Kazuto; Tamura, Satoshi; Hayamizu, Satoru.

2016 Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016. Institute of Electrical and Electronics Engineers Inc., 2017. p. 12-15 7918976 (2016 Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Ukai, K, Tamura, S & Hayamizu, S 2017, Development of audio-visual speech corpus toward speaker-independent Japanese LVCSR. in 2016 Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016., 7918976, 2016 Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016, Institute of Electrical and Electronics Engineers Inc., pp. 12-15, 19th Annual Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016, Bali, Indonesia, 16/10/26. https://doi.org/10.1109/ICSDA.2016.7918976

Ukai K, Tamura S, Hayamizu S. Development of audio-visual speech corpus toward speaker-independent Japanese LVCSR. In 2016 Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016. Institute of Electrical and Electronics Engineers Inc. 2017. p. 12-15. 7918976. (2016 Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016). doi: 10.1109/ICSDA.2016.7918976

Ukai, Kazuto ; Tamura, Satoshi ; Hayamizu, Satoru. / Development of audio-visual speech corpus toward speaker-independent Japanese LVCSR. 2016 Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016. Institute of Electrical and Electronics Engineers Inc., 2017. pp. 12-15 (2016 Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016).

@inproceedings{308e1cd7111c4c6f95cc7539f9605d9a,

title = "Development of audio-visual speech corpus toward speaker-independent Japanese LVCSR",

abstract = "In the speech recognition literature, building corpora for Large Vocabulary Continuous Speech Recognition (LVCSR) is quite important. In addition, in order to overcome performance decrease caused by noise, using visual information such as lip images is effective. In this paper, therefore, we focus on collecting speech and lip-image data for audio-visual LVCSR. Audio-visual speech data were obtained from 12 speakers, each who uttered ATR503 phonetically-balanced sentences. These data were recorded in acoustically and visually clean environments. Using the data, we conducted recognition experiments. Mel Frequency Cepstral Coefficients (MFCCs) and eigenlip features were obtained, and multi-stream Hidden Markov Models (HMMs) were built. We compared the performance in clean condition to those in noisy environments. It is found that visual information is able to compensate the performance. In addition, it turns out that we should improve visual speech recognition for high-performance audio-visual LVCSR.",

keywords = "LVCSR, audio-visual speech recognition, lipreading, multi-stream HMM",

author = "Kazuto Ukai and Satoshi Tamura and Satoru Hayamizu",

note = "Publisher Copyright: {\textcopyright} 2016 IEEE.; 19th Annual Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016 ; Conference date: 26-10-2016 Through 28-10-2016",

year = "2017",

month = may,

day = "3",

doi = "10.1109/ICSDA.2016.7918976",

language = "English",

series = "2016 Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "12--15",

booktitle = "2016 Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016",

}

TY - GEN

T1 - Development of audio-visual speech corpus toward speaker-independent Japanese LVCSR

AU - Ukai, Kazuto

AU - Tamura, Satoshi

AU - Hayamizu, Satoru

PY - 2017/5/3

Y1 - 2017/5/3

N2 - In the speech recognition literature, building corpora for Large Vocabulary Continuous Speech Recognition (LVCSR) is quite important. In addition, in order to overcome performance decrease caused by noise, using visual information such as lip images is effective. In this paper, therefore, we focus on collecting speech and lip-image data for audio-visual LVCSR. Audio-visual speech data were obtained from 12 speakers, each who uttered ATR503 phonetically-balanced sentences. These data were recorded in acoustically and visually clean environments. Using the data, we conducted recognition experiments. Mel Frequency Cepstral Coefficients (MFCCs) and eigenlip features were obtained, and multi-stream Hidden Markov Models (HMMs) were built. We compared the performance in clean condition to those in noisy environments. It is found that visual information is able to compensate the performance. In addition, it turns out that we should improve visual speech recognition for high-performance audio-visual LVCSR.

AB - In the speech recognition literature, building corpora for Large Vocabulary Continuous Speech Recognition (LVCSR) is quite important. In addition, in order to overcome performance decrease caused by noise, using visual information such as lip images is effective. In this paper, therefore, we focus on collecting speech and lip-image data for audio-visual LVCSR. Audio-visual speech data were obtained from 12 speakers, each who uttered ATR503 phonetically-balanced sentences. These data were recorded in acoustically and visually clean environments. Using the data, we conducted recognition experiments. Mel Frequency Cepstral Coefficients (MFCCs) and eigenlip features were obtained, and multi-stream Hidden Markov Models (HMMs) were built. We compared the performance in clean condition to those in noisy environments. It is found that visual information is able to compensate the performance. In addition, it turns out that we should improve visual speech recognition for high-performance audio-visual LVCSR.

KW - LVCSR

KW - audio-visual speech recognition

KW - lipreading

KW - multi-stream HMM

UR - http://www.scopus.com/inward/record.url?scp=85020176208&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85020176208&partnerID=8YFLogxK

U2 - 10.1109/ICSDA.2016.7918976

DO - 10.1109/ICSDA.2016.7918976

M3 - Conference contribution

AN - SCOPUS:85020176208

T3 - 2016 Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016

SP - 12

EP - 15

BT - 2016 Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 19th Annual Conference of the Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Techniques, O-COCOSDA 2016

Y2 - 26 October 2016 through 28 October 2016

ER -

Development of audio-visual speech corpus toward speaker-independent Japanese LVCSR

Abstract

Publication series

Other

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this