Stream Weight Estimation Using Higher Order Statistics in Multi-modal Speech Recognition

Kazuto Ukai; Satoshi Tamura; Satoru Hayamizu

Stream Weight Estimation Using Higher Order Statistics in Multi-modal Speech Recognition

Kazuto Ukai, Satoshi Tamura, Satoru Hayamizu

Green Computing Systems Research Organization

Research output: Contribution to conference › Paper › peer-review

Abstract

In this paper, stream weight optimization for multi-modal speech recognition using audio information and visual information is examined. In a conventional multi-stream Hidden Markov Model (HMM) used in multi-modal speech recognition, a constraint in which the summation of audio and visual weight factors should be one is employed. This means balance between transition and observation probabilities of HMM is fixed. We study an effective weight estimation indicator when releasing the constraint. Recognition experiments were conducted using an audio-visual corpus CENSREC-1-AV [1]. In noisy environments, effectiveness of deactivating the constraint is clarified for improving recognition accuracy. Subsequently higher-order statistical parameter (kurtosis) based stream weights were proposed and tested. Through recognition experiments, it is found proposed stream weights are successful.

Original language	English
Pages	181-184
Number of pages	4
Publication status	Published - 2015
Event	1st Joint Conference on Facial Analysis, Animation, and Auditory-Visual Speech Processing, FAAVSP 2015 - Vienna, Austria Duration: 2015 Sep 11 → 2015 Sep 13

Conference

Conference	1st Joint Conference on Facial Analysis, Animation, and Auditory-Visual Speech Processing, FAAVSP 2015
Country/Territory	Austria
City	Vienna
Period	15/9/11 → 15/9/13

Keywords

Kurtosis
Multi-modal speech recognition
Multi-stream HMM
Stream weight optimization

ASJC Scopus subject areas

Language and Linguistics
Speech and Hearing
Otorhinolaryngology

Cite this

@conference{1ae585b8facc4ec4b8f459d28c0e1443,

title = "Stream Weight Estimation Using Higher Order Statistics in Multi-modal Speech Recognition",

abstract = "In this paper, stream weight optimization for multi-modal speech recognition using audio information and visual information is examined. In a conventional multi-stream Hidden Markov Model (HMM) used in multi-modal speech recognition, a constraint in which the summation of audio and visual weight factors should be one is employed. This means balance between transition and observation probabilities of HMM is fixed. We study an effective weight estimation indicator when releasing the constraint. Recognition experiments were conducted using an audio-visual corpus CENSREC-1-AV [1]. In noisy environments, effectiveness of deactivating the constraint is clarified for improving recognition accuracy. Subsequently higher-order statistical parameter (kurtosis) based stream weights were proposed and tested. Through recognition experiments, it is found proposed stream weights are successful.",

keywords = "Kurtosis, Multi-modal speech recognition, Multi-stream HMM, Stream weight optimization",

author = "Kazuto Ukai and Satoshi Tamura and Satoru Hayamizu",

note = "Funding Information: A part of this work was supported by JSPS KAKENHI Grant Number 25730109. Publisher Copyright: {\textcopyright} 2015 Auditory-Visual Speech Processing 2015, AVSP 2015, held in conjunction with Facial Analysis and Animation, FAA 2015 - 1st Joint Conference on Facial Analysis, Animation, and Auditory-Visual Speech Processing, FAAVSP 2015. All rights reserved.; 1st Joint Conference on Facial Analysis, Animation, and Auditory-Visual Speech Processing, FAAVSP 2015 ; Conference date: 11-09-2015 Through 13-09-2015",

year = "2015",

language = "English",

pages = "181--184",

}

TY - CONF

T1 - Stream Weight Estimation Using Higher Order Statistics in Multi-modal Speech Recognition

AU - Ukai, Kazuto

AU - Tamura, Satoshi

AU - Hayamizu, Satoru

N1 - Funding Information: A part of this work was supported by JSPS KAKENHI Grant Number 25730109. Publisher Copyright: © 2015 Auditory-Visual Speech Processing 2015, AVSP 2015, held in conjunction with Facial Analysis and Animation, FAA 2015 - 1st Joint Conference on Facial Analysis, Animation, and Auditory-Visual Speech Processing, FAAVSP 2015. All rights reserved.

PY - 2015

Y1 - 2015

N2 - In this paper, stream weight optimization for multi-modal speech recognition using audio information and visual information is examined. In a conventional multi-stream Hidden Markov Model (HMM) used in multi-modal speech recognition, a constraint in which the summation of audio and visual weight factors should be one is employed. This means balance between transition and observation probabilities of HMM is fixed. We study an effective weight estimation indicator when releasing the constraint. Recognition experiments were conducted using an audio-visual corpus CENSREC-1-AV [1]. In noisy environments, effectiveness of deactivating the constraint is clarified for improving recognition accuracy. Subsequently higher-order statistical parameter (kurtosis) based stream weights were proposed and tested. Through recognition experiments, it is found proposed stream weights are successful.

AB - In this paper, stream weight optimization for multi-modal speech recognition using audio information and visual information is examined. In a conventional multi-stream Hidden Markov Model (HMM) used in multi-modal speech recognition, a constraint in which the summation of audio and visual weight factors should be one is employed. This means balance between transition and observation probabilities of HMM is fixed. We study an effective weight estimation indicator when releasing the constraint. Recognition experiments were conducted using an audio-visual corpus CENSREC-1-AV [1]. In noisy environments, effectiveness of deactivating the constraint is clarified for improving recognition accuracy. Subsequently higher-order statistical parameter (kurtosis) based stream weights were proposed and tested. Through recognition experiments, it is found proposed stream weights are successful.

KW - Kurtosis

KW - Multi-modal speech recognition

KW - Multi-stream HMM

KW - Stream weight optimization

UR - http://www.scopus.com/inward/record.url?scp=85133419498&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85133419498&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:85133419498

SP - 181

EP - 184

T2 - 1st Joint Conference on Facial Analysis, Animation, and Auditory-Visual Speech Processing, FAAVSP 2015

Y2 - 11 September 2015 through 13 September 2015

ER -

Stream Weight Estimation Using Higher Order Statistics in Multi-modal Speech Recognition

Abstract

Conference

Keywords

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this