Automatic scoring method for open answer task in the SJ-CAT speaking test considering utterance difficulty level

Hao Lu; Takeshi Yamada; Shingo Imai; Takahiro Shinozaki; Ryuichi Nisimura; Kenkichi Ishizuka; Shoji Makino; Nobuhiko Kitawaki

doi:10.1109/APSIPA.2014.7041583

Automatic scoring method for open answer task in the SJ-CAT speaking test considering utterance difficulty level

Hao Lu, Takeshi Yamada, Shingo Imai, Takahiro Shinozaki, Ryuichi Nisimura, Kenkichi Ishizuka, Shoji Makino, Nobuhiko Kitawaki

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In this paper, we propose an automatic scoring method for the open answer task of the Japanese speaking test SJ-CAT. The proposed method first extracts a set of features from an input answer utterance and then estimates a vocabulary richness score by human raters, which ranges from 0 to 4, by employing SVR (support vector regression). We devised a novel set of features, namely text statistics weighted by word reliability, to assess the abundance of vocabulary and expression, and degree of word relevance based on the hierarchical distance in a thesaurus to evaluate the suitability of vocabulary. We confirmed experimentally that the proposed method provides good estimates of the human richness score, with a correlation coefficient of 0.92 and an RMSE (root mean square error) of 0.56. We also showed that the proposed method is relatively robust to differences among examinees and among questions used for training and testing.

Original language	English
Title of host publication	2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9786163618238
DOIs	https://doi.org/10.1109/APSIPA.2014.7041583
Publication status	Published - 2014 Feb 12
Externally published	Yes
Event	2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014 - Chiang Mai, Thailand Duration: 2014 Dec 9 → 2014 Dec 12

Other

Other	2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014
Country	Thailand
City	Chiang Mai
Period	14/12/9 → 14/12/12

ASJC Scopus subject areas

Signal Processing
Information Systems

Access to Document

10.1109/APSIPA.2014.7041583

Fingerprint Dive into the research topics of 'Automatic scoring method for open answer task in the SJ-CAT speaking test considering utterance difficulty level'. Together they form a unique fingerprint.

Cite this

Lu, H., Yamada, T., Imai, S., Shinozaki, T., Nisimura, R., Ishizuka, K., Makino, S., & Kitawaki, N. (2014). Automatic scoring method for open answer task in the SJ-CAT speaking test considering utterance difficulty level. In 2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014 [7041583] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/APSIPA.2014.7041583

Automatic scoring method for open answer task in the SJ-CAT speaking test considering utterance difficulty level. / Lu, Hao; Yamada, Takeshi; Imai, Shingo; Shinozaki, Takahiro; Nisimura, Ryuichi; Ishizuka, Kenkichi; Makino, Shoji; Kitawaki, Nobuhiko.

2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014. Institute of Electrical and Electronics Engineers Inc., 2014. 7041583.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Lu, H, Yamada, T, Imai, S, Shinozaki, T, Nisimura, R, Ishizuka, K, Makino, S & Kitawaki, N 2014, Automatic scoring method for open answer task in the SJ-CAT speaking test considering utterance difficulty level. in 2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014., 7041583, Institute of Electrical and Electronics Engineers Inc., 2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014, Chiang Mai, Thailand, 14/12/9. https://doi.org/10.1109/APSIPA.2014.7041583

Lu H, Yamada T, Imai S, Shinozaki T, Nisimura R, Ishizuka K et al. Automatic scoring method for open answer task in the SJ-CAT speaking test considering utterance difficulty level. In 2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014. Institute of Electrical and Electronics Engineers Inc. 2014. 7041583 https://doi.org/10.1109/APSIPA.2014.7041583

Lu, Hao ; Yamada, Takeshi ; Imai, Shingo ; Shinozaki, Takahiro ; Nisimura, Ryuichi ; Ishizuka, Kenkichi ; Makino, Shoji ; Kitawaki, Nobuhiko. / Automatic scoring method for open answer task in the SJ-CAT speaking test considering utterance difficulty level. 2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014. Institute of Electrical and Electronics Engineers Inc., 2014.

@inproceedings{b28e780c8c324309a0b699b11a4564a0,

title = "Automatic scoring method for open answer task in the SJ-CAT speaking test considering utterance difficulty level",

abstract = "In this paper, we propose an automatic scoring method for the open answer task of the Japanese speaking test SJ-CAT. The proposed method first extracts a set of features from an input answer utterance and then estimates a vocabulary richness score by human raters, which ranges from 0 to 4, by employing SVR (support vector regression). We devised a novel set of features, namely text statistics weighted by word reliability, to assess the abundance of vocabulary and expression, and degree of word relevance based on the hierarchical distance in a thesaurus to evaluate the suitability of vocabulary. We confirmed experimentally that the proposed method provides good estimates of the human richness score, with a correlation coefficient of 0.92 and an RMSE (root mean square error) of 0.56. We also showed that the proposed method is relatively robust to differences among examinees and among questions used for training and testing.",

author = "Hao Lu and Takeshi Yamada and Shingo Imai and Takahiro Shinozaki and Ryuichi Nisimura and Kenkichi Ishizuka and Shoji Makino and Nobuhiko Kitawaki",

year = "2014",

month = feb,

day = "12",

doi = "10.1109/APSIPA.2014.7041583",

language = "English",

booktitle = "2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

note = "2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014 ; Conference date: 09-12-2014 Through 12-12-2014",

}

TY - GEN

T1 - Automatic scoring method for open answer task in the SJ-CAT speaking test considering utterance difficulty level

AU - Lu, Hao

AU - Yamada, Takeshi

AU - Imai, Shingo

AU - Shinozaki, Takahiro

AU - Nisimura, Ryuichi

AU - Ishizuka, Kenkichi

AU - Makino, Shoji

AU - Kitawaki, Nobuhiko

PY - 2014/2/12

Y1 - 2014/2/12

N2 - In this paper, we propose an automatic scoring method for the open answer task of the Japanese speaking test SJ-CAT. The proposed method first extracts a set of features from an input answer utterance and then estimates a vocabulary richness score by human raters, which ranges from 0 to 4, by employing SVR (support vector regression). We devised a novel set of features, namely text statistics weighted by word reliability, to assess the abundance of vocabulary and expression, and degree of word relevance based on the hierarchical distance in a thesaurus to evaluate the suitability of vocabulary. We confirmed experimentally that the proposed method provides good estimates of the human richness score, with a correlation coefficient of 0.92 and an RMSE (root mean square error) of 0.56. We also showed that the proposed method is relatively robust to differences among examinees and among questions used for training and testing.

AB - In this paper, we propose an automatic scoring method for the open answer task of the Japanese speaking test SJ-CAT. The proposed method first extracts a set of features from an input answer utterance and then estimates a vocabulary richness score by human raters, which ranges from 0 to 4, by employing SVR (support vector regression). We devised a novel set of features, namely text statistics weighted by word reliability, to assess the abundance of vocabulary and expression, and degree of word relevance based on the hierarchical distance in a thesaurus to evaluate the suitability of vocabulary. We confirmed experimentally that the proposed method provides good estimates of the human richness score, with a correlation coefficient of 0.92 and an RMSE (root mean square error) of 0.56. We also showed that the proposed method is relatively robust to differences among examinees and among questions used for training and testing.

UR - http://www.scopus.com/inward/record.url?scp=84949924127&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84949924127&partnerID=8YFLogxK

U2 - 10.1109/APSIPA.2014.7041583

DO - 10.1109/APSIPA.2014.7041583

M3 - Conference contribution

AN - SCOPUS:84949924127

BT - 2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014

Y2 - 9 December 2014 through 12 December 2014

ER -