Abstract
In this paper, we propose an automatic scoring method for the open answer task of the Japanese speaking test SJ-CAT. The proposed method first extracts a set of features from an input answer utterance and then estimates a vocabulary richness score by human raters, which ranges from 0 to 4, by employing SVR (support vector regression). We devised a novel set of features, namely text statistics weighted by word reliability, to assess the abundance of vocabulary and expression, and degree of word relevance based on the hierarchical distance in a thesaurus to evaluate the suitability of vocabulary. We confirmed experimentally that the proposed method provides good estimates of the human richness score, with a correlation coefficient of 0.92 and an RMSE (root mean square error) of 0.56. We also showed that the proposed method is relatively robust to differences among examinees and among questions used for training and testing.
| Original language | English |
|---|---|
| Title of host publication | 2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014 |
| Publisher | Institute of Electrical and Electronics Engineers Inc. |
| ISBN (Electronic) | 9786163618238 |
| DOIs | |
| Publication status | Published - 2014 Feb 12 |
| Externally published | Yes |
| Event | 2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014 - Chiang Mai, Thailand Duration: 2014 Dec 9 → 2014 Dec 12 |
Other
| Other | 2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014 |
|---|---|
| Country | Thailand |
| City | Chiang Mai |
| Period | 14/12/9 → 14/12/12 |
Fingerprint
ASJC Scopus subject areas
- Signal Processing
- Information Systems
Cite this
Automatic scoring method for open answer task in the SJ-CAT speaking test considering utterance difficulty level. / Lu, Hao; Yamada, Takeshi; Imai, Shingo; Shinozaki, Takahiro; Nisimura, Ryuichi; Ishizuka, Kenkichi; Makino, Shoji; Kitawaki, Nobuhiko.
2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014. Institute of Electrical and Electronics Engineers Inc., 2014. 7041583.Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
}
TY - GEN
T1 - Automatic scoring method for open answer task in the SJ-CAT speaking test considering utterance difficulty level
AU - Lu, Hao
AU - Yamada, Takeshi
AU - Imai, Shingo
AU - Shinozaki, Takahiro
AU - Nisimura, Ryuichi
AU - Ishizuka, Kenkichi
AU - Makino, Shoji
AU - Kitawaki, Nobuhiko
PY - 2014/2/12
Y1 - 2014/2/12
N2 - In this paper, we propose an automatic scoring method for the open answer task of the Japanese speaking test SJ-CAT. The proposed method first extracts a set of features from an input answer utterance and then estimates a vocabulary richness score by human raters, which ranges from 0 to 4, by employing SVR (support vector regression). We devised a novel set of features, namely text statistics weighted by word reliability, to assess the abundance of vocabulary and expression, and degree of word relevance based on the hierarchical distance in a thesaurus to evaluate the suitability of vocabulary. We confirmed experimentally that the proposed method provides good estimates of the human richness score, with a correlation coefficient of 0.92 and an RMSE (root mean square error) of 0.56. We also showed that the proposed method is relatively robust to differences among examinees and among questions used for training and testing.
AB - In this paper, we propose an automatic scoring method for the open answer task of the Japanese speaking test SJ-CAT. The proposed method first extracts a set of features from an input answer utterance and then estimates a vocabulary richness score by human raters, which ranges from 0 to 4, by employing SVR (support vector regression). We devised a novel set of features, namely text statistics weighted by word reliability, to assess the abundance of vocabulary and expression, and degree of word relevance based on the hierarchical distance in a thesaurus to evaluate the suitability of vocabulary. We confirmed experimentally that the proposed method provides good estimates of the human richness score, with a correlation coefficient of 0.92 and an RMSE (root mean square error) of 0.56. We also showed that the proposed method is relatively robust to differences among examinees and among questions used for training and testing.
UR - http://www.scopus.com/inward/record.url?scp=84949924127&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84949924127&partnerID=8YFLogxK
U2 - 10.1109/APSIPA.2014.7041583
DO - 10.1109/APSIPA.2014.7041583
M3 - Conference contribution
AN - SCOPUS:84949924127
BT - 2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA 2014
PB - Institute of Electrical and Electronics Engineers Inc.
ER -