The use of semantic similarity tools in automated content scoring of fact-based essays written by EFL learners

Qiao Wang*

*この研究の対応する著者

研究成果: Article査読

抄録

This study searched for open-source semantic similarity tools and evaluated their effectiveness in automated content scoring of fact-based essays written by English-as-a-Foreign-Language (EFL) learners. Fifty writing samples under a fact-based writing task from an academic English course in a Japanese university were collected and a gold standard was produced by a native expert. A shortlist of carefully selected tools, including InferSent, spaCy, DKPro, ADW, SEMILAR and Latent Semantic Analysis, generated semantic similarity scores between student writing samples and the expert sample. Three teachers who were lecturers of the course manually graded the student samples on content. To ensure validity of human grades, samples with discrepant agreement were excluded and an inter-rater reliability test was conducted on remaining samples with quadratic weighted kappa. After the grades of the remaining samples were proven valid, a Pearson correlation analysis between semantic similarity scores and human grades was conducted and results showed that InferSent was the most effective tool in predicting the human grades. The study further pointed to the limitations of the six tools and suggested three alternatives to traditional methods in turning semantic similarity scores into reporting grades on content.

本文言語English
ページ(範囲)13021-13049
ページ数29
ジャーナルEducation and Information Technologies
27
9
DOI
出版ステータスPublished - 2022 11月

ASJC Scopus subject areas

  • 教育
  • 図書館情報学

フィンガープリント

「The use of semantic similarity tools in automated content scoring of fact-based essays written by EFL learners」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル