The use of semantic similarity tools in automated content scoring of fact-based essays written by EFL learners

Qiao Wang*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

This study searched for open-source semantic similarity tools and evaluated their effectiveness in automated content scoring of fact-based essays written by English-as-a-Foreign-Language (EFL) learners. Fifty writing samples under a fact-based writing task from an academic English course in a Japanese university were collected and a gold standard was produced by a native expert. A shortlist of carefully selected tools, including InferSent, spaCy, DKPro, ADW, SEMILAR and Latent Semantic Analysis, generated semantic similarity scores between student writing samples and the expert sample. Three teachers who were lecturers of the course manually graded the student samples on content. To ensure validity of human grades, samples with discrepant agreement were excluded and an inter-rater reliability test was conducted on remaining samples with quadratic weighted kappa. After the grades of the remaining samples were proven valid, a Pearson correlation analysis between semantic similarity scores and human grades was conducted and results showed that InferSent was the most effective tool in predicting the human grades. The study further pointed to the limitations of the six tools and suggested three alternatives to traditional methods in turning semantic similarity scores into reporting grades on content.

Original languageEnglish
JournalEducation and Information Technologies
DOIs
Publication statusAccepted/In press - 2022

Keywords

  • Automated content scoring
  • Automated writing evaluation
  • EFL learners
  • Fact-based writing
  • Open-source semantic similarity tools
  • Semantic similarity

ASJC Scopus subject areas

  • Education
  • Library and Information Sciences

Fingerprint

Dive into the research topics of 'The use of semantic similarity tools in automated content scoring of fact-based essays written by EFL learners'. Together they form a unique fingerprint.

Cite this