Construct validation of analytic rating scales in a speaking assessment: Reporting a score profile and a composite

Research output: Contribution to journalArticle

40 Citations (Scopus)

Abstract

This is a construct validation study of a second language speaking assessment that reported a language profile based on analytic rating scales and a composite score. The study addressed three key issues: score dependability, convergent/discriminant validity of analytic rating scales and the weighting of analytic ratings in the composite score. Confirmatory factor analysis (CFA) and multivariate generalizability theory (G theory) were combined to analyze the responses of 214 admits to a study-abroad program to two role-play speaking tasks in a Spanish speaking assessment designed for student placement and diagnosis. The CFA and G theory approaches provided complementary information, which generally confirmed the key features of the assessment design: (1) the multicomponential and yet highly correlated nature of the five analytic rating scales: Pronunciation, Vocabulary, Cohesion, Organization and Grammar, (2) the high dependability of the ratings and the resulting placement decisions appropriate for the high-stakes decision-making context based on these analytic rating scales and the composite score, and (3) the largest contribution of Grammar to the composite score variance, which was consistent with the intention of program faculty members to reflect in the test design the relative importance of knowledge of grammar for students' academic success in the study-abroad program.

Original languageEnglish
Pages (from-to)355-390
Number of pages36
JournalLanguage Testing
Volume24
Issue number3
DOIs
Publication statusPublished - 2007 Jul
Externally publishedYes

ASJC Scopus subject areas

  • Language and Linguistics
  • Social Sciences (miscellaneous)
  • Linguistics and Language

Fingerprint Dive into the research topics of 'Construct validation of analytic rating scales in a speaking assessment: Reporting a score profile and a composite'. Together they form a unique fingerprint.

  • Cite this