Construct validation of analytic rating scales in a speaking assessment

Reporting a score profile and a composite

Research output: Contribution to journalArticle

35 Citations (Scopus)

Abstract

This is a construct validation study of a second language speaking assessment that reported a language profile based on analytic rating scales and a composite score. The study addressed three key issues: score dependability, convergent/discriminant validity of analytic rating scales and the weighting of analytic ratings in the composite score. Confirmatory factor analysis (CFA) and multivariate generalizability theory (G theory) were combined to analyze the responses of 214 admits to a study-abroad program to two role-play speaking tasks in a Spanish speaking assessment designed for student placement and diagnosis. The CFA and G theory approaches provided complementary information, which generally confirmed the key features of the assessment design: (1) the multicomponential and yet highly correlated nature of the five analytic rating scales: Pronunciation, Vocabulary, Cohesion, Organization and Grammar, (2) the high dependability of the ratings and the resulting placement decisions appropriate for the high-stakes decision-making context based on these analytic rating scales and the composite score, and (3) the largest contribution of Grammar to the composite score variance, which was consistent with the intention of program faculty members to reflect in the test design the relative importance of knowledge of grammar for students' academic success in the study-abroad program.

Original languageEnglish
Pages (from-to)355-390
Number of pages36
JournalLanguage Testing
Volume24
Issue number3
DOIs
Publication statusPublished - 2007 Jul
Externally publishedYes

Fingerprint

rating scale
speaking
grammar
studies abroad
factor analysis
rating
academic success
role play
weighting
language
vocabulary
student
Rating Scales
decision making
organization
Grammar
Confirmatory Factor Analysis
Language
Study Abroad
Generalizability Theory

ASJC Scopus subject areas

  • Linguistics and Language

Cite this

Construct validation of analytic rating scales in a speaking assessment : Reporting a score profile and a composite. / Sawaki, Yasuyo.

In: Language Testing, Vol. 24, No. 3, 07.2007, p. 355-390.

Research output: Contribution to journalArticle

@article{c2037f0e7f0a461b99b2cd3f6567d8c6,
title = "Construct validation of analytic rating scales in a speaking assessment: Reporting a score profile and a composite",
abstract = "This is a construct validation study of a second language speaking assessment that reported a language profile based on analytic rating scales and a composite score. The study addressed three key issues: score dependability, convergent/discriminant validity of analytic rating scales and the weighting of analytic ratings in the composite score. Confirmatory factor analysis (CFA) and multivariate generalizability theory (G theory) were combined to analyze the responses of 214 admits to a study-abroad program to two role-play speaking tasks in a Spanish speaking assessment designed for student placement and diagnosis. The CFA and G theory approaches provided complementary information, which generally confirmed the key features of the assessment design: (1) the multicomponential and yet highly correlated nature of the five analytic rating scales: Pronunciation, Vocabulary, Cohesion, Organization and Grammar, (2) the high dependability of the ratings and the resulting placement decisions appropriate for the high-stakes decision-making context based on these analytic rating scales and the composite score, and (3) the largest contribution of Grammar to the composite score variance, which was consistent with the intention of program faculty members to reflect in the test design the relative importance of knowledge of grammar for students' academic success in the study-abroad program.",
author = "Yasuyo Sawaki",
year = "2007",
month = "7",
doi = "10.1177/0265532207077205",
language = "English",
volume = "24",
pages = "355--390",
journal = "Language Testing",
issn = "0265-5322",
publisher = "SAGE Publications Ltd",
number = "3",

}

TY - JOUR

T1 - Construct validation of analytic rating scales in a speaking assessment

T2 - Reporting a score profile and a composite

AU - Sawaki, Yasuyo

PY - 2007/7

Y1 - 2007/7

N2 - This is a construct validation study of a second language speaking assessment that reported a language profile based on analytic rating scales and a composite score. The study addressed three key issues: score dependability, convergent/discriminant validity of analytic rating scales and the weighting of analytic ratings in the composite score. Confirmatory factor analysis (CFA) and multivariate generalizability theory (G theory) were combined to analyze the responses of 214 admits to a study-abroad program to two role-play speaking tasks in a Spanish speaking assessment designed for student placement and diagnosis. The CFA and G theory approaches provided complementary information, which generally confirmed the key features of the assessment design: (1) the multicomponential and yet highly correlated nature of the five analytic rating scales: Pronunciation, Vocabulary, Cohesion, Organization and Grammar, (2) the high dependability of the ratings and the resulting placement decisions appropriate for the high-stakes decision-making context based on these analytic rating scales and the composite score, and (3) the largest contribution of Grammar to the composite score variance, which was consistent with the intention of program faculty members to reflect in the test design the relative importance of knowledge of grammar for students' academic success in the study-abroad program.

AB - This is a construct validation study of a second language speaking assessment that reported a language profile based on analytic rating scales and a composite score. The study addressed three key issues: score dependability, convergent/discriminant validity of analytic rating scales and the weighting of analytic ratings in the composite score. Confirmatory factor analysis (CFA) and multivariate generalizability theory (G theory) were combined to analyze the responses of 214 admits to a study-abroad program to two role-play speaking tasks in a Spanish speaking assessment designed for student placement and diagnosis. The CFA and G theory approaches provided complementary information, which generally confirmed the key features of the assessment design: (1) the multicomponential and yet highly correlated nature of the five analytic rating scales: Pronunciation, Vocabulary, Cohesion, Organization and Grammar, (2) the high dependability of the ratings and the resulting placement decisions appropriate for the high-stakes decision-making context based on these analytic rating scales and the composite score, and (3) the largest contribution of Grammar to the composite score variance, which was consistent with the intention of program faculty members to reflect in the test design the relative importance of knowledge of grammar for students' academic success in the study-abroad program.

UR - http://www.scopus.com/inward/record.url?scp=34447286855&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=34447286855&partnerID=8YFLogxK

U2 - 10.1177/0265532207077205

DO - 10.1177/0265532207077205

M3 - Article

VL - 24

SP - 355

EP - 390

JO - Language Testing

JF - Language Testing

SN - 0265-5322

IS - 3

ER -