Understanding learner strengths and weaknesses: Assessing performance on an integrated writing task

Yasuyo Sawaki, Thomas Quinlan, Yong Won Lee

    Research output: Contribution to journalArticle

    16 Citations (Scopus)

    Abstract

    The present study examined the factor structures across features of 446 examinees' responses to a writing task that integrates reading and listening modalities as well as reading and listening comprehension items of the TOEFL iBT® (Internet-based test). Both human and automated scores obtained for the integrated essays were utilized. Based on a series of preliminary factor analyses, a confirmatory factor analysis (CFA) identified a model that specified a higher order factor for comprehension. In the model, the Comprehension factor underlay factors representing content of the written essay as well as reading and listening comprehension. The Comprehension factor correlated with two writing factors-Productive Vocabulary and Sentence Conventions. Furthermore, follow-up CFA models with covariates (multiple indicators multiple causes models, or MIMIC models) were tested to compare performance between a group of 190 examinees scoring above a frequently used TOEFL iBT Total score requirement for international student admission and the other group of 128 examinees scoring below the requirement. The higher ability group performed significantly better than the lower ability group on all three constructs: Comprehension, Productive Vocabulary, and Sentence Conventions. The identification of the multiple distinct factors in this study may hold promise for obtaining writing profiles that inform instruction in contexts such as test preparation.

    Original languageEnglish
    Pages (from-to)73-95
    Number of pages23
    JournalLanguage Assessment Quarterly
    Volume10
    Issue number1
    DOIs
    Publication statusPublished - 2013 Jan 1

    Fingerprint

    comprehension
    listening comprehension
    performance
    factor analysis
    vocabulary
    Group
    Internet
    ability
    instruction
    cause
    student
    World Wide Web
    Scoring
    Confirmatory Factor Analysis
    Vocabulary
    Listening Comprehension
    Reading Comprehension
    TOEFL

    ASJC Scopus subject areas

    • Language and Linguistics
    • Linguistics and Language

    Cite this

    Understanding learner strengths and weaknesses : Assessing performance on an integrated writing task. / Sawaki, Yasuyo; Quinlan, Thomas; Lee, Yong Won.

    In: Language Assessment Quarterly, Vol. 10, No. 1, 01.01.2013, p. 73-95.

    Research output: Contribution to journalArticle

    @article{8b70ab1a6d334889a07ba474e53dd281,
    title = "Understanding learner strengths and weaknesses: Assessing performance on an integrated writing task",
    abstract = "The present study examined the factor structures across features of 446 examinees' responses to a writing task that integrates reading and listening modalities as well as reading and listening comprehension items of the TOEFL iBT{\circledR} (Internet-based test). Both human and automated scores obtained for the integrated essays were utilized. Based on a series of preliminary factor analyses, a confirmatory factor analysis (CFA) identified a model that specified a higher order factor for comprehension. In the model, the Comprehension factor underlay factors representing content of the written essay as well as reading and listening comprehension. The Comprehension factor correlated with two writing factors-Productive Vocabulary and Sentence Conventions. Furthermore, follow-up CFA models with covariates (multiple indicators multiple causes models, or MIMIC models) were tested to compare performance between a group of 190 examinees scoring above a frequently used TOEFL iBT Total score requirement for international student admission and the other group of 128 examinees scoring below the requirement. The higher ability group performed significantly better than the lower ability group on all three constructs: Comprehension, Productive Vocabulary, and Sentence Conventions. The identification of the multiple distinct factors in this study may hold promise for obtaining writing profiles that inform instruction in contexts such as test preparation.",
    author = "Yasuyo Sawaki and Thomas Quinlan and Lee, {Yong Won}",
    year = "2013",
    month = "1",
    day = "1",
    doi = "10.1080/15434303.2011.633305",
    language = "English",
    volume = "10",
    pages = "73--95",
    journal = "Language Assessment Quarterly",
    issn = "1543-4303",
    publisher = "Routledge",
    number = "1",

    }

    TY - JOUR

    T1 - Understanding learner strengths and weaknesses

    T2 - Assessing performance on an integrated writing task

    AU - Sawaki, Yasuyo

    AU - Quinlan, Thomas

    AU - Lee, Yong Won

    PY - 2013/1/1

    Y1 - 2013/1/1

    N2 - The present study examined the factor structures across features of 446 examinees' responses to a writing task that integrates reading and listening modalities as well as reading and listening comprehension items of the TOEFL iBT® (Internet-based test). Both human and automated scores obtained for the integrated essays were utilized. Based on a series of preliminary factor analyses, a confirmatory factor analysis (CFA) identified a model that specified a higher order factor for comprehension. In the model, the Comprehension factor underlay factors representing content of the written essay as well as reading and listening comprehension. The Comprehension factor correlated with two writing factors-Productive Vocabulary and Sentence Conventions. Furthermore, follow-up CFA models with covariates (multiple indicators multiple causes models, or MIMIC models) were tested to compare performance between a group of 190 examinees scoring above a frequently used TOEFL iBT Total score requirement for international student admission and the other group of 128 examinees scoring below the requirement. The higher ability group performed significantly better than the lower ability group on all three constructs: Comprehension, Productive Vocabulary, and Sentence Conventions. The identification of the multiple distinct factors in this study may hold promise for obtaining writing profiles that inform instruction in contexts such as test preparation.

    AB - The present study examined the factor structures across features of 446 examinees' responses to a writing task that integrates reading and listening modalities as well as reading and listening comprehension items of the TOEFL iBT® (Internet-based test). Both human and automated scores obtained for the integrated essays were utilized. Based on a series of preliminary factor analyses, a confirmatory factor analysis (CFA) identified a model that specified a higher order factor for comprehension. In the model, the Comprehension factor underlay factors representing content of the written essay as well as reading and listening comprehension. The Comprehension factor correlated with two writing factors-Productive Vocabulary and Sentence Conventions. Furthermore, follow-up CFA models with covariates (multiple indicators multiple causes models, or MIMIC models) were tested to compare performance between a group of 190 examinees scoring above a frequently used TOEFL iBT Total score requirement for international student admission and the other group of 128 examinees scoring below the requirement. The higher ability group performed significantly better than the lower ability group on all three constructs: Comprehension, Productive Vocabulary, and Sentence Conventions. The identification of the multiple distinct factors in this study may hold promise for obtaining writing profiles that inform instruction in contexts such as test preparation.

    UR - http://www.scopus.com/inward/record.url?scp=84874482549&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=84874482549&partnerID=8YFLogxK

    U2 - 10.1080/15434303.2011.633305

    DO - 10.1080/15434303.2011.633305

    M3 - Article

    AN - SCOPUS:84874482549

    VL - 10

    SP - 73

    EP - 95

    JO - Language Assessment Quarterly

    JF - Language Assessment Quarterly

    SN - 1543-4303

    IS - 1

    ER -