Application of three cognitive diagnosis models to ESL reading and listening assessments

Yong Won Lee, Yasuyo Sawaki

    Research output: Contribution to journalArticle

    31 Citations (Scopus)

    Abstract

    The present study investigated the functioning of three psychometric models for cognitive diagnosis - the general diagnostic model, the fusion model, and latent class analysis - when applied to large-scale English as a second language listening and reading comprehension assessments. Data used in this study were scored item responses and incidence matrices, or Q-matrices (Tatsuoka, 1983, 1990), for the Listening and Reading sections of two field test forms of the TOEFL® iBT (Test of English as a Foreign Language™). Results of empirical data analyses based on the three cognitive diagnosis models above revealed that (a) all three models adequately separated examinees into two mastery levels on most of the reading and listening skills, (b) a moderate level of across-form consistency of examinee skill mastery classification was achieved for most of the reading and listening skills as defined in this study, but (c) a large proportion of examinees were classified into "flat" profiles (i.e., "nonmasters of all skills" and "masters of all skills"). Overall, the three models produced similar results in terms of examinee classification, but some subtle differences across them were identified as well.

    Original languageEnglish
    Pages (from-to)239-263
    Number of pages25
    JournalLanguage Assessment Quarterly
    Volume6
    Issue number3
    DOIs
    Publication statusPublished - 2009 Jul

    Fingerprint

    psychometrics
    foreign language
    diagnostic
    comprehension
    incidence
    language
    Mastery
    Empirical Data
    English as a Foreign Language
    Fusion
    Listening Comprehension
    Reading Comprehension
    Psychometrics
    English as a Second Language
    TOEFL
    Diagnostics
    Second Language Listening
    Latent Class Analysis
    Proportion

    ASJC Scopus subject areas

    • Language and Linguistics
    • Linguistics and Language

    Cite this

    Application of three cognitive diagnosis models to ESL reading and listening assessments. / Lee, Yong Won; Sawaki, Yasuyo.

    In: Language Assessment Quarterly, Vol. 6, No. 3, 07.2009, p. 239-263.

    Research output: Contribution to journalArticle

    @article{80b44f4a6a6e4ddaa913f3919271d7d8,
    title = "Application of three cognitive diagnosis models to ESL reading and listening assessments",
    abstract = "The present study investigated the functioning of three psychometric models for cognitive diagnosis - the general diagnostic model, the fusion model, and latent class analysis - when applied to large-scale English as a second language listening and reading comprehension assessments. Data used in this study were scored item responses and incidence matrices, or Q-matrices (Tatsuoka, 1983, 1990), for the Listening and Reading sections of two field test forms of the TOEFL{\circledR} iBT (Test of English as a Foreign Language™). Results of empirical data analyses based on the three cognitive diagnosis models above revealed that (a) all three models adequately separated examinees into two mastery levels on most of the reading and listening skills, (b) a moderate level of across-form consistency of examinee skill mastery classification was achieved for most of the reading and listening skills as defined in this study, but (c) a large proportion of examinees were classified into {"}flat{"} profiles (i.e., {"}nonmasters of all skills{"} and {"}masters of all skills{"}). Overall, the three models produced similar results in terms of examinee classification, but some subtle differences across them were identified as well.",
    author = "Lee, {Yong Won} and Yasuyo Sawaki",
    year = "2009",
    month = "7",
    doi = "10.1080/15434300903079562",
    language = "English",
    volume = "6",
    pages = "239--263",
    journal = "Language Assessment Quarterly",
    issn = "1543-4303",
    publisher = "Routledge",
    number = "3",

    }

    TY - JOUR

    T1 - Application of three cognitive diagnosis models to ESL reading and listening assessments

    AU - Lee, Yong Won

    AU - Sawaki, Yasuyo

    PY - 2009/7

    Y1 - 2009/7

    N2 - The present study investigated the functioning of three psychometric models for cognitive diagnosis - the general diagnostic model, the fusion model, and latent class analysis - when applied to large-scale English as a second language listening and reading comprehension assessments. Data used in this study were scored item responses and incidence matrices, or Q-matrices (Tatsuoka, 1983, 1990), for the Listening and Reading sections of two field test forms of the TOEFL® iBT (Test of English as a Foreign Language™). Results of empirical data analyses based on the three cognitive diagnosis models above revealed that (a) all three models adequately separated examinees into two mastery levels on most of the reading and listening skills, (b) a moderate level of across-form consistency of examinee skill mastery classification was achieved for most of the reading and listening skills as defined in this study, but (c) a large proportion of examinees were classified into "flat" profiles (i.e., "nonmasters of all skills" and "masters of all skills"). Overall, the three models produced similar results in terms of examinee classification, but some subtle differences across them were identified as well.

    AB - The present study investigated the functioning of three psychometric models for cognitive diagnosis - the general diagnostic model, the fusion model, and latent class analysis - when applied to large-scale English as a second language listening and reading comprehension assessments. Data used in this study were scored item responses and incidence matrices, or Q-matrices (Tatsuoka, 1983, 1990), for the Listening and Reading sections of two field test forms of the TOEFL® iBT (Test of English as a Foreign Language™). Results of empirical data analyses based on the three cognitive diagnosis models above revealed that (a) all three models adequately separated examinees into two mastery levels on most of the reading and listening skills, (b) a moderate level of across-form consistency of examinee skill mastery classification was achieved for most of the reading and listening skills as defined in this study, but (c) a large proportion of examinees were classified into "flat" profiles (i.e., "nonmasters of all skills" and "masters of all skills"). Overall, the three models produced similar results in terms of examinee classification, but some subtle differences across them were identified as well.

    UR - http://www.scopus.com/inward/record.url?scp=70849114979&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=70849114979&partnerID=8YFLogxK

    U2 - 10.1080/15434300903079562

    DO - 10.1080/15434300903079562

    M3 - Article

    AN - SCOPUS:70849114979

    VL - 6

    SP - 239

    EP - 263

    JO - Language Assessment Quarterly

    JF - Language Assessment Quarterly

    SN - 1543-4303

    IS - 3

    ER -