Q-matrix construction: Defining the link between constructs and test items in large-scale reading and listening comprehension assessments

Yasuyo Sawaki, Hae Jin Kim, Claudia Gentile

    研究成果: Article

    20 引用 (Scopus)

    抄録

    In cognitive diagnosis a Q-matrix (Tatsuoka, 1983, 1990), which is an incidence matrix that defines the relationships between test items and constructs of interest, has great impact on the nature of performance feedback that can be provided to score users. The purpose of the present study was to identify meaningful skill coding categories that reflect core language skills and processes assessed in the Reading and Listening sections of the Test of English as a Foreign LanguageTM Internet-based Test (TOEFL® iBT). The study was conducted as part of a research activity to explore the possibility of developing a detailed score report for low-stakes use by taking a cognitive diagnosis approach. Content experts conducted a test content analysis to develop draft Q-matrices, while measurement experts empirically analyzed the draft Q-matrices with examinee performance data using a cognitively diagnostic psychometric model called the fusion model (DiBello, Stout, & Roussos, 1995; Hartz, 2002). The draft Q-matrices were refined by repeating fusion model analysis and revision of skill definitions and item coding. This resulted in a set of Q-matrices that represented substantively meaningful score reporting categories of a suitable grain size for score reporting, while maintaining an acceptable level of examinee classification consistency.

    元の言語English
    ページ(範囲)190-209
    ページ数20
    ジャーナルLanguage Assessment Quarterly
    6
    発行部数3
    DOI
    出版物ステータスPublished - 2009 7

    Fingerprint

    listening comprehension
    coding
    expert
    model analysis
    psychometrics
    performance
    agricultural product
    content analysis
    diagnostic
    incidence
    Internet
    Listening Comprehension
    Reading Comprehension
    language
    Draft

    ASJC Scopus subject areas

    • Language and Linguistics
    • Linguistics and Language

    これを引用

    @article{ee30b322604c4d59b8e5fd38865f8ddb,
    title = "Q-matrix construction: Defining the link between constructs and test items in large-scale reading and listening comprehension assessments",
    abstract = "In cognitive diagnosis a Q-matrix (Tatsuoka, 1983, 1990), which is an incidence matrix that defines the relationships between test items and constructs of interest, has great impact on the nature of performance feedback that can be provided to score users. The purpose of the present study was to identify meaningful skill coding categories that reflect core language skills and processes assessed in the Reading and Listening sections of the Test of English as a Foreign LanguageTM Internet-based Test (TOEFL{\circledR} iBT). The study was conducted as part of a research activity to explore the possibility of developing a detailed score report for low-stakes use by taking a cognitive diagnosis approach. Content experts conducted a test content analysis to develop draft Q-matrices, while measurement experts empirically analyzed the draft Q-matrices with examinee performance data using a cognitively diagnostic psychometric model called the fusion model (DiBello, Stout, & Roussos, 1995; Hartz, 2002). The draft Q-matrices were refined by repeating fusion model analysis and revision of skill definitions and item coding. This resulted in a set of Q-matrices that represented substantively meaningful score reporting categories of a suitable grain size for score reporting, while maintaining an acceptable level of examinee classification consistency.",
    author = "Yasuyo Sawaki and Kim, {Hae Jin} and Claudia Gentile",
    year = "2009",
    month = "7",
    doi = "10.1080/15434300902801917",
    language = "English",
    volume = "6",
    pages = "190--209",
    journal = "Language Assessment Quarterly",
    issn = "1543-4303",
    publisher = "Routledge",
    number = "3",

    }

    TY - JOUR

    T1 - Q-matrix construction

    T2 - Defining the link between constructs and test items in large-scale reading and listening comprehension assessments

    AU - Sawaki, Yasuyo

    AU - Kim, Hae Jin

    AU - Gentile, Claudia

    PY - 2009/7

    Y1 - 2009/7

    N2 - In cognitive diagnosis a Q-matrix (Tatsuoka, 1983, 1990), which is an incidence matrix that defines the relationships between test items and constructs of interest, has great impact on the nature of performance feedback that can be provided to score users. The purpose of the present study was to identify meaningful skill coding categories that reflect core language skills and processes assessed in the Reading and Listening sections of the Test of English as a Foreign LanguageTM Internet-based Test (TOEFL® iBT). The study was conducted as part of a research activity to explore the possibility of developing a detailed score report for low-stakes use by taking a cognitive diagnosis approach. Content experts conducted a test content analysis to develop draft Q-matrices, while measurement experts empirically analyzed the draft Q-matrices with examinee performance data using a cognitively diagnostic psychometric model called the fusion model (DiBello, Stout, & Roussos, 1995; Hartz, 2002). The draft Q-matrices were refined by repeating fusion model analysis and revision of skill definitions and item coding. This resulted in a set of Q-matrices that represented substantively meaningful score reporting categories of a suitable grain size for score reporting, while maintaining an acceptable level of examinee classification consistency.

    AB - In cognitive diagnosis a Q-matrix (Tatsuoka, 1983, 1990), which is an incidence matrix that defines the relationships between test items and constructs of interest, has great impact on the nature of performance feedback that can be provided to score users. The purpose of the present study was to identify meaningful skill coding categories that reflect core language skills and processes assessed in the Reading and Listening sections of the Test of English as a Foreign LanguageTM Internet-based Test (TOEFL® iBT). The study was conducted as part of a research activity to explore the possibility of developing a detailed score report for low-stakes use by taking a cognitive diagnosis approach. Content experts conducted a test content analysis to develop draft Q-matrices, while measurement experts empirically analyzed the draft Q-matrices with examinee performance data using a cognitively diagnostic psychometric model called the fusion model (DiBello, Stout, & Roussos, 1995; Hartz, 2002). The draft Q-matrices were refined by repeating fusion model analysis and revision of skill definitions and item coding. This resulted in a set of Q-matrices that represented substantively meaningful score reporting categories of a suitable grain size for score reporting, while maintaining an acceptable level of examinee classification consistency.

    UR - http://www.scopus.com/inward/record.url?scp=70849116692&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=70849116692&partnerID=8YFLogxK

    U2 - 10.1080/15434300902801917

    DO - 10.1080/15434300902801917

    M3 - Article

    AN - SCOPUS:70849116692

    VL - 6

    SP - 190

    EP - 209

    JO - Language Assessment Quarterly

    JF - Language Assessment Quarterly

    SN - 1543-4303

    IS - 3

    ER -