Fair and accurate assessment method for groupwork by mutual evaluation by interative trust network calculation

Yumeno Shiba, Toshiharu Sugawara

    Research output: Contribution to journalArticle

    Abstract

    We propose a fair and accurate peer assessment method for group work using a multi-agent trust network. Although group work is an effective educational method, accurately assessing individual students is not easy. Mutual evaluation is often used to assess group work because students can observe the contributions of other students. However, mutual evaluation presents some potential problems to discuss such as irresponsible evaluations and collusion. Our proposed method identifies and excludes such cheating and unfair ratings on the basis of trust networks that are often used to evaluate sellers in e-market places by using customers’ ratings. We assume a group-work course in a semester in which students mutually evaluate other group members a few (three to five) times since too many chances for evaluation burden students. We introduce the iterative method for alternately generating trust networks and calculating cluster-trust values, which represent similarity of evaluations in a cluster network. Using a multi-agent simulation, we experimentally show that our method can find the irresponsible students and collusive groups and considerably improve accuracy of final marks with only a few chances for mutual evaluations. Thus, our method can provide useful information for assessments to instructors and reduce free-riders’ incentives for cheating behaviors.

    Original languageEnglish
    JournalTransactions of the Japanese Society for Artificial Intelligence
    Volume31
    Issue number6
    DOIs
    Publication statusPublished - 2016

    Fingerprint

    Students
    Iterative methods

    Keywords

    • Collusion
    • Group work
    • Multi-agent simulation
    • Mutual evaluation
    • Trust network

    ASJC Scopus subject areas

    • Software
    • Artificial Intelligence

    Cite this

    @article{2fa305d59e124aaaa3a130cd8dab9856,
    title = "Fair and accurate assessment method for groupwork by mutual evaluation by interative trust network calculation",
    abstract = "We propose a fair and accurate peer assessment method for group work using a multi-agent trust network. Although group work is an effective educational method, accurately assessing individual students is not easy. Mutual evaluation is often used to assess group work because students can observe the contributions of other students. However, mutual evaluation presents some potential problems to discuss such as irresponsible evaluations and collusion. Our proposed method identifies and excludes such cheating and unfair ratings on the basis of trust networks that are often used to evaluate sellers in e-market places by using customers’ ratings. We assume a group-work course in a semester in which students mutually evaluate other group members a few (three to five) times since too many chances for evaluation burden students. We introduce the iterative method for alternately generating trust networks and calculating cluster-trust values, which represent similarity of evaluations in a cluster network. Using a multi-agent simulation, we experimentally show that our method can find the irresponsible students and collusive groups and considerably improve accuracy of final marks with only a few chances for mutual evaluations. Thus, our method can provide useful information for assessments to instructors and reduce free-riders’ incentives for cheating behaviors.",
    keywords = "Collusion, Group work, Multi-agent simulation, Mutual evaluation, Trust network",
    author = "Yumeno Shiba and Toshiharu Sugawara",
    year = "2016",
    doi = "10.1527/tjsai.AG-C",
    language = "English",
    volume = "31",
    journal = "Transactions of the Japanese Society for Artificial Intelligence",
    issn = "1346-0714",
    publisher = "Japanese Society for Artificial Intelligence",
    number = "6",

    }

    TY - JOUR

    T1 - Fair and accurate assessment method for groupwork by mutual evaluation by interative trust network calculation

    AU - Shiba, Yumeno

    AU - Sugawara, Toshiharu

    PY - 2016

    Y1 - 2016

    N2 - We propose a fair and accurate peer assessment method for group work using a multi-agent trust network. Although group work is an effective educational method, accurately assessing individual students is not easy. Mutual evaluation is often used to assess group work because students can observe the contributions of other students. However, mutual evaluation presents some potential problems to discuss such as irresponsible evaluations and collusion. Our proposed method identifies and excludes such cheating and unfair ratings on the basis of trust networks that are often used to evaluate sellers in e-market places by using customers’ ratings. We assume a group-work course in a semester in which students mutually evaluate other group members a few (three to five) times since too many chances for evaluation burden students. We introduce the iterative method for alternately generating trust networks and calculating cluster-trust values, which represent similarity of evaluations in a cluster network. Using a multi-agent simulation, we experimentally show that our method can find the irresponsible students and collusive groups and considerably improve accuracy of final marks with only a few chances for mutual evaluations. Thus, our method can provide useful information for assessments to instructors and reduce free-riders’ incentives for cheating behaviors.

    AB - We propose a fair and accurate peer assessment method for group work using a multi-agent trust network. Although group work is an effective educational method, accurately assessing individual students is not easy. Mutual evaluation is often used to assess group work because students can observe the contributions of other students. However, mutual evaluation presents some potential problems to discuss such as irresponsible evaluations and collusion. Our proposed method identifies and excludes such cheating and unfair ratings on the basis of trust networks that are often used to evaluate sellers in e-market places by using customers’ ratings. We assume a group-work course in a semester in which students mutually evaluate other group members a few (three to five) times since too many chances for evaluation burden students. We introduce the iterative method for alternately generating trust networks and calculating cluster-trust values, which represent similarity of evaluations in a cluster network. Using a multi-agent simulation, we experimentally show that our method can find the irresponsible students and collusive groups and considerably improve accuracy of final marks with only a few chances for mutual evaluations. Thus, our method can provide useful information for assessments to instructors and reduce free-riders’ incentives for cheating behaviors.

    KW - Collusion

    KW - Group work

    KW - Multi-agent simulation

    KW - Mutual evaluation

    KW - Trust network

    UR - http://www.scopus.com/inward/record.url?scp=84994582572&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=84994582572&partnerID=8YFLogxK

    U2 - 10.1527/tjsai.AG-C

    DO - 10.1527/tjsai.AG-C

    M3 - Article

    VL - 31

    JO - Transactions of the Japanese Society for Artificial Intelligence

    JF - Transactions of the Japanese Society for Artificial Intelligence

    SN - 1346-0714

    IS - 6

    ER -