Conducting laboratory experiments properly with statistical tools: An easy hands-on tutorial

Tetsuya Sakai

doi:10.1145/3209978.3210182

Conducting laboratory experiments properly with statistical tools: An easy hands-on tutorial

Tetsuya Sakai

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

3 Citations (Scopus)

Abstract

This hands-on half-day tutorial consists of two 90-minute sessions. Part I covers the following topics: paired and two-sample t -tests, confidence intervals (with Excel and R); familywise error rate, multiple comparison procedures; ANOVA (with Excel and R); Tukey's HSD test, simultaneous confidence intervals (with R). Part II covers the following topics: randomised Tukey HSD test (with Discpower); what's wrong with statistical significance tests?; effect sizes, statistical power; topic set size design (with Excel); power analysis (with R); summary: how to report your results. Participants should have some prior knowledge about the very basics of statistical significance testing and are strongly encouraged to bring a laptop with R already installed. The tutorial participants will be able to design and conduct statistical significance tests for comparing the mean effectiveness scores of two or more systems appropriately, and to report on the test results in an informative manner.

Original language	English
Title of host publication	41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018
Publisher	Association for Computing Machinery, Inc
Pages	1369-1370
Number of pages	2
ISBN (Electronic)	9781450356572
DOIs	https://doi.org/10.1145/3209978.3210182
Publication status	Published - 2018 Jun 27
Event	41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018 - Ann Arbor, United States Duration: 2018 Jul 8 → 2018 Jul 12

Other

Other	41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018
Country	United States
City	Ann Arbor
Period	18/7/8 → 18/7/12

Fingerprint

Keywords

Confidence intervals
Effect sizes
Multiple comparison procedures
Randomisation test
Sample sizes
Statistical power
Statistical significance
T-test
Tukey's honestly significant difference test

ASJC Scopus subject areas

Software
Computer Graphics and Computer-Aided Design
Information Systems

Cite this

Sakai, T. (2018). Conducting laboratory experiments properly with statistical tools: An easy hands-on tutorial. In 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018 (pp. 1369-1370). Association for Computing Machinery, Inc. https://doi.org/10.1145/3209978.3210182

Sakai, T 2018, Conducting laboratory experiments properly with statistical tools: An easy hands-on tutorial. in 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018. Association for Computing Machinery, Inc, pp. 1369-1370, 41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018, Ann Arbor, United States, 18/7/8. https://doi.org/10.1145/3209978.3210182

@inproceedings{0cbeafe1fadc44e89e167419d633994a,

title = "Conducting laboratory experiments properly with statistical tools: An easy hands-on tutorial",

abstract = "This hands-on half-day tutorial consists of two 90-minute sessions. Part I covers the following topics: paired and two-sample t -tests, confidence intervals (with Excel and R); familywise error rate, multiple comparison procedures; ANOVA (with Excel and R); Tukey's HSD test, simultaneous confidence intervals (with R). Part II covers the following topics: randomised Tukey HSD test (with Discpower); what's wrong with statistical significance tests?; effect sizes, statistical power; topic set size design (with Excel); power analysis (with R); summary: how to report your results. Participants should have some prior knowledge about the very basics of statistical significance testing and are strongly encouraged to bring a laptop with R already installed. The tutorial participants will be able to design and conduct statistical significance tests for comparing the mean effectiveness scores of two or more systems appropriately, and to report on the test results in an informative manner.",

keywords = "Confidence intervals, Effect sizes, Multiple comparison procedures, Randomisation test, Sample sizes, Statistical power, Statistical significance, T-test, Tukey's honestly significant difference test",

author = "Tetsuya Sakai",

year = "2018",

month = jun

day = "27",

doi = "10.1145/3209978.3210182",

language = "English",

pages = "1369--1370",

booktitle = "41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018",

publisher = "Association for Computing Machinery, Inc",

note = "41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018 ; Conference date: 08-07-2018 Through 12-07-2018",

}

TY - GEN

T1 - Conducting laboratory experiments properly with statistical tools

T2 - 41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018

AU - Sakai, Tetsuya

PY - 2018/6/27

Y1 - 2018/6/27

N2 - This hands-on half-day tutorial consists of two 90-minute sessions. Part I covers the following topics: paired and two-sample t -tests, confidence intervals (with Excel and R); familywise error rate, multiple comparison procedures; ANOVA (with Excel and R); Tukey's HSD test, simultaneous confidence intervals (with R). Part II covers the following topics: randomised Tukey HSD test (with Discpower); what's wrong with statistical significance tests?; effect sizes, statistical power; topic set size design (with Excel); power analysis (with R); summary: how to report your results. Participants should have some prior knowledge about the very basics of statistical significance testing and are strongly encouraged to bring a laptop with R already installed. The tutorial participants will be able to design and conduct statistical significance tests for comparing the mean effectiveness scores of two or more systems appropriately, and to report on the test results in an informative manner.

AB - This hands-on half-day tutorial consists of two 90-minute sessions. Part I covers the following topics: paired and two-sample t -tests, confidence intervals (with Excel and R); familywise error rate, multiple comparison procedures; ANOVA (with Excel and R); Tukey's HSD test, simultaneous confidence intervals (with R). Part II covers the following topics: randomised Tukey HSD test (with Discpower); what's wrong with statistical significance tests?; effect sizes, statistical power; topic set size design (with Excel); power analysis (with R); summary: how to report your results. Participants should have some prior knowledge about the very basics of statistical significance testing and are strongly encouraged to bring a laptop with R already installed. The tutorial participants will be able to design and conduct statistical significance tests for comparing the mean effectiveness scores of two or more systems appropriately, and to report on the test results in an informative manner.

KW - Confidence intervals

KW - Effect sizes

KW - Multiple comparison procedures

KW - Randomisation test

KW - Sample sizes

KW - Statistical power

KW - Statistical significance

KW - T-test

KW - Tukey's honestly significant difference test

UR - http://www.scopus.com/inward/record.url?scp=85051473247&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85051473247&partnerID=8YFLogxK

U2 - 10.1145/3209978.3210182

DO - 10.1145/3209978.3210182

M3 - Conference contribution

AN - SCOPUS:85051473247

SP - 1369

EP - 1370

BT - 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018

PB - Association for Computing Machinery, Inc

Y2 - 8 July 2018 through 12 July 2018

ER -

Access to Document

10.1145/3209978.3210182