TurkScanner

Predicting the hourly wage of microtasks

Susumu Saito, Teppei Nakano, Chun Wei Chiang, Tetsunori Kobayashi, Saiph Savage, Jeffrey P. Bigham

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Workers in crowd markets struggle to earn a living. One reason for this is that it is difficult for workers to accurately gauge the hourly wages of microtasks, and they consequently end up performing labor with little pay. In general, workers are provided with little information about tasks, and are left to rely on noisy signals, such as textual description of the task or rating of the requester. This study explores various computational methods for predicting the working times (and thus hourly wages) required for tasks based on data collected from other workers completing crowd work. We provide the following contributions. (i) A data collection method for gathering real-world training data on crowd-work tasks and the times required for workers to complete them; (ii) TurkScanner: a machine learning approach that predicts the necessary working time to complete a task (and can thus implicitly provide the expected hourly wage). We collected 9,155 data records using a web browser extension installed by 84 Amazon Mechanical Turk workers, and explored the challenge of accurately recording working times both automatically and by asking workers. TurkScanner was created using ∼150 derived features, and was able to predict the hourly wages of 69.6% of all the tested microtasks within a 75% error. Directions for future research include observing the effects of tools on people's working practices, adapting this approach to a requester tool for better price setting, and predicting other elements of work (e.g., the acceptance likelihood and worker task preferences).

Original languageEnglish
Title of host publicationThe Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019
PublisherAssociation for Computing Machinery, Inc
Pages3187-3193
Number of pages7
ISBN (Electronic)9781450366748
DOIs
Publication statusPublished - 2019 May 13
Event2019 World Wide Web Conference, WWW 2019 - San Francisco, United States
Duration: 2019 May 132019 May 17

Publication series

NameThe Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019

Conference

Conference2019 World Wide Web Conference, WWW 2019
CountryUnited States
CitySan Francisco
Period19/5/1319/5/17

Fingerprint

Wages
Web browsers
Computational methods
Gages
Learning systems
Personnel

Keywords

  • Amazon Mechanical Turk
  • Crowdsourcing
  • Hourly wage

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Software

Cite this

Saito, S., Nakano, T., Chiang, C. W., Kobayashi, T., Savage, S., & Bigham, J. P. (2019). TurkScanner: Predicting the hourly wage of microtasks. In The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019 (pp. 3187-3193). (The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019). Association for Computing Machinery, Inc. https://doi.org/10.1145/3308558.3313716

TurkScanner : Predicting the hourly wage of microtasks. / Saito, Susumu; Nakano, Teppei; Chiang, Chun Wei; Kobayashi, Tetsunori; Savage, Saiph; Bigham, Jeffrey P.

The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019. Association for Computing Machinery, Inc, 2019. p. 3187-3193 (The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Saito, S, Nakano, T, Chiang, CW, Kobayashi, T, Savage, S & Bigham, JP 2019, TurkScanner: Predicting the hourly wage of microtasks. in The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019. The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019, Association for Computing Machinery, Inc, pp. 3187-3193, 2019 World Wide Web Conference, WWW 2019, San Francisco, United States, 19/5/13. https://doi.org/10.1145/3308558.3313716
Saito S, Nakano T, Chiang CW, Kobayashi T, Savage S, Bigham JP. TurkScanner: Predicting the hourly wage of microtasks. In The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019. Association for Computing Machinery, Inc. 2019. p. 3187-3193. (The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019). https://doi.org/10.1145/3308558.3313716
Saito, Susumu ; Nakano, Teppei ; Chiang, Chun Wei ; Kobayashi, Tetsunori ; Savage, Saiph ; Bigham, Jeffrey P. / TurkScanner : Predicting the hourly wage of microtasks. The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019. Association for Computing Machinery, Inc, 2019. pp. 3187-3193 (The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019).
@inproceedings{d755a7630c0a4f56a5523b15920016c0,
title = "TurkScanner: Predicting the hourly wage of microtasks",
abstract = "Workers in crowd markets struggle to earn a living. One reason for this is that it is difficult for workers to accurately gauge the hourly wages of microtasks, and they consequently end up performing labor with little pay. In general, workers are provided with little information about tasks, and are left to rely on noisy signals, such as textual description of the task or rating of the requester. This study explores various computational methods for predicting the working times (and thus hourly wages) required for tasks based on data collected from other workers completing crowd work. We provide the following contributions. (i) A data collection method for gathering real-world training data on crowd-work tasks and the times required for workers to complete them; (ii) TurkScanner: a machine learning approach that predicts the necessary working time to complete a task (and can thus implicitly provide the expected hourly wage). We collected 9,155 data records using a web browser extension installed by 84 Amazon Mechanical Turk workers, and explored the challenge of accurately recording working times both automatically and by asking workers. TurkScanner was created using ∼150 derived features, and was able to predict the hourly wages of 69.6{\%} of all the tested microtasks within a 75{\%} error. Directions for future research include observing the effects of tools on people's working practices, adapting this approach to a requester tool for better price setting, and predicting other elements of work (e.g., the acceptance likelihood and worker task preferences).",
keywords = "Amazon Mechanical Turk, Crowdsourcing, Hourly wage",
author = "Susumu Saito and Teppei Nakano and Chiang, {Chun Wei} and Tetsunori Kobayashi and Saiph Savage and Bigham, {Jeffrey P.}",
year = "2019",
month = "5",
day = "13",
doi = "10.1145/3308558.3313716",
language = "English",
series = "The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019",
publisher = "Association for Computing Machinery, Inc",
pages = "3187--3193",
booktitle = "The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019",

}

TY - GEN

T1 - TurkScanner

T2 - Predicting the hourly wage of microtasks

AU - Saito, Susumu

AU - Nakano, Teppei

AU - Chiang, Chun Wei

AU - Kobayashi, Tetsunori

AU - Savage, Saiph

AU - Bigham, Jeffrey P.

PY - 2019/5/13

Y1 - 2019/5/13

N2 - Workers in crowd markets struggle to earn a living. One reason for this is that it is difficult for workers to accurately gauge the hourly wages of microtasks, and they consequently end up performing labor with little pay. In general, workers are provided with little information about tasks, and are left to rely on noisy signals, such as textual description of the task or rating of the requester. This study explores various computational methods for predicting the working times (and thus hourly wages) required for tasks based on data collected from other workers completing crowd work. We provide the following contributions. (i) A data collection method for gathering real-world training data on crowd-work tasks and the times required for workers to complete them; (ii) TurkScanner: a machine learning approach that predicts the necessary working time to complete a task (and can thus implicitly provide the expected hourly wage). We collected 9,155 data records using a web browser extension installed by 84 Amazon Mechanical Turk workers, and explored the challenge of accurately recording working times both automatically and by asking workers. TurkScanner was created using ∼150 derived features, and was able to predict the hourly wages of 69.6% of all the tested microtasks within a 75% error. Directions for future research include observing the effects of tools on people's working practices, adapting this approach to a requester tool for better price setting, and predicting other elements of work (e.g., the acceptance likelihood and worker task preferences).

AB - Workers in crowd markets struggle to earn a living. One reason for this is that it is difficult for workers to accurately gauge the hourly wages of microtasks, and they consequently end up performing labor with little pay. In general, workers are provided with little information about tasks, and are left to rely on noisy signals, such as textual description of the task or rating of the requester. This study explores various computational methods for predicting the working times (and thus hourly wages) required for tasks based on data collected from other workers completing crowd work. We provide the following contributions. (i) A data collection method for gathering real-world training data on crowd-work tasks and the times required for workers to complete them; (ii) TurkScanner: a machine learning approach that predicts the necessary working time to complete a task (and can thus implicitly provide the expected hourly wage). We collected 9,155 data records using a web browser extension installed by 84 Amazon Mechanical Turk workers, and explored the challenge of accurately recording working times both automatically and by asking workers. TurkScanner was created using ∼150 derived features, and was able to predict the hourly wages of 69.6% of all the tested microtasks within a 75% error. Directions for future research include observing the effects of tools on people's working practices, adapting this approach to a requester tool for better price setting, and predicting other elements of work (e.g., the acceptance likelihood and worker task preferences).

KW - Amazon Mechanical Turk

KW - Crowdsourcing

KW - Hourly wage

UR - http://www.scopus.com/inward/record.url?scp=85066912657&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85066912657&partnerID=8YFLogxK

U2 - 10.1145/3308558.3313716

DO - 10.1145/3308558.3313716

M3 - Conference contribution

T3 - The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019

SP - 3187

EP - 3193

BT - The Web Conference 2019 - Proceedings of the World Wide Web Conference, WWW 2019

PB - Association for Computing Machinery, Inc

ER -