TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences

There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

Abstract

In recent years, Mechanical Turk (MTurk) has revolutionized social science by providing a way to collect behavioral data with unprecedented speed and efficiency. However, MTurk was not intended to be a research tool, and many common research tasks are difficult and time-consuming to implement as a result. TurkPrime was designed as a research platform that integrates with MTurk and supports tasks that are common to the social and behavioral sciences. Like MTurk, TurkPrime is an Internet-based platform that runs on any browser and does not require any downloads or installation. Tasks that can be implemented with TurkPrime include: excluding participants on the basis of previous participation, longitudinal studies, making changes to a study while it is running, automating the approval process, increasing the speed of data collection, sending bulk e-mails and bonuses, enhancing communication with participants, monitoring dropout and engagement rates, providing enhanced sampling options, and many others. This article describes how TurkPrime saves time and resources, improves data quality, and allows researchers to design and implement studies that were previously very difficult or impossible to carry out on MTurk. TurkPrime is designed as a research tool whose aim is to improve the quality of the crowdsourcing data collection process. Various features have been and continue to be implemented on the basis of feedback from the research community. TurkPrime is a free research platform.

Related collections

Most cited references 19

Record: found
Abstract: found
Article: not found

The weirdest people in the world?

Joseph Henrich, Steven Heine, Ara Norenzayan (2010)

Behavioral scientists routinely publish broad claims about human psychology and behavior in the world's top journals based on samples drawn entirely from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies. Researchers - often implicitly - assume that either there is little variation across human populations, or that these "standard subjects" are as representative of the species as any other population. Are these assumptions justified? Here, our review of the comparative database from across the behavioral sciences suggests both that there is substantial variability in experimental results across populations and that WEIRD subjects are particularly unusual compared with the rest of the species - frequent outliers. The domains reviewed include visual perception, fairness, cooperation, spatial reasoning, categorization and inferential induction, moral reasoning, reasoning styles, self-concepts and related motivations, and the heritability of IQ. The findings suggest that members of WEIRD societies, including young children, are among the least representative populations one could find for generalizing about humans. Many of these findings involve domains that are associated with fundamental aspects of psychology, motivation, and behavior - hence, there are no obvious a priori grounds for claiming that a particular behavioral phenomenon is universal based on sampling from a single subpopulation. Overall, these empirical patterns suggests that we need to be less cavalier in addressing questions of human nature on the basis of data drawn from this particularly thin, and rather unusual, slice of humanity. We close by proposing ways to structurally re-organize the behavioral sciences to best tackle these challenges.

0 comments Cited 1519 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

Improving the Quality of Web Surveys: The Checklist for Reporting Results of Internet E-Surveys (CHERRIES)

Gunther Eysenbach (2004)

Analogous to checklists of recommendations such as the CONSORT statement (for randomized trials), or the QUORUM statement (for systematic reviews), which are designed to ensure the quality of reports in the medical literature, a checklist of recommendations for authors is being presented by the Journal of Medical Internet Research (JMIR) in an effort to ensure complete descriptions of Web-based surveys. Papers on Web-based surveys reported according to the CHERRIES statement will give readers a better understanding of the sample (self-)selection and its possible differences from a “representative” sample. It is hoped that author adherence to the checklist will increase the usefulness of such reports.

0 comments Cited 1494 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?

Michael Buhrmester, Tracy Kwang, Samuel D. Gosling (2011)

Amazon's Mechanical Turk (MTurk) is a relatively new website that contains the major elements required to conduct research: an integrated participant compensation system; a large participant pool; and a streamlined process of study design, participant recruitment, and data collection. In this article, we describe and evaluate the potential contributions of MTurk to psychology and other social sciences. Findings indicate that (a) MTurk participants are slightly more demographically diverse than are standard Internet samples and are significantly more diverse than typical American college samples; (b) participation is affected by compensation rate and task length, but participants can still be recruited rapidly and inexpensively; (c) realistic compensation rates do not affect data quality; and (d) the data obtained are at least as reliable as those obtained via traditional methods. Overall, MTurk can be used to obtain high-quality data inexpensively and rapidly.

0 comments Cited 998 times – based on 0 reviews      Review now

Bookmark

All references

Author and article information

Contributors

Leib Litman: leib.litman@touro.edu

Journal

Journal ID (nlm-ta): Behav Res Methods

Journal ID (iso-abbrev): Behav Res Methods

Title: Behavior Research Methods

Publisher: Springer US (New York )

ISSN (Print): 1554-351X

ISSN (Electronic): 1554-3528

Publication date (Electronic): 12 April 2016

Publication date PMC-release: 12 April 2016

Publication date (Print): 2017

Volume: 49

Issue: 2

Pages: 433-442

Affiliations

[1 ]Lander College, Flushing, New York USA

[2 ]Department of Psychology, Lander College, 75-31 150th Street, Flushing, New York 11367 USA

Article

Publisher ID: 727

DOI: 10.3758/s13428-016-0727-z

PMC ID: 5405057

PubMed ID: 27071389

SO-VID: 4308a2c0-2234-493d-ad31-db995dcd679c

License:

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.