Studying MTurk Workers with I-O Psychology

Author(s): Alice M. Brawley Newlin, Ph.D.

Originally posted: September 17, 2016

More and more, we’re hearing about the gig economy, which is vastly different from a “traditional” job that’s with one employer for quite some time. Part of the gig economy is crowdsourcing, which occurs on a number of sites, including Amazon’s Mechanical Turk (MTurk). A lot of researchers have been exploring this type of work from the viewpoints of several different fields, like human computation.

What’s surprising is that the field of research that studies work experiences – my field, I-O psychology – hasn’t explored this type of work very much yet. So we set out to study this type of common (and only growing) type of work experience in a recently published article. (Actually, this work grew out of making a few mistakes in my first days as a Requester! Don’t forget about the people who are doing the work – see my thoughts for starting out on MTurk here.)

Did I-O Psychology Findings Hold Up in the MTurk Setting?

We asked Workers to tell us about their good and bad experiences on MTurk, and tested how well some of our traditional I-O psychology predictions held up. In short, some findings did hold up – for example, the happier a Worker is with a Requester, the less likely they are to refuse to work for that Requester later on; but many findings didn’t apply to this setting – for example, many traditional predictors of job satisfaction, such as personality variables and job characteristics, didn’t significantly affect job satisfaction.

We also had some surprising findings, like one variable that was very unreliable – task significance. This variable refers to the impact your work has on other people. Other researchers have noted that it can be hard for MTurk Workers to know the impact their work has, since there’s usually a very limited interaction with the Requester, much less knowledge of who the Requester is and how the Workers’ input might ultimately be used. So it makes sense in retrospect that this variable – task significance – might not be a reliable, meaningful part of MTurk Workers’ experiences. Or, MTurk Workers might experience the impact of their work in a different way that we’re just not picking up on with our current questionnaire items.

Best and Worst Requester Practices

Because we asked Workers about good and bad experiences, we could also look for themes in those stories. We found about 20 themes each in the good and bad stories, and sorted them by their average job satisfaction ratings – this gave us a list of Best and Worst Requester Practices. The #1 best Requester practice was (drum roll?) building relationships – staying connected with Workers over time and sending out work to Workers with whom you had those relationships. This doesn’t seem very different from what we might think a traditional worker would care about – does your boss know you? Does your boss send you work that you’d be great for? That same process can be implemented to improve Workers’ satisfaction.

On the other side, the #1 worst Requester practice was unfair pay. This is an issue that’s generated a lot of discussion. What does “fair” pay look like? We detailed some guidelines for how we recommend determining fair pay (and some guidelines for what to avoid) in our paper. Our general preference is to use (at least) minimum wage for the reasonable estimated time we expect the task to take.

This relates to another one of our findings: about 40% of the Workers in our study said that MTurk was their primary job. That’s nearly half of our sample! And this was true both for Workers from the United States as well as Workers from India. With that in mind, it becomes really important to ponder what fair pay means for at least some Workers completing your tasks who may be relying on MTurk as their main or only job.

While some of what we know from I-O seemed to apply to Workers on MTurk, a lot of what we’ve seen in traditional jobs didn’t seem to apply – it looks like there is plenty more research to be done in this growing industry.

Check out a short overview of our article below, or read the full article here:

Brawley, A. M., & Pury, C. L. S. (2016). Work experiences on MTurk: Job satisfaction, turnover, and information sharing. Computers in Human Behavior, 54, 531-546. doi:10.1016/j.chb.2015.08.031

Or review a slideshow/ppt here: https://drive.google.com/file/d/197A4gr-7wwPx-MYqM_ucwB8FuGdg9Mi6/view?usp=sharing