Google pays outside contractors $14.50 an hour to rate search results for relevance, then feeds this data back into its search algorithms. But to get the job, testers have to pass a couple of difficult exams.
This comes from a Search Engine Land interview with one of the unnamed Search Quality Raters — he sent along screenshots to prove he was legitimate — and it’s an interesting inside glimpse into how Google collects data to improve its most important product.
- Google assigns the jobs through contractors, who place listings on Craigslist and elsewhere for “Quality Raters”. The listings don’t mention Google, and raters never have direct contact with Google.
- To get the job, respondents have to pass two exams. The first tests how well they understand some basic instructions. On the second exam, they have to evaluate 140 search results by going to the Web sites and putting them in one of four categories: vital, useful, relevant, and off-topic/useless. They have to get 90% of the listings in each category right to get the job. This person says it was the hardest test he ever took — including in graduate level courses.
- On the job, they’re asked to compare two sets of search results — one set as they appear today, and one as they would appear after Google changes something in its search algorithm. In every case, they must actually visit the Web site.
- Multiple raters look at the results at the same time. If there’s disagreement, they have to come to consensus through online debate and comments. If consensus can’t be reached, a moderator steps in, looks at the comments, and makes the call. If a rater is too far away from the consensus too often, the contractor fires them.
- The testers get a lot of extra context like the hypothetical user’s location and what they were looking at immediately before they search. For instance, a search on “weather” that returns results in Florida is more useful if the searcher is in Florida than if he’s in Seattle.
- This tester earned $14.50 an hour, and was allowed to work 20 hours a week and 80 hours a month maximum.
Read the whole interview here.