The internet was jolted into a frenzy on Sunday after The Washington Post published a story about a Qualtrics poll that found a whopping 52% of Republican respondents believed President-elect Donald Trump won the popular vote.
Of course, Democratic presidential nominee Hillary Clinton won the popular vote, and is currently ahead by nearly 3 million votes as a small sliver of ballots continue to be counted.
So what happened? A somewhat leading question might be to blame for the surprising number of those who answered in the affirmative, as the result stood in contrast to data from a recent Pew poll that showed 32% of Republican respondents either thought Trump won or were unsure.
The Qualtrics question was phrased as followed: “In last month’s election, Donald Trump won the majority of votes in the Electoral College. Who do you think won the most popular votes?”
It’s a classic case of a “leading question,” one that is frowned upon in polling.
As Carl Bialik wrote in The Wall Street Journal in 2010: “Subtle differences in how poll questions are phrased, or in which choices are offered as responses, have a significant effect on polling results.”
Mike Dixon, an assistant professor of operations management at the Naval Postgraduate School’s Graduate School of Business and Public Policy, helped to better explain this in a presentation on avoiding poor survey questions.
On leading questions, he wrote they suggest “to the respondent that the researcher expects or desires a certain answer.”
Dixon offered a question as an example: “Now that you’ve seen how you can save time, would you buy our product?” it asked. The professor said it “tipped” the hand of the questioner, indicating a desire for a “yes” answer.
That question was fairly similar to the Trump question, which stated that Trump won the Electoral College before asking, “Who do you think won the most popular votes?”
It was an indication that Trump was the winner, before the person had to answer who, in fact, won.
The polling firm could have listed Trump and Clinton as options and more directly phrased the question as, “Who won the popular vote?” Another option: “Did Hillary Clinton or Donald Trump win the popular vote?”
After the Post went with the headline, “A new poll shows 52% of Republicans actually think Trump won the popular vote” for the story, it found life online. (Business Insider also published a story on the poll, noting how the question was phrased.)
Many on Twitter jumped on the number, which jumped off the page at first glance:
Says it all MT @ianbremmer: Who thinks Trump won the popular vote?
52% of Republicans
7% of Democrats
24% of Independents
— Glenn Thrush (@GlennThrush) December 19, 2016
Reality: Clinton won popular vote by 3 million.
52% of Republicans believe Trump won.
— Alexander Nazaryan (@alexnazaryan) December 18, 2016
Trump claimed he would have won the popular vote if “millions” didn’t vote illegally, an outlandish claim made without a shred of supporting evidence.
Also, in answering the question, a vastly smaller number of Democrats and independent voters answered that Trump won the popular vote.
That, however, doesn’t excuse the poor framing, which may have played as big, if not a bigger, role in the results than those other two factors.
Business Insider Emails & Alerts
Site highlights each day to your inbox.