Under pressure from their employers and journals to produce statistically significant results, researchers turn to a number of tricks to make the numbers come out better.
As a result, more transparency is needed in psychological research, say Joseph Simmons and Uri Simonsohn of Wharton and Leif Nelson of UC Berkeley in their recent article, “A 21 Word Solution.”
It’s a follow-up to their 2011 paper, “False Positive Psychology,” which described some of the possible problems in psychology research. One of the major issues is “p-hacking,” or the practice of changing assumptions or data in an experiment to make sure that the probability (“p”) that an opposite hypothesis (“null”) contradicts the research is below a certain level. Ultimately, “p-hacking” makes research less valid and increases the number of “false positives.”
In response, the authors put together a 21-word statement every researcher should use as a disclosure, which they hope will make the field more transparent:
“We report how we determined our sample size, all data exclusions (if any), all manipulations, and all measures in the study.”
The authors perfectly sum up the transparency problem with an analogy and a photo. Whereas coffee shops are required to label milk containers, scientists don’t have to “label their milk.” In other words, researchers don’t have to disclose what data they started out with, whether they took observations out, or whether they’ve dropped things from their model:
The table below of simulated results from their earlier paper shows how much these unreported techniques can impact statistical significance:
The lesson? Look for disclosures in any scientific paper, and always be sceptical.Read the full article here
NOW READ: 18 Tips On Making Smarter Decisions
NOW WATCH: Ideas videos
Business Insider Emails & Alerts
Site highlights each day to your inbox.