Watch Out for Unsound Research Practices

A savvy consumer or sponsor of research must avoid being taken in by common tricks and fallacies.  You’ve heard of unsound practices such as cherry-picking and fishing expeditions.  Maybe you’ve heard of the Texas Sharpshooter Fallacy.  Can you put your finger on why these are problematic?

​In cherry-picking, results are chosen and presented that best fit the idea being promoted, at the exclusion of the other findings.  In other words, what you're shown is a biased selection.
​

​A fishing expedition is related.  In this questionable practice, researchers continue to seek out findings (whether group differences, relationships, or what have you) until they come upon some that support their desired position.  They analyze for as long as it takes to find the “right” results.  Then they report those, downplaying or excluding all the others obtained along the way.  (Another related term:  “torturing the data until they confess.”)
​

The Texas Sharpshooter Fallacy is related as well.  Imagine a person who sprays the side of a barn with a shotgun.  Then he walks up to the barn and locates a spot where a few hits have formed a tight cluster.  He paints a target around these; paints over all the rest; and proudly proclaims that "the target" is where he was aiming all along.

Underlying all three types of errors is the principle that the more analyses one conducts on a given topic, the greater the chance of a false positive.  In a false positive, one is fooled into thinking a result is noteworthy when in fact it is caused by nothing more than chance.

In addition, all three can be seen as examples of the unsound practice of Hypothesizing After Results are Known, or HARKing.  HARKing is opportunistic; it inadvisedly focuses on what often turn out to be chance findings.  You will find these types of errors discussed in the context of the Multiple Comparison Problem and, more subtly, The Garden of Forking Paths as described by leading statistician Andrew Gelman in his blog.
​
Recognizing these errors when others fall for them will make you a savvier interpreter of research.  Avoiding these types of mistakes will go a long way toward making your own work more sound. 
 
               Contact:  [email protected]