Watch Out for Unsound Research Practices

7/23/2024

A savvy consumer or sponsor of research must avoid being taken in by common tricks and fallacies. You’ve heard of unsound practices such as cherry-picking and fishing expeditions. Maybe you’ve heard of the Texas Sharpshooter Fallacy. Can you put your finger on why these are problematic?

In cherry-picking, results are chosen and presented that best fit the idea being promoted, at the exclusion of the other findings. In other words, what you're shown is a biased selection.

A fishing expedition is related. In this questionable practice, researchers continue to seek out findings (whether group differences, relationships, or what have you) until they come upon some that support their desired position. They analyze for as long as it takes to find the “right” results. Then they report those, downplaying or excluding all the others obtained along the way. (Another related term: “torturing the data until they confess.”)

The Texas Sharpshooter Fallacy is related as well. Imagine a person who sprays the side of a barn with a shotgun. Then he walks up to the barn and locates a spot where a few hits have formed a tight cluster. He paints a target around these; paints over all the rest; and proudly proclaims that "the target" is where he was aiming all along.

Underlying all three types of errors is the principle that the more analyses one conducts on a given topic, the greater the chance of a false positive. In a false positive, one is fooled into thinking a result is noteworthy when in fact it is caused by nothing more than chance.

In addition, all three can be seen as examples of the unsound practice of Hypothesizing After Results are Known, or HARKing. HARKing is opportunistic; it inadvisedly focuses on what often turn out to be chance findings. You will find these types of errors discussed in the context of the Multiple Comparison Problem and, more subtly, The Garden of Forking Paths as described by leading statistician Andrew Gelman in his blog.

Recognizing these errors when others fall for them will make you a savvier interpreter of research. Avoiding these types of mistakes will go a long way toward making your own work more sound.

Contact: [email protected]

9 Comments

David Dodenhoff

7/24/2024 10:37:03 am

Good stuff, Roland. All of these problems have begun to pollute the discourse much more now that anyone/everyone can pull down research studies off the internet with the snap of a finger.

For those of us who try to be conscientious about the research we do and use, maybe you could write a follow-up post that includes some examples of studies plagued by the kinds of problems you're talking about and studies that avoid them.

Cheers!

Roland Stark

7/24/2024 01:51:01 pm

Good points, David. Lest people become pessimistic in the face of "lies, damn lies, and statistics," I'll quote Frederick Mosteller: "It's easy to lie with statistics. It's easier to lie without them."

Gigi Taylor link

7/25/2024 09:33:19 am

Nice! Thank you Roland for this entry and your entire blog! As a consumer of survey research, I frequently find myself in the position of EVALUATING the quality of statistical analysis. These kinds of tricks or fallacies are great things to know. Here is my question: How will I know if I am being presented with biased results? Do I ask about the analysis process? Should I ask to see all the results, not just the ones presented to me?

7/25/2024 02:43:40 pm

Thank you for your reactions and your excellent question, Gigi. Sometimes an author will indicate that they pursued analyses whose results they do not report. More often one needs to read between the lines to infer that that was the case. (Are there analytic approaches that would be more intuitive than those reported? Might they have been conducted, then glossed over?) Certainly an honest researcher--and there are many--will disclose any additional analyses if asked about this. I wish I could offer a guaranteed way to know :-)

7/26/2024 07:23:54 am

Another answer to your question: Sometimes it suggests a problem with multiple comparisons if a researcher chooses what seems to be an over-complicated, excessively abstract series of analyses. As in, start with multivariate analysis A; feed the scores from that into multivariate analysis B; then use those results to construct multivariate analysis C. By the time the author is finished, it's hard to tell just what the model represents in any real-life terms. This signals to me that the more basic and more intuitive approach may have been tried and set aside after it failed to yield the desired results.

Laurie Courtney

7/25/2024 05:25:15 pm

I always appreciate your to-the-point, easy to read and share reminders. It helps me to sharpen my reading eye, which seems to continually dullen without the reminders!

Jason

7/25/2024 08:29:49 pm

I remember a book called How to Lie with Statistics, written in 1954. I wonder how much has changed since then. Are we better liars now?

7/26/2024 07:20:22 am

What a fascinating question!

Alecos Papadopoulos

7/29/2024 11:54:17 am

Good work. These things should be reminded to the world, at every opportunity.

The rest are underworld criminals, but HARKing is the interesting one, because it brushes with real science: observing the world, not only casually but also through systematic collection and combination of data, is how we listen to the world (societies included).

The subtle distinction is whether the methodology that was used to "obtain the results" was structurally biased in favor of these results, which is not an easy issue to settle. But if there is no such bias, there is nothing wrong with looking at the results, scratching your head and saying "what the hell is going on here? We have to try to explain this".

Blog

Watch Out for Unsound Research Practices

Leave a Reply.

Author

Archives