Over the past few years, social psychologists have come under fire for publishing work based on falsified and non-reproducible evidence. And now one social psychologist has published an awe-inspiringly clueless rant about this situation that will leave you smashing your face into your desk.
At issue in this essay by Harvard's Jason Mitchell is the specific accusation, leveled against many social psychologists, that their results cannot be reproduced. Though the idea of reproducibility is essential to the scientific process — indeed, some would argue the very definition of it — Mitchell believes that the emphasis on reproducibility is nothing more than "hand-wringing."
Then he goes on to offer a point-by-point analysis of why replication in science is overrated. In this section, he explains that it's practically impossible to ever replicate anything, and so getting in a fuss about replicating science is really absurd:
The argument goes [that] if someone cannot reproduce your results when following your recipe, something must be wrong with either the original method or in the findings it generated.
This is a barren defense. I have a particular cookbook that I love, and even though I follow the recipes as closely as I can, the food somehow never quite looks as good as it does in the photos. Does this mean that the recipes are deficient, perhaps even that the authors have misrepresented the quality of their food? Or could it be that there is more to great cooking than simply following a recipe? I do wish the authors would specify how many millimeters constitutes a "thinly" sliced onion, or the maximum torque allowed when "fluffing" rice, or even just the acceptable range in degrees Fahrenheit for "medium" heat. They don't, because they assume that I share tacit knowledge of certain culinary conventions and techniques; they also do not tell me that the onion needs to be peeled and that the chicken should be plucked free of feathers before browning. If I do not possess this tacit know-how—perhaps because I am globally incompetent, or am relatively new to cooking, or even just new to cooking Middle Eastern food specifically—then naturally, my outcomes will differ from theirs.
Likewise, there is more to being a successful experimenter than merely following what's printed in a method section. Experimenters develop a sense, honed over many years, of how to use a method successfully. Much of this knowledge is implicit. Collecting meaningful neuroimaging data, for example, requires that participants remain near-motionless during scanning, and thus in my lab, we go through great lengths to encourage participants to keep still. We whine about how we will have spent a lot of money for nothing if they move, we plead with them not to sneeze or cough or wiggle their foot while in the scanner, and we deliver frequent pep talks and reminders throughout the session. These experimental events, and countless more like them, go unreported in our method section for the simple fact that they are part of the shared, tacit know-how of competent researchers in my field; we also fail to report that the experimenters wore clothes and refrained from smoking throughout the session. Someone without full possession of such know-how—perhaps because he is globally incompetent, or new to science, or even just new to neuroimaging specifically—could well be expected to bungle one or more of these important, yet unstated, experimental details. And because there are many more ways to do an experiment badly than to do one well, recipe-following will commonly result in failure to replicate.
Because recipes and scientific experiments are exactly the same thing, especially when it comes to analysis of the human brain. This is just the beginning of an eye-poppingly ill-informed essay, where a person who claims to be a scientist rejects the basic tenets of the scientific method. You have to read the whole thing to fully appreciate the WTFery.
It led science journalist Ben Lillie to comment on Twitter:
If you are a scientist and you like punching your computer screen, I suggest reading the whole essay here.