Yesterday I put up a post about David Peterson’s ethnography The Baby Factory, an ethnography of 3 baby labs that discusses Peterson’s experience as a participant observer. My post was mostly excerpts, with a short introduction at the beginning and a little discussion at the end. That was mostly to encourage people to go read it. (It’s open-access!)
Today I’d like to say a little more.
How you approach the article probably depends a lot on what background and context you come to it with. It would be a mistake to look to an ethnography for a generalizable estimate of something about a population, in this case about how common various problematic practices are. That’s not what ethnography is for. But at this point in history, we are not lacking for information about the ways we need to improve psychological science. There have been surveys and theoretical analyses and statistical analyses and single-lab replications and coordinated many-lab replications and all the rest. It’s getting harder and harder to claim that the evidence is cherry-picked without seriously considering the possibility that you’re in the middle of a cherry orchard. As Simine put it so well:
even if you look at your own practices and those of everyone you know, and you don’t see much p-hacking going on, the evidence is becoming overwhelming that p-hacking is happening a lot. my guess is that the reason people can’t reconcile that with the practices they see happening in their labs and their friends’ labs is that we’re not very good at recognizing p-hacking when it’s happening, much less after the fact. we can’t rely on our intuitions about p-hacking. we have to face the facts. and, in my view, the facts are starting to look pretty damning.
You don’t even have to go as far as Simine or me. You just have to come into reading the ethnography with a realistic belief that problematic practices are at least at a high enough rate to be worrisome. And then the ethnography does what ethnographies do, and well in my judgment: it illustrates what these things look like, out there in the world, when they are happening.
In particular, I think a valuable part of Peterson’s ethnography is that it shows how problematic practices don’t just have to happen furtively by one person with the door closed. Instead, they can work their way into the fabric of how members of a lab talk and interact. When Leslie John et al. introduced the term questionable research practices, they defined it as “exploitation of the gray area of acceptable practice.” The Baby Factory gives us a view into how that can be a social process. Gray zones are by definition ambiguous; should we be shocked to find out that people working closely together will come to a socially shared understanding of them?
Another thing Peterson’s ethnography does is talk about the larger context where all this is happening, and try to interpret his observations in that context. He writes about the pressures for creating a public narrative of science that looks sharp and clean, about the need to make the most of very limited resources and opportunities, and about the very real challenges of working with babies (the “difficult research objects” of the subtitle). A commenter yesterday thought he came to the project with an axe to grind. But his interpretive framing was very sympathetic to the challenges of doing infant cognition research. And his concluding paragraphs were quite optimistic, suggesting that the practices he observed may be part of a “local culture” that has figured out how they can promote positive scientific development. I wish he’d developed that argument more. I don’t think infant cognition research has lacked for important scientific discoveries — but I would say it is in spite of the compromises researchers have sometimes had to make, not because of them.
I do think it would be a mistake to come away thinking this is something limited to infant cognition research. Peterson grounds his discussion in the specific challenges of studying babies, who have a habit of getting distracted or falling asleep or putting your stimuli in their mouths. Those particular problems may be distinctive to having babies as subjects, and I can understand why that framing might make baby researchers feel especially uncomfortable. But anybody who is asking big questions about the human mind is working with a difficult research object, and we all face the same larger pressures and challenges. There are some great efforts under way to understand the particular challenges of research practice and replicability in infant research, but whatever we learn from that is going to be about how broader problems are manifesting in a specific area. I don’t really see how you can fairly conclude otherwise.
The Hardest Science 