Simine Vazire has a great post contemplating how we should evaluate counterintuitive claims. For me that brings up the question: what do we mean when we say something is “counterintuitive?”
First, let me say what I think counterintuitive isn’t. The “intuitive” part points to the fact that when we label something counterintuitive, we are usually not talking about contradicting a formal, well-specified theory. For example, you probably wouldn’t say that the double-slit experiment was “counterintuitive;” you’d say it falsified classical mechanics.
In any science, though, you have areas of inquiry where there is not an existing theory that makes precise predictions. In social and personality psychology that is the majority of what we are studying. (But it’s true in other sciences too, probably more than we appreciate.) Beyond the reach of formal theory, scientists develop educated guesses, hunches, and speculations based on their knowledge and experience. So the “intuitive” in counterintuitive could refer to the intuitions of experts.
But in social and personality psychology we study phenomena that regular people reflect on and speculate about too. A connection to everyday lived experience is almost definitional to our field, whether you think it is something that we should actively pursue or just inevitably creeps in. So we have an extra source of intuitions – the intuitions of people who are living the phenomena that we study. Which includes ourselves, since social and personality psychologists are all human beings too.
And when you are talking about something that (a) people reflect on and wonder about and (b) is not already well settled, then chances are pretty good that people have had multiple, potentially contradictory ideas about it. Sometimes different people having different ideas; sometimes the same person having different ideas at different times. The contradictory ideas might even have made their way into cultural wisdom – like “birds of a feather flock together” versus “opposites attract.”
What I suspect that means is that “counterintuitive” is often just a rhetorical strategy for writing introduction sections and marketing our work. No matter how your results turned out, you can convince your audience that they once thought the opposite. Because chances are very good that they did. A skilled writer can exploit the same mechanisms that lead to hindsight bias to set people up, and then surprise! show them that the results went the other way.
I would not claim that this describes all instances of counterintuitive, but I think it describes a lot of them. As Simine points out, many people in psychology say that counterintuitive findings are more valuable — so clearly there is an incentive to frame things that way. (Counterintuitive framing is also a great way to sell a lot of books.)
Of course, it does not have to be that way. After all, we are the field that specializes in measuring and explaining people’s intuitions. Why don’t we ask our colleagues to back up their claims of being “counterintuitive” with data? Describe the procedure fully and neutrally to a group of people (experts or nonexperts, depending whose intuitions you want to claim to be counter to) and ask what they think will happen. Milgram famously did that with his obedience experiments.
We should also revisit why we think “counterintuitive” is valuable. Sometimes it clearly is. For example, when intuition systematically leads people to make consequentially bad decisions it can be important to document that and understand why. But being counterintuitive for counterintuitive’s sake? If intuitions vary widely — and so do results, across contexts and populations — then we run the risk that placing too much value on counterintuitive findings will do more to incentive rhetorical flash than substantive discoveries.
The Hardest Science 
I prefer instead of valuing being counter-intuitive, to value counter-examples to the idea that “all good things go together” (or all bad things go together). I suppose this might still be categorized as trying to be counterintuitive, though I think it is more about striving for nuance. An example that comes to mind is deviancy training, as it seems like we would want to encourage youth becoming closer to each other and maintaining close relationships, but if these relationships are with youths engaged in delinquency, maintaining connectedness may entail the youth engaging in more of these behaviors.
I agree that exceptions to “good goes with good” are often interesting, but not necessarily because they are counterintuitive. There are too many reasons you might observe good with good: because of methodological artifacts like self- and informant-report bias, because of shared broad-spectrum causes, or because of common mechanisms (which is usually what you’re looking for). Whereas the exceptions tend to have a narrower list of plausible explanations.
I like that point and I plan on “borrowing” it to use at some point in the future.
i was somewhat surprised to realize that matthias mehl and i actually did survey lay people (only 61 of them though, we were young and naive. and mturk didn’t exist.) to ask them what their predictions would be about our main study in our 2008 paper (Vazire & Mehl, JPSP). we of course used these results to show how counterintuitive our actual results were. (first sentence from the conclusion: “the results from these studies show that contrary to the strong intuitions of most, people are not always their own best expert.”).
I’m not at all surprised you thought of this before I did. :)
Self-knowledge / self-deception is probably another one of those areas where the relationship between research findings and people’s intuitions has deeper, substantive significance.
Now I wonder what my “intuitions” are about the meaning of the word “counterintuitive,” and what sort of information about the way the word is used might contradict those “intuitions.” I tend to think of “intuitions” as thoughts that pop into our heads as the result of conditioned reflexes, even if that conditioning is part of scientific training. So it’s counterintuitive to me to observe that we don’t call the results of the double slit experiment “counterintuitive.”
I think there is a logic to wanting to call just about anything “counterintuitive.” The logic goes as follows. Suppose I’m promoting a research idea. One thing I have to explain is why, if my idea is so great, people didn’t consider it earlier. “Counterintuitive” is one reason.
The other thing about “counterintuitive” is that it makes the idea seem more impressive. I’m reminded of the description of Steven Levitt as a “rogue” economist, a characterization that seems inappropriate to me, given that he received his Ph.D. from MIT, holds the title of Alvin H. Baum Professor at the University of Chicago, and has served as editor of the completely mainstream Journal of Political Economy. Further “rogue” credentials revealed by Levitt’s online C.V. include an undergraduate degree from Harvard, a research fellowship with the American Bar Foundation, membership in the Harvard Society of Fellows, a fellowship at the National Bureau of Economic Research, and a stint as a consultant for “Corporate Decisions, Inc.”