Etienne LeBel writes:
My colleague [Lorne Campbell] and I just got a paper accepted at Psych Science that reports on the outcome of two strict direct replications where we worked very closely with the original author to have all methodological design specifications as similar as those in the original study (and unfortunately did not reproduce the original finding).
We believe this is an important achievement for the “replication movement” because it shows that (a) attitudes are changing at the journal level with regard to rewarding direct replication efforts (to our knowledge this is the first strictly direct replications to be published at a top journal like Psych Science [JPSP eventually published large-scale failed direct replications of Bem’s ESP findings, but this was of course a special case]) and (b) that direct replication endeavors can contribute new knowledge concerning a theoretical idea while maintaining a cordial, non-adversarial atmosphere with the original author. We really want to emphasize this point the most to encourage other researchers to engage in similar direct replication efforts. Science should first and foremost be about the ideas rather than the people behind the ideas; we’re hoping that examples like ours will sensibilize people to a more functional research culture where it is OK and completely normal for ideas to be revised given new evidence.
An important achievement indeed. The original paper was published in Psychological Science too, so it is especially good to see the journal owning the replication attempt. And hats off to LeBel and Campbell for taking this on. Someday direct replications will hopefully be more normal, but in world we currently live in it takes some gumption to go out and try one.
I also appreciated the very fact-focused and evenhanded tone of the writeup. If I can quibble, I would have ideally liked to see a statistical test contrasting their effect against the original one – testing the hypothesis that the replication result is different from the original result. I am sure it would have been significant, and it would have been preferable over comparing the original paper’s significant rejection of the null versus the replications non-significant test against the null. But that’s a small thing compared to what a large step forward this is.
Now let’s see what happens with all those other null replications of studies about relationships and physical warmth.
The Hardest Science 
Following up on your quibble, I did indeed run some analyses in R that speaks to this question post-acceptance of our paper, which indicate that our effect is indeed (statistically) significantly different from the original finding. I did this using a bootstrapping approach where I repeatedly (with replacement) took sub-samples of N=56 (original sample size) from our combined sample of N=452, executed the target moderated multiple regression on each of these sub-samples and counted the number of times the (standardized) interaction coefficient was as large as in the original paper (Beta=-.42). Of 10,000 sub-samples, only 120 yielded interaction coefficients as large (or larger) than the original -.42. This is basically equivalent to saying that our (combined sample) interaction coefficient of .021 is different from the original -.42 at a statistical significance level of .012 (or p <.012).
For those interested, the R-code (vessReplicationBootstrap.R) for this analysis (and data file needed to run the code: replicationData.csv) is available in the project node on the OSF: http://openscienceframework.org/project/QsNVB/files
I think it’s great that this study was published and hope more replications will be conducted. However, let’s stay focused as well.
Some folks may be critical on the relationship between physical warmth and other variables. But, let’s not forget that within a particular domain of research, people may be critical as well and a catch all is not right either. In the first place, if research show an effect between a ‘disembodied’ cue of warmth and a third variable, it seems a far-fetched (though not impossible) effect, just as it does when there is an effect on a more stable trait (attachment, SVO, trait loneliness).
The replication can only tell us (after multiple labs do it, across contexts) that that particular variable does not hold. Nothing more. Admittedly, all other studies in psychology are underpowered, but this is all we got so far (and the theory and the other experiments seem to hold). And it is no different for other studies. So saying that ‘physical warmth’ studies are not replicated is not true (yet?), if you pick the targets that seem less likely to be replicated theoretically.