I’m teaching the first section of a structural equation modeling class tomorrow morning. This is the 3rd time I’m teaching the course, and I find that the more times I teach it, the less traditional SEM I actually cover. I’m dedicating quite a bit of the first week to discussing principles of causal inference, spending the second week re-introducing regression as a modeling framework (rather than a toolbox statistical test), and returning to causal inference later when we talk about path analysis and mediation (including assigning a formidable critique by John Bullock et al. coming out soon in JPSP).
The reason I’m moving in that direction is that I’ve found that a lot of students want to rush into questionable uses of SEM without understanding what they’re getting into. I’m probably guilty of having done that, and I’ll probably do it again someday, but I’d like to think I’m learning to be more cautious about the kinds of inferences I’m willing to make. To people who don’t know better, SEM often seems like magical fairy dust that you can sprinkle on cross-sectional observational data to turn it into something causally conclusive. I’ve probably been pretty far on the permissive end of the spectrum that Andrew Gelman talks about, in part because I think experimental social psychology sometimes overemphasizes internal validity to the exclusion of external validity (and I’m not talking about the special situations that Mook gets over-cited for). But I want to instill an appropriate level of caution.
BTW, I just came across this quote from Donald Campbell and William Shadish: “When it comes to causal inference from quasi-experiments, design rules, not statistics.” I’d considered writing “IT’S THE DESIGN, STUPID” on the board tomorrow morning, but they probably said it nicer.
The Hardest Science 
I think this “When it comes to causal inference from quasi-experiments, design rules, not statistics.” is an excellent point. I would say however that when it comes to SEM strong theory is King. I always feel that SEM is best when used to test competing models.
In passing have you read Judea Pearl’s thoughts on SEM in his book ‘Causality’.
That Bullock et al critique is out now.
@Phil, that’s an interesting point about theory and design. I’d say that for internal validity, design rules; but for external validity, there is a place for models built on strong theory using data with more permissive designs. I made a related point here. Randomized experiments have a clear place. But if your goal is to understand how a complex process unfolds in the real world, then sometimes you also need to fit models where you don’t use randomization to artificially set associations to zero.
BTW, I picked up Pearl’s book some years ago but didn’t get very far. I’ve been thinking of taking another crack at it. Was there something in particular that you were thinking about?
@Joe, thanks for the heads-up. I’m waiting to see if all of social and personality psychology now grinds to a halt.
I don’t have the book on me (moved over seas and had to leave my books at home) so this is from memory. Besides Pearl’s neat introduction to the Bayesian approach the chapter on SEM has an interesting historical analysis. Pearl points out that SEM used to be called causal modelling because that what it was designed to do but somewhere along the line the meaning got lost. The chapter then provides a nice overview of how and when SEM can be used to infer causality. Might be a bit heavy for a class but given your focus on SEM and causality could be interesting.
In passing would be interested on your take on causality and cross-lag panel designs. Particularly in relation to personality and relationship effects. In particular I am interested on your take about the interpretation of correlated residuals.
I think cross-lag designs can be useful for the right applications. In terms of causal inference, I see cross-lag models as occupying a middle ground between randomized experiments and cross-sectional observational studies. They help rule out some third-variable causes but not all of them. I especially like Bollen and Curran’s autoregressive latent trajectory (ALT) models, which combine cross-lag with latent-trait and growth curve models. Chris Fraley and Brent Roberts’s Psych Review paper on personality-environment transactions is essentially a meta-analytic ALT model (independently derived as far as I know).
I’m not sure I have a general interpretation of correlated residuals — it sort of depends on the particulars of the model and the phenonemon(-a) being modeled. But that reminds me of something I heard once — can’t remember where now. Somebody said that a major difference between econometric models and psychometric models is that the economists start by assuming a saturated error structure and then look for reasons to fix some covariances to zero during respecification, whereas psychologists start by assuming independent errors and then look for reasons to free some covariances during respecification. I’m more of a pragmatist — if it fits and is interpretable then why not? Even if that means being a little bit of an economist.
Ah I thought Roberts was against the cross-lag approach when it came to this issue. I always thought he was set on using residual change scores (such as in his papers with Robins etc). Interesting!
I am aware of the ALT approach and have the papers by Bollen. Perhaps it is time I actually read them. Anyway enough of me derailing your blog. I hope your students enjoy the statistical ouija board that is SEM.
@Sanjay, thanks for pointing out that ATL business. It is always nice to come for the comments and get a nice surprise—not something that I was familiar with.
Why do you think external validity is ignored? Just because it is hard or because people aren’t really comfortable with the types of inferences and tools you need to really get the job done? as you point out in your example on self-help books.
Mark, I don’t really know why it’s ignored, I can just guess. There’s an argument that I am sympathetic to but don’t fully agree with, that internal validity is a necessary precondition to external validity. In a nutshell: how can you draw conclusions about causality in the real world when you cannot establish causality in your study? I think that paints things in too black-and-white of terms, but it’s not completely off base.
Mook’s argument is that in many cases of theory-testing, external validity is not relevant. If a theory lets you derive testable hypotheses about what should happen in your controlled lab setting, then an externally “invalid” experiment can still advance theory. In principle he may be right, but I think he gets wayyyy over-cited by social psychologists. Social psych almost by definition is trying to draw conclusions about what happens in the messy real world. I don’t believe an experiment has to superficially resemble a naturalistic setting, but most of the time I think it *is* incumbent on the experimenter to demonstrate that the experiment is a meaningful analog for some real-world process.