Every once in a while I get emails asking me about norms for the Big Five Inventory. I got one the other day, and I figured that if more than one person has asked about it, it’s probably worth a blog post.
There’s a way of thinking about norms — which I suspect is the most common way of thinking about norms — that treats them as some sort of absolute interpretive framework. The idea is that you could tell somebody, hey, if you got this score on the Agreeableness scale, it means you have this amount of agreeableness.
But I generally think that’s not the right way of thinking about it. Lew Goldberg put it this way:
One should be very wary of using canned “norms” because it isn’t obvious that one could ever find a population of which one’s present sample is a representative subset. Most “norms” are misleading, and therefore they should not be used.
That is because “norms” are always calculated in reference to some particular sample, drawn from some particular population (which BTW is pretty much never “the population of all human beings”). Norms are most emphatically NOT an absolute interpretation — they are unavoidably comparative.
So the problem arises because the usual way people talk about norms tends to bury that fact. So people say, oh, you scored at the 70th percentile. They don’t go on to say the 70th percentile of what. For published scales that give normed scores, it often turns out to mean the 70th percentile of the distribution of people who somehow made it into the scale author’s convenience sample 20 years ago.
So what should you do to help people interpret their scores? Lew’s advice is to use the sample you have at hand to construct local norms. For example, if you’re giving feedback to students in a class, tell them their percentile relative to the class.
Another approach is to use distributional information from existing dataset and just be explicit about what comparison you are making and where the data come from. For the BFI, I sometimes refer people to a large dataset of adult American Internet users that I used for a paper. Sample descriptives are in the paper, and we’ve put up a table of means and SDs broken down by age and gender for people who want to make those finer distinctions. You can then use those means and SDs to convert your raw scores into z-scores, and then calculate or look up the normal-distribution percentile. You would then say something like, “This is where you stand relative to a bunch of Internet users who took this questionnaire online.” (You don’t have to use that dataset, of course. Think about what would be an appropriate comparison group and then poke around Google Scholar looking for a paper that reports descriptive statistics for the kind of sample you want.)
Either the “local norms” approach or the “comparison sample” approach can work for many situations, though local norms may be difficult for very small samples. If the sample as a whole is unusual in some way, the local norms will remove the average “unusualness” whereas the comparison-sample approach will keep it in there, and you can decide which is the more useful comparison. (For example, an astronaut who scores in the 50th percentile of conscientiousness relative to other astronauts would be around the 93rd percentile relative to college undergrads.) But the most important thing is to avoid anything that sounds absolute. Be consistent and clear about the fact that you are making comparisons and about who you are comparing somebody to.
The Hardest Science 