Norms for the Big Five Inventory and other personality measures

Every once in a while I get emails asking me about norms for the Big Five Inventory. I got one the other day, and I figured that if more than one person has asked about it, it’s probably worth a blog post.

There’s a way of thinking about norms — which I suspect is the most common way of thinking about norms — that treats them as some sort of absolute interpretive framework. The idea is that you could tell somebody, hey, if you got this score on the Agreeableness scale, it means you have this amount of agreeableness.

But I generally think that’s not the right way of thinking about it. Lew Goldberg put it this way:

One should be very wary of using canned “norms” because it isn’t obvious that one could ever find a population of which one’s present sample is a representative subset. Most “norms” are misleading, and therefore they should not be used.

That is because “norms” are always calculated in reference to some particular sample, drawn from some particular population (which BTW is pretty much never “the population of all human beings”). Norms are most emphatically NOT an absolute interpretation — they are unavoidably comparative.

So the problem arises because the usual way people talk about norms tends to bury that fact. So people say, oh, you scored at the 70th percentile. They don’t go on to say the 70th percentile of what. For published scales that give normed scores, it often turns out to mean the 70th percentile of the distribution of people who somehow made it into the scale author’s convenience sample 20 years ago.

So what should you do to help people interpret their scores? Lew’s advice is to use the sample you have at hand to construct local norms. For example, if you’re giving feedback to students in a class, tell them their percentile relative to the class.

Another approach is to use distributional information from existing dataset and just be explicit about what comparison you are making and where the data come from. For the BFI, I sometimes refer people to a large dataset of adult American Internet users that I used for a paper. Sample descriptives are in the paper, and we’ve put up a table of means and SDs broken down by age and gender for people who want to make those finer distinctions. You can then use those means and SDs to convert your raw scores into z-scores, and then calculate or look up the normal-distribution percentile. You would then say something like, “This is where you stand relative to a bunch of Internet users who took this questionnaire online.” (You don’t have to use that dataset, of course. Think about what would be an appropriate comparison group and then poke around Google Scholar looking for a paper that reports descriptive statistics for the kind of sample you want.)

Either the “local norms” approach or the “comparison sample” approach can work for many situations, though local norms may be difficult for very small samples. If the sample as a whole is unusual in some way, the local norms will remove the average “unusualness” whereas the comparison-sample approach will keep it in there, and you can decide which is the more useful comparison. (For example, an astronaut who scores in the 50th percentile of conscientiousness relative to other astronauts would be around the 93rd percentile relative to college undergrads.) But the most important thing is to avoid anything that sounds absolute. Be consistent and clear about the fact that you are making comparisons and about who you are comparing somebody to.

Two obviously wrong statements about personality and political ideology

On the heels of yesterday’s post about the link between religiosity and conservatism, I came across a New York Magazine article discussing recent research on personality, genetics, and political ideology. The article summarizes a lot of really interesting work by John Jost on ideology, Jonathan Haidt on moral foundations, David Pizarro on emotional responses and politics, etc. etc. But when it says things like…

Over the past few years, researchers haven’t just tied basic character traits to liberalism and conservatism, they’ve begun to finger specific genes they say hard-wire those ideologies.

… I just cringe. Research on personality and genetics does not support the conclusion that ideology is hard-wired, any more than our work on how political discourse ties religiosity to politics shows that ideology is a blank-slate social artifact.

Any attempt to understand the role of personality and genetics in political attitudes and ideology will have to avoid endorsing 2 obviously wrong conclusions:

1. Ideology and political attitudes have nothing to do with personality or genes.

2. Genes code for ideology and political attitudes in a clear, unconditional way.

Maybe in some distal and complex way our genes code for variations in how different psychological response systems work — under what conditions they are more and less active, how sensitive they are to various inputs, how strongly they produce their various responses, etc. In situ, these individual differences are going to interact with things like how messages are framed, how they are presented in conjunction with other information and stimuli, who is presenting the information, what we think the leaders and fellow members of our important social groups think and feel, etc.

What this interactivity means for doing science is that if you hold one thing constant (whether by experimental control or by averaging over differences) and let the other one vary, you will find an effect of the one you let vary. For example, if you look at how different people respond to the same set of sociopolitical issues, you are going to get reliable patterns of different responses that reflect people’s personalities. And if you frame and present the same issue in several different ways, and measure the average effect of the different framings, you are going to get different average responses that reflect message effects. Both are interesting experimental results, but both are testing only pieces of a plausible theoretical model.

Most researchers know this, I think. For example, from the NYMag article:

Fowler laughs at the idea that he had isolated a single gene responsible for liberalism—an idea circulated in much of the chatter about the study. “There are hundreds if not thousands of genes that are all interacting to affect complex social behaviors,” Fowler says, and scientists have only a rough sense of that process. “There’s a really long, complex causal chain at work here,” says UC-Berkeley political scientist Laura Stoker, “and we won’t get any real understanding without hundreds and hundreds of years’ more research.”

Let’s stay away from lazy and boring concepts like hard-wired. The real answers are going to be a lot more interesting.

Where does the link between religiosity and conservatism come from?

My collaborator Ari Malka has an op-ed titled Are religious Americans always conservative?

Why, then, does religiosity relate to conservatism at all? One possibility is that there is some type of organic connection between being a religious person and being a conservative person. Perhaps the traits, moral standards and ways of thinking that characterize religious people also naturally lead them to prefer conservative social outcomes and policies. Another possibility, however, is that this relation really has to do with the messages from political and religious discourse, and how some people respond to these messages.

Two pieces of evidence support this latter explanation…

The evidence comes from a new paper we have out in Political Psychology. Here’s the abstract:

Some argue that there is an organic connection between being religious and being politically conservative. We evaluate an alternative thesis that the relation between religiosity and political conservatism largely results from engagement with political discourse that indicates that these characteristics go together. In a combined sample of national survey respondents from 1996-2008, religiosity was associated with conservative positions on a wide range of attitudes and values among the highly politically engaged, but this association was generally weaker or nonexistent among those less engaged with politics. The specific political characteristics for which this pattern existed varied across ethno-religious groups. These results suggest that whether religiosity translates into political conservatism depends to an important degree on level of engagement with political discourse.

Malka, A., Lelkes, Y., Srivastava, S., Cohen, A. B., & Miller, D. T. (2012). The association of religiosity and political conservatism: The role of political engagement. Political Psychology, 33, 275-299.

The adaptive and flexible workplace personality: What should I talk about?

I was invited to give a talk at an in-service day for my university’s library staff. They are asking people from around the university to contribute, and since our library staff does so much for the rest of us, I thought it would be nice to help out. (Seriously, who doesn’t think librarians are awesome?)

The theme of the in-service day is “Exercising Your Adaptability and Flexibility” (which I think is geared toward helping people think about changes in technology and other kinds of workplace changes). The working title for my talk is “The Adaptive and Flexible Workplace Personality.” They gave me pretty wide latitude to come up with something that fits that theme, and obviously I want to keep it grounded in research. I have a few ideas, but I thought I’d see if I can use the blog to generate some more.

Personality and social psychologists, what would you talk about? What do you think would be important and useful to include? I have one hour, and the staff will be a mix of professional librarians, IT folks, other library staff, etc. I’d like to keep it lively, and maybe focus on 2 or 3 take-home points that people would find useful or thought-provoking.

Fun with Google Correlate

A new tool called Google Correlate lets you input a search term and then creates a state-by-state map of how many people search for it. It then it shows you what other search terms have similar state-by-state patterns.

A search for my name (what else would I have plugged in first?) shows the most searches coming from my home state of Oregon, and a notable lack of interest stemming from the Great Plains. Of note: interest in McBain: The Movie follows a very similar regional pattern:

Google Correlate search for sanjay srivastava and mcbain: the movie

I’m trying to think of a good scientific use for this tool, but I keep getting stuck on the fact that the top regional correlate of personality is “nipple stimulation.”

Personality traits are unrelated to health (if you only measure traits that are unrelated to health)

In the NY Times, Richard Sloan writes:

It’s true that in some respects we do have control over our health. By exercising, eating nutritious foods and not smoking, we reduce our risk of heart disease and cancer. But the belief that a fighting spirit helps us to recover from injury or illness goes beyond healthful behavior. It reflects the persistent view that personality or a way of thinking can raise or reduce the likelihood of illness.

But there’s no evidence to back up the idea that an upbeat attitude can prevent any illness or help someone recover from one more readily. On the contrary, a recently completed study of nearly 60,000 people in Finland and Sweden who were followed for almost 30 years found no significant association between personality traits and the likelihood of developing or surviving cancer. Cancer doesn’t care if we’re good or bad, virtuous or vicious, compassionate or inconsiderate. Neither does heart disease or AIDS or any other illness or injury.

Sloan, a researcher in behavioral medicine, is trying to make a point about “a fighting spirit,” but in the process he makes a larger point about personality traits being unassociated with health. And when he overreaches, he is clearly and demonstrably wrong.

That study of 60,000 people (which the Times helpfully links to) used the Eysenck Personality Inventory and thus only looked at two personality traits, extraversion and neuroticism. They found no association between those traits and incidence of cancer or survival after cancer. But the problem is that the researchers didn’t measure conscientiousness, the personality trait factor that has been most robustly associated with all kinds of health behaviors and health outcomes (including early mortality).

Of course, conscientiousness isn’t really about upbeat attitude or a fighting spirit. It’s more about diligently taking care of yourself in many small ways over a lifetime. In that respect Sloan’s central point about “fighting spirit” isn’t disputed by the conscientiousness findings. (Researchers working in the substantial optimism and health literature may or may not feel differently.) Moreover, the moral and philosophical implications — whether we should praise or blame sick people for their attitudes — go well beyond the empirical science (though they certainly can and should be informed by it). But a reader could easily get confused that Sloan is making a broader point that personality doesn’t matter in health outcomes — and that just ain’t so.

I’m not sure Sloan intended to take such a broad swipe against personality traits, given that his own research has examined links between hostility and cardiac outcomes. Then again, browsing his publications leaves me confused. His op-ed says that being “compassionate or inconsiderate” has nothing to do with heart disease; but this abstract from one of his empirical studies concludes that “[trait] hostility may be associated with risk for cardiovascular disease through its effects on interpersonal interactions.” I haven’t read his papers — I just Google Scholared him this morning — so I’ll give him the benefit of the doubt that there’s some distinction I’m missing out on.

Self-selection into online or face-to-face studies

A new paper by Edward Witt, Brent Donellan, and Matthew Orlando looks at self-selection biases in subject pools:

Just over 500 Michigan State University undergrads (75 per cent were female) had the option, at a time of their choosing during the Spring 2010 semester, to volunteer either for an on-line personality study, or a face-to-face version…

Just 30 per cent of the sample opted for the face-to-face version. Predictably enough, these folk tended to score more highly on extraversion. The effect size was small (d=-.26) but statistically significant. Regards more specific personality traits, the students who chose the face-to-face version were also more altruistic and less cautious.

What about choice of semester week? As you might expect, it was the more conscientious students who opted for dates earlier in the semester (r=.-.20). What’s more, men were far more likely to volunteer later in the semester, even after controlling for average personality difference between the sexes. For example, 18 per cent of week one participants were male compared with 52 per cent in the final, 13th week.

Self-selection in subject pools is not a new topic — I’ve heard plenty of people talk about an early-participant conscientiousness effect (though I don’t know if that’s been documented or if it’s just lab-lore). But the analyses of personality differences in who takes online versus in-person studies are new, as far as I know — and they definitely add a new wrinkle.

My lab’s experience has been that we get a lot more students responding to postings for online studies than face-to-face, but it seems like we sometimes get better data from the face-to-face studies. Personality measures don’t seem to be much different in quality (in terms of reliabilities, factor structures, etc.), but with experiments where we need subjects’ focused attention for some task, the data are a lot less noisy when they come from the lab. That could be part of the selection effect (altruistic students might be “better” subjects to help the researchers), though I bet a lot of it has to do with old-fashioned experimental control of the testing environment.

What could be done? When I was an undergrad taking intro to psych, each student was given a list of studies to participate in. All you knew was the codenames of the studies and some contact information, and it was your responsibility to arrange with the experimenter to take the experiment. It was a pain on all sides, but it was a good way to avoid these kinds of self-selection biases.

Of course, some people would argue that the use of undergraduate subject pools itself is a bigger problem. But given that they aren’t going away, this is definitely something to pay attention to.