Blog Coordinator

Knobe's X-Phi Page

X-Phi Grad Programs

« Data Repository | Main | Moral from Non-Moral »



Feed You can follow this conversation by subscribing to the comment feed for this post.

Aaron Boyden

Despite not being sympathetic to the conservative views in question, I'm mostly sympathetic with the comment your reader made. These results may give one reason to further examine these beliefs, and see where they come from and what justifications there might be for holding them. But until such further examination is carried out, it's premature to assume they're bad just because some of the people who believe them are bad.

Of course, I may only think this because I'm not a virtue theorist. But I have some inclination to think that if virtue theory encourages one to think ad hominem arguments are good, that's a problem for virtue theory.


Marcus, I don't get the reader's comment. Like all informal fallacies, ad hominems reflect an inference pattern that, in some situations, are legitimate (c.f. slippery slope arguments are not problematic because of the inference pattern they utilize, but the dubiousness of the conditionals they utilize)

I take it that what you argue is that background facts about individuals' moral traits are relevant for their assessing moral intuitions.

I don't see what is ad hominem here. If it is, I would like to see the argument for why criticizing intuitions based on any cross-cultural data is not therefore also ad hominem. E.g. I have issues with the Machery et al. (2004) paper, but it is certainly not that it is ad hominem against Kripke.

Marcus Arvan

Hi Aaron,

Thank you for your comments. I'm sympathetic with the worry too, for obvious reasons. However, one of the main reasons I'm interested in the line of thought is that sometimes we find ourselves faced with "stalemates" -- argumentative situations where each side digs in, asserts their favored premises, denies the other side's premises, and where little more seems available to be said. This is, of course, not only a problem in moral and political philosophy but elsewhere as well. Philosophy is supposed to be a matter of argument -- but still, at some point, "arguments run out", and we find ourselves faced with (what appear) to be truly fundamental disagreements about premises.

This is where I find the virtue-theoretic line of thought I'm toying with (at the very least) intriguing. Perhaps, when we reach a loggerheads, there *is* something more we can say. Perhaps we can appeal to empirical research to at least motivate *some* reasons to think that one side's premises may be more justified than the other.

Indeed, for what it's worth, I can't help but feel the pull of this (as an individual). As an in-principle supporter of capital punishment, I find my judgment's correlation with "dark" personality traits to be disturbing -- and, if I am entirely honest with myself (which is of course difficult), I have to admit that I am a flawed person whose judgments about capital punishments may well be in some subtle way corrupted by flaws of personal character.

Perhaps, then, instead of justifying ad hominem arguments against other people, experimental results like mine might lead us to (productively) question our own intuitions, and indeed, become less bull-headed about our own arguments and positions when others disagree with us.

That's a potentially illuminating approach, no? Anyway, thanks again for your comments. I'm really not sure at this point what I make of the line of thought. It's good to hear others' opinions.


Marcus Arvan

Hi Brendan,

Thanks - I agree. I'm presently going back and forth on this. If there's truly wide cross-cultural agreement about the moral value of a trait (e.g. as "bad"), then it's not clear to me why it isn't legitimate to criticize moral intuitions that correlate with those traits -- particularly if we're talking about *premises* for arguments (see my comment above). After all, if a person is appealing to a moral judgment as a *premise*, then it seems perfectly cogent to say, "Look, I know you find that premise attractive, but it's linked with narcissism and maybe you should doubt it."



These are certainly interesting data, but I'm not sure about some of the interpretation offered here. (Admittedly, I'm a psychologist and not a philosopher)

First, a word about correlations and significance levels. You appear to have run at least 39 correlations between MIS item scores and D3 traits. I understand your going back to re-run the tests at the .01 level, and indeed, with that many correlation coefficients, some methodologists would suggest that even the .01 level is too liberal. The usual significance level (.05) could be put over the number of tests (39) to suggest the proper level of significance for each test (~.0013). See:

But for me the choice of a significance level is not as important as actually considering the conceptual logic of significance tests. Basically, when you're examining relationships between many individual items (the MIS items) and three D3 traits, it's likely that some of the apparent relationships will be present just by chance, and may not replicate (at least at the same level) in a separate sample. To interpret the relationships that happened to be significant, post hoc, is something of an overinterpretation.

And there's another issue - statistical significance isn't as important as inspecting the correlation coefficients. Only one of your correlation coefficients is greater than 0.3, and in general, they're much lower than that. Now, correlations of 0.3 can be important - even correlations of 0.1 can be important. (I'm ignoring the sign of the coefficients; absolute value is what's important.) But I think that one of your interpretations requires a stronger relationship. You seem to think that you can say something, categorically, about the virtue of a person based on their MIS responses. But that implies a reliability in prediction that you can't get with a correlation of 0.3 (and most of your correlations aren't even that strong). Look at a scatterplot graph with a correlation of 0.3 and see how well you can predict Y given X. Given that there's a correlation (again, a weak but possibly meaningful one) between gender and D3 traits, you might as well say that the virtuous agent is not male.

Marcus Arvan

Hi Ben - thanks for your comments. Here are some thoughts in reply:

First, even if the .01 cutoff was still too liberal -- and I don't think it is (see below) -- many of my results were significant well beyond that cut-off. Indeed, many of my results were significant at a level of .

Second, I don't think the .01 cut-off is too liberal, for two reasons. The first (more minor) reason is that the study was hypothesis-based, and my results cohered with my hypotheses. When this is the case, one has pre-hoc reasons to think the results are probably genuine (not false positives). Second, and I think much more importantly, the precise nature of my results strongly suggests that the results are genuine. After all, it is not as though I simply found isolated correlations between moral judgments and the dark personality traits. Wherever there were any significant correlations, there were correlations between the moral judgment in question and *all three* traits (at a .01 level). Because the three traits are related to one another but by no means identical or coextensive, the statistical probability that a moral judgment would correlate at a .01 level with *all three* traits is vanishingly small (

Finally, I think you've understood me as wishing to make a stronger claim than I wish to make. I do not mean to suggest, categorically, that my results show (e.g.) supporters of capital punishment or economic libertarianism to be "all Machiavellians" or "all narcissists." That, obviously, is a very strong claim -- but I don't think I need to make it. Here's why. My results are (as you imply) perfectly consistent with the hypothesis that plenty of supporters of capital punishment (etc.) do *not* score high in Machiavellianism, narcissism, or psychopathy. My results merely show that people who *do* score high in these areas are statistically far more likely to have those beliefs (than have beliefs against capital punishment, etc.). Okay, fair enough. My results don't show anything categorical about people. But -- and this perhaps clarifies the kind of philosophical move I am tempted to make -- isn't it disturbing enough to know (even if one isn't a Machiavellian or psychopath oneself) that one's moral beliefs are often *held* my people of that sort? So, for example, consider my stance on capital punishment (I tend to favor it in principle). I must confess that although I don't consider myself a Machiavellian or a psychopath (the jury is still out on whether I'm a narcissist:), I can't help but find it epistemically disturbing that, far and away, people who *are* Machiavellians, narcissists, and psychopaths share my view. (Indeed, even though I'm *not* cruel, the fact that the Short D3 item, "I can be cruel to others", correlated at a

To put the matter more simply: isn't "guilt by association" (with bad people) something to at least worry about (even if one isn't a bad person oneself and categorical claims are unwarranted)?


Hi Marcus,

Thanks very much for your reply - a few thoughts:

I don't want to spend too much time on exactly what p-value should be used to determine significance, and I actually don't have a strong opinion on that issue (in fact, part of my point is that it's fairly arbitrary what you choose). That said, I do want to address something related to the p-value issue. You say that your study was hypothesis-driven. True, but exactly were your hypotheses? Nothing focused, just that "there would be many significant positive correlations" between conservative views and D3 traits. Essentially, this just says that conservatives would on average have higher levels of the D3 traits. You conducted 39 statistical tests of this hypothesis. The chance that *some* of those 30 tests would show a relationship is what your false positive rate is (basically). So although I don't care too much about p-values, I think there's a case that even .01 isn't stringent enough.

But to me the larger issue is that it's not terribly important that a relationship exists. In the social sciences, most variables (traits, attitudes, etc.) are related in some way. The question is: what's the magnitude of that relationship? Now, you're claiming that the magnitude is *strong*. You just made the following two claims:

"My results merely show that people who *do* score high in these areas are statistically far more likely to have those beliefs (than have beliefs against capital punishment, etc.)."


"far and away, people who *are* Machiavellians, narcissists, and psychopaths share my view."

Those words "far more" and "far and away" are, in my view, simply inaccurate based on the small magnitude of your correlations. Let's be generous and assume that you had correlations of 0.3 (rather than just one at that level). How can you interpret such a correlation?

One method is by squaring it, obtaining the "coefficient of determination"; you would find that .09 (9%) of the variability in one variable is accounted for by variability in the other variable. So about 10% of variability in attitudes toward, say, capital punishment is accounted for by variability in D3 traits. That's not much. That's not "far more" anything.

But some researchers suggest that the coefficient of determination makes relationships seem less important. OK, Another way of interpreting the correlation is to calculate the "binomial effect size display," which is 50r+50. That tells you the proportion of folks who are above average in one variable, given that they are above average on the other. Without knowing someone's view on capital punishment, you'd guess their D3 trait level (just guessing whether they're above or below average) with 50% accuracy. If you know their view on capital punishment, and there's a 0.3 correlation, you'd increase your accuracy to 50(.3)+50 or 65% accuracy. That's not even being right two times out of three. Again, you don't have much predictive power.

(I should admit that I'm oversimplifying some things in this presentation, but I think it's a good introduction to how to interpret correlations, and why you need to actually look at their size, rather than just testing their significance, when the latter really just tells you whether you had a lot of participants.)


P.S. - a small correction to my discussion of the binomial effect size display. I defined it but didn't apply it exactly right.

If D3-ness (the tendency to have higher D3 trait levels) and pro-death-penalty-ness were not related at all, then 50% of those who are above average in their pro-death-penalty views would be above average in their D3-ness (complete randomness).

If the correlation between the variables is .3, that 50% only goes up to 65%, which isn't even two out of three, and isn't too different from 50%.

I posted too hastily - sorry.

Marcus Arvan

Hi Ben - thanks again for the comments. Your points are well taken. However, I have a couple of thoughts in reply. First, any claim that an observed effect is "large" or "small" is a matter of interpretation, and intuitively depends on context. You suggest that 10% effect of predictive variability is small. I would counter that, given the context, it's a pretty large effect. After all (and I recognize this is a bit crude), suppose 10% of people who are pro-death penalty views have those views *because* of the personality traits at issue. That, intuitively, seems to me like a pretty big deal. Indeed, if I were to find out that 10% more people who hold my moral views (as opposed to an opposing view) scored high in narcissism, Machiavellianism, or psychopathy, that would concern me. Further, if (as you note) I could predict with 65% accuracy whether a person with pro-death penalty beliefs is likely to score high in narcissism or psychopathy, that too seems to me pretty significant. Second, and on a related note, I've heard different things about what sort of correlation coefficient researchers consider to indicate large effects. At least one person I've spoken to (a psychologist who helped me with my stats) has told me that .2 is pretty good, and .3 is very good. Was I misinformed?



Hi Marcus,

As you note, what counts as a small or large correlation depends on the context, and reasonable people can disagree about what counts as large or small, both in general, and in specific contexts.

So long as we're talking about correlation coefficients and not p-values, we've already made tremendous progress, since the smallness of p-values does not indicate the "largeness" of an effect. (If you have a sample of 10,000, most of your variables will probably be significantly correlated at the .01 level, regardless of what variables they are.)

One convention comes from the psychologist Jacob Cohen, who noted that in many areas of the behavioral sciences, .1 can be seen as a small effect, .3 as a medium effect, and .5 as a large effect, all *compared to the effects we normally see in the behavioral sciences*.

But you seem to have a more direct interpretation of correlations. You're using something like the binomial effect size display when you say that "if I were to find out that 10% more people who hold my moral views (as opposed to an opposing view) scored high in narcissism, Machiavellianism, or psychopathy, that would concern me."

This surprises me; I can't imagine being concerned about an effect like this in most domains. First, why care about your moral views instead of caring about your D3 traits themselves? Why care about a (weak) statistical predictor of something when you can measure the something directly? If you have high levels of D3 traits (and not merely above average, but truly high levels), that should perhaps concern you. Why would you worry if you merely have another trait (death penalty views) that make it 10% more likely that you'll have high D3 traits?

(I should note that this isn't quite the same thing as 10% of the variability being explained, but let's put that to one side.)

Predictors are useful when we can't, at the time, measure what they're predicting. We use SAT scores to predict college GPA because we don't know what the college GPA of high school seniors is. But you have your D3 scores (or you can have them, easily), so why care about a (poor) index of them? (And by the way, SAT scores are a significantly better predictor of college GPA than even your highest correlations.)

Remember, when you only have a 0.3 correlation, *very many people* will be below average in D3 traits despite being above average in their support for the death penalty. (It's not just that the 0.3 correlation is "consistent" with that; it all but guarantees that many people in a reasonably sized sample will be above average in one and below average in the other.)

Second, why is a 10% increased risk important, in this case? Let's call someone a D3 person is they are above average in D3 traits. Say that 45% of people who are anti-death penalty are D3 and 55% of people who are pro-death penalty are D3. (Admittedly, this is only one way of defining a "10% increased risk.") Do you really care about this? Do you really care about being above the average in D3 traits? These traits vary continuously, but it's likely only at the high ends that these traits are maladaptive and pathological. And your correlations only tell you about relatively low and high scores, and scores that are relative to your sample, very few of whom may have clinically (or dangerously high D3 traits).

It also depends on your particular measures. (Right now there's a very active debate in psychology about whether above average narcissism scores are really such a bad thing.)

I guess that it seems that you want to always be on the statistically correct side of everything, in the sense of always being at a lower risk of things that in turn may predict other things... I think I can understand this preference, but I can't understand *concern* about mildly elevated risks for personality traits.

I should note that I'm a supporter of the experimental philosophy movement, but at times I see what seem to me to be facile interpretations of data. Of course, we're disagreeing (and there will always be disagreement) over what's facile and what's not. Perhaps it's that those of us who work with empirical data take them less seriously in a certain way. I don't know that taking them less seriously is always defensible, but it's interesting to speculate about the psychological bases of this kind of disagreement.

Clayton Littlejohn

I loved the post, Marcus. I just came across this paragraph from a Politico piece on Rick Perry and his supporters:

"Veterans of Sen. Kay Bailey Hutchison’s unsuccessful 2010 primary challenge to Perry recalled being stunned at the way attacks bounced off the governor in a strongly conservative state gripped by tea party fever. Multiple former Hutchison advisers recalled asking a focus group about the charge that Perry may have presided over the execution of an innocent man — Cameron Todd Willingham — and got this response from a primary voter: “It takes balls to execute an innocent man.”


Marcus, regarding your last comment about whether correlations of .2 and .3 are "good". That depends. If your are an organizational psychologist, who is trying to save a company money, then maybe. If you are doing psychophysics, then no.

Based on your correlations, the r-squares are quite small as Ben said. For your highest correlation, that is about .09. 9% of the variance in your dependent(s) is accounted for by D3 (roughly). That means that D3 allows you to predict just 9% of the variance--but presumably a lot more than that CAN be predicted (it isn't like the other 91% is all error.) A predictor that can only account for 9% of the variance is not great, in this case, based on my knowledge of stats (well, so I was taught, at any rate!)

Sorry, I feel like I am just repeating some of what Ben said...maybe this doesn't add much.

Marcus Arvan

Brendan and Ben: thanks again for the comments. Your worries are well-taken, but I can't help but wonder whether it's really all that important for my observed effects to be "strong" (though I admit I did put it that way in the paper). After all, suppose (as you both suggest) the magnitude of the correlations are better described as modest-to-moderate. Why isn't that enough for my argumentative strategy? Isn't it still a bit disturbing -- and epistemically interesting -- if there is even be a *modest* link between dark personality traits and substantive moral judgments? I think so.

Clayton: I'm glad you enjoyed the post, and thanks for the amusing anecdote. Here's something in return that you might find amusing. I came across the following quote from Robert Nozick to a reporter:

"For a while I thought: 'Well, the arguments are right, capitalism is the best system, but only bad people would think so. Then, at some point, my mind and my heart were in unison."

The thing I find most amusing about this quote is that there are at least two ways to disambiguate its main point -- one charitable (and obvious), and one terribly uncharitable (but really funny). ;)

Considering that you're looking at political views. I would think that something along the lines of Rawl's reflective equilibrium might be more useful than simply virtues. In constructing a society, we may view these three traits as undesirable in a population at large and so also view political theories that endorse them as being incorrect in our idealized society. Additionally, this allows you to link up your study with a possible theory of justice over and above what virtues might be at play and the need to make moral arguments against these views.

That's just my thinking, but the first thing that popped into my head was that one can analyze these political philosophies as either standing or falling in reflective equilibrium. Which given your study results, they may not if they correlate with these three bad virtues that we wouldn't want in a society.

Marcus Arvan

Hi James - thanks for the comments. I think those are very good ideas. I sort of had them in mind. We might not only use the data to raise epistemic doubts about the intuitions themselves, but also about the kind of people a given society is likely to produce -- something that (as you are right to point out) we might in turn use in reflective equilibrium to evaluate political principles. Thanks again - I think the idea is spot on!

The comments to this entry are closed.

3QD Prize 2012: Wesley Buckwalter