Blog Coordinator

Knobe's X-Phi Page

X-Phi Grad Programs

« New paper on the philosophical role of x-phi | Main | How to Better X-Phi »



Feed You can follow this conversation by subscribing to the comment feed for this post.

John Turri

HI David,

Congratulations on the acceptance! Really great to see this. It's especially noteworthy that increasing the number of fakes didn't make a significant difference. The conventional philosophical wisdom about these cases is so wrong.

I'm less confident about the demographic finding, though. Were you able to replicate it (even if it was only in unpublished follow-up work)?

Joshua Knobe

This is a really nice study, very well-designed and with a super-surprising result. (It's pretty exciting to see these age effects on epistemic intuitions.) I would be happy to run a quick replication on mturk to see if we can get this demographic effect to come out again.

David et al., if you send me the full materials, I can run the replication and then post the results here on this blog later this week.

(By the way, if any of you are interested in general in seeing which experimental philosophy studies have successfully replicated and which have not, you can find all the results up at

Joshua Knobe

Hi all,

All the materials for the replication of this very nice study are now up at:

(Each time you go to the URL, you will be randomly assigned to one of the four conditions.)

Please let me know if there is anything at all I should be changing before I go ahead and run it. Let's hope this thing works!

Zach H.

Hey Josh,

I don't know if it is too late at this point but if you are going to run a replication, maybe add some additional control scales along with age to see if the age effect is being driven by age itself or some other variable that often correlates with age (e.g., general cognitive ability, attention to detail etc.) If age predicts gettier judgments above and beyond relevant control variables, that would make the data much stronger.

Joshua Knobe

David Colaco just wrote to me with some very helpful points about this proposed replication. Participants in Mechanical Turk studies tend to be in the 18-40 age range. I had originally been thinking that this range would be sufficient to replicate the effect, but David rightly points out that the only way to do this adequately would be to include participants in their 60's, 70's and 80's.

An inspection of the scatterplot shows that there is almost no effect of age within the 18-40 range. Participants within that range pretty much all see this as a case of knowledge. The entire effect is basically driven by the participants who are considerably older. Their responses are all over the place, yielding a mean response that is right about at the midpoint of the scale. (This may connect up with Zach's point above.)

So, sadly, just as Colaco suggests, it seems that mturk will not be appropriate for this study, and it would be necessary to use some other method. (Personally, my guess is that this is a real effect and that it would replicate successfully.)

John Turri

Just eyeballing it, it looks to me like any effect starts at 50 rather than 60. Anyway, about 5% of my murk participants tend to be ~60+, and it looks like David and colleagues had 7 participants (out of about 85) in the fake-barn condition who were 60+. So these seem to be roughly comparable, no?

Joshua Knobe

Nice point, John. So looking through the studies you’ve done on epistemic intuitions, what do you find? Is there an age effect in your studies too?

David Colaco

Thanks, everyone, for the comments so far. I would like to attempt to replicate the findings of our study. If it is the case that m-turk has an age range comparable to that of the original study, then I would be fine running a replication study using it. Repeating the original protocol is both laborious and costly; if this is a viable alternative (I have only marginal experience using m-turk), then it is worth replicating online.

Shen-yi Liao

I just want to add another data point to the interesting general discussion, even though this is obviously not an instance of direct replication.

I recently ran a study with fake barn cases on MTurk. (I had a wild idea that the protagonist name might make a difference. It didn't pan out; there was literally no difference between conditions.) The story is the same as the Swain et al. one. 7-point scale on which higher ratings corresponds to stronger attribution of knowledge.

Out of 198 participants who reported age, 13 were fifty-year-old or above (6.6%) and the maximum is seventy-three-years-old. While it's not exactly like the Colaco et al. study's sample, it does include some participants above the threshold that Turri eyeballed.

Anyway, I ran a simple correlation test between attribution of knowledge in fake barn and age. To my surprise, there was a significant, small correlation -- but it went the other way -- older participants were more likely to attribute knowledge (p = 0.027, Pearson's r = 0.157). The correlation is not statistically significant, by conventional standard, using nonparametric tests (Kendall's tau = 0.101, p = 0.061 / Spearman's rho = 0.135, p = 0.059).

Then, I used the median split procedure reported in the paper. Perhaps not too coincidentally, the median age from this data set is also around 30. Again, the 30-and-over group was more likely to attribute knowledge (M = 5.56; SD = 1.431) than the under-30 group (M = 5.17; SD = 1.615), but the difference is not statistically significant by conventional standard (t(196) = -1.776; p = 0.077).

Eyeballing the scatterplot, it seems that the results from this data set are mainly driven by a few young people not attributing (much) knowledge.

Marcus Arvan

David: I can't access the paper, but what was the sample size? Following Shen-yi a few other commenters, I can't help but wonder whether the significant results may be the result of a few extreme responses in the higher age-range -- which could well be an artifact of the sample. Sample size is important!

David Colaco

Marcus: In total, the sample included 169 participants who correctly answered the comprehension questions.

Jonathan Phillips

Just chiming in with a few additional points:

- I wanted to point out how impressive and gracious David's response (and the discussion as a whole) has been. It stands in stark contrast to almost all of the discussion surrounding replication in psychology, if anyone's been following that.

- I also think that replicating this experiment online is an excellent idea. As a couple people have pointed out the number of participants older than 50 was small, so in replicating, it would be good to collect around 2.5 * the original sample size. This has become pretty standard when replicating experiments, and luckily mturk is very cheap.

- Regardless of what happens with the replication of this one unpredicted effected, the central findings reported in David's nice paper also seem worthy of further discussion!

John Turri

Great points, Jonathan! Absolutely, much credit to David for the way it's been handled. And the team's central finding is really cool and persuasive.

Joshua Knobe

Very sorry for the delay. We have now run two replication studies, and we will be back in touch with the results shortly.

The comments to this entry are closed.

3QD Prize 2012: Wesley Buckwalter