The State of Reproducibility in Experimental Philosophy

Over at Feminist Philosophers, Dan Hicks casually mentions in a comment:

I haven’t had too many interactions with folks interested in experimental philosophy, but based on those interactions it seems like (1) x-phi folks are modeling their methods explicitly on these kinds of studies in psychology, while (2) not being aware of or responsive to the epistemic crisis surrounding these methods in psychology. So I would encourage x-phi folks to check out the links above.

Despite some initial defensiveness, I've come to think that Hicks's remarks are actually quite reasonable for those philosophers who haven't really kept up with the nitty-gritty happenings in experimental philosophy. Having heard about the replicability crisis in psychology, it is easy to assume that experimental philosophy is susceptible to the same problems. Having not kept up with the nitty-gritty happenings in experimental philosophy, it is easy to assume that experimental philosophers are unaware of or unresponsive to these issues.

The point of this post is to dispel those assumptions.

To be clear, my aim in writing this post isn't really to pick on a casual comment, but to give a more general state of the art on what experimental philosophy, as a community, is doing and has been doing about methodological and epistemological issues related to reproducible science.

* * *

To start thinking about issues with reproducibility, we can go back to the context of Hicks's comment: a "sexy" result from a low-powered social psychology study that says you can reduce implicit bias in your sleep. The replicability crisis in psychology has, in large part, to do with with replicating findings of a similar sort. As such, having learned about the replicability crisis, a philosopher is prima facie reasonable to cast a skeptical eye toward experimental philosophy findings.

The skeptical eye, I take it, has to do with a companions-in-guilt charge. Experimental philosophy often borrows methods from (social) psychology; since there's a crisis in psychology, would it be such a surprise if experimental philosophy doesn't suffer from the same methodological and epistemological issues?

There is something to the companions-in-guilt charge, but we should clarify who the companions really are. It's not just psychology that suffers from a replicability crisis. It's also cancer research, pharmaceutics, political science, behavioral economics, etc. In other words, there is a replicability crisis in science. So, philosophers should worry about experimental philosophy... insofar as and as much as they are worried about science.

* * *

 Is there reason to be more worried about experimental philosophy compared to psychology and other sciences? Not only do I think the answer to that question is 'no', but I think we in fact have reasons to be relatively optimistic about experimental philosophy.

The experimental philosophy community has been aware of the issues that underlie the replicability crises even before their recent publicity. For a long time (in a young research program), Joshua Knobe (and later Christian Mott) has maintained the experimental philosophy replication page, which documents experimental philosophy findings that have (and have not) replicated well. Looking at the replication page, one can see that the majority of effects in experimental philosophy have, in fact, been well-replicated. (The big exception is the cluster of demographic effects.) So, not only are experimental philosophers aware of methodological and epistemological issues with replicability, they have already taken some steps to address it.

The replication page is one exemplar of a broader pattern. The community, in general, is very welcoming to cumulative research practices, which encourage internal corrections and challenges. For example, on this very blog there was an extended discussion about the replicability of an age effect. For example, Chandra Sripada and Sara Konrath shared their data so that David Rose, Jonathan Livengood, Justin Sytsma, and Edouard Machery can re-analyze it to challenge Sripada and Konrath's interpretation. For example, Adam Feltz and Florian Cova have systematically examined the moral responsibility and free will literature via a meta-analysis.

I'll let you in on a little secret. In the earlier days of experimental philosophy (and to a lesser extent even today), many people ran their online studies using Knobe's Qualtrics account. So everyone can in principle see what everyone else is working on, and even download the data if they wanted. In my view, the general culture of transparency discourages outright frauds like data manipulation or even faking data.

In addition to a general culture of transparent and cumulative research, experimental philosophers may also be in a better position than, say, psychologists because, well, they're philosophers. And philosophy has a long tradition of thinking about the relationships between experiments, statistical inference, theory confirmation, etc. Some experimental philosophers are also philosophers of science who are attentive to such issues. For example, Edouard Machery has written about the interpretation of null results. But even the experimental philosophers who don't also specialize in philosophy of science will have some exposure to these issues. And, based on the attendance at the Preconference Workshop on Replication in the Sciences at SPP, experimental philosophers are very keen to learn even more about these issues from both philosophers of statistics like Deborah Mayo, as well as methods experts from diverse cognate fields.

* * *

None of this is to say that current practices of experimental philosophy are anywhere near perfect. There is much more we can all do to do better: post data on repositories, run better powered studies, check with statistical and methodological consultants, keep up with latest best practices, etc. [I also have some thoughts about how to make the experimental philosophy replication page even more useful, but I'll save that for another occasion.] But we can acknowledge the need for improvements in the future, while also acknowledge the efforts for improvements in the past. We can recognize how far experimental philosophy still has to go, with respect to the methodological and epistemological issues associated with the replicability crises, while also recognize how far it has come.

[Much of this post is based on a talk at the 2014 Experimental Philosophy UK Workshop. I thank the organizers for giving me the occasion to think more systematically about these issues.]

Comments

[Brent Strickland]

I agree that Xphi might be in a better position than some of the other sciences, but for a slightly different reason. All things being equal, you would expect the non-replicability rate in a given field to scale with the cost of running direct replications. If it is going to take a lot of time and money to directly replicate a study (such as in a large clinical study), it seems more likely that a non-replicable finding will make it to publication. Xphi is in a fairly unique position in that the cost of direct replication is extraordinarily low since 95% of the studies are run on-line. If you find a marginally significant effect, it's really easy to just replicate it internally. So I'd also expect that if someone were to do a large scale replicability type study, they'd find a very low non-replicability rate compared to other fields.

[Shen-yi Liao]

That's a great additional point, Brent. In this respect, perhaps the (stereotyped) simplicity of experimental philosophy turns out to be a strength---when it comes to replicability.

(By the way, for other readers of the blog, another great example of the self-critical and self-correctional culture of experimental philosophy is Brent Strickland and Aysu Suben's "Experimenter Philosophy: the Problem of Experimenter Bias in Experimental Philosophy" available at http://philpapers.org/rec/STREPT .)

[Brent Strickland]

Thanks for the shout-out. It would be really interesting to do something similar to the Brian Nosek-led massive replicability project but for Xphi studies. Has anything like this been attempted yet? Given how easy these studies are to replicate, it could be done very quickly and easily, and my bet would be that Xphi would come out of this in a very positive light.

[Joshua Knobe]

I completely agree with Shen-yi and Brent that replication seems to play a different role in experimental philosophy from the role it plays in social psychology. Some of the difference is doubtless just a matter of logistics (it is often easier to replicate x-phi studies than social psych studies), but I also think there is something a little bit deeper going on here. Specifically, it seems like there might be a real cultural difference at work. Folks in the experimental philosophy community show a very strong tendency not to see replication as some kind of aggressive act but just as a normal part of the way research gets done.

The lion's share of the credit for this should go to Jonathan Weinberg in particular. He has done an amazing job of fostering exactly the right sort of culture here, and I know that a lot of us really admire him for it.

[Shen-yi Liao]

I am not sure if this falls under logistics, but I also think there's an important difference in incentives of professional structures.

As many readers know, one big issue in psychology is the so-called "file drawer effect", which means that many null results and non-replications go unpublished, which in turn means that many researchers feel incentivized to produce positive results. The incentive structure seems very different in philosophy. For example, the Gettier case from Weinberg, Nichols, and Stich (2001) has now generated two publications that are primarily non-replications: Seyedsayamdost (2015) and Kim and Yuan (forthcoming).

there are now three publications of non-replications of

[Jonathan Scott Phillips]

Nice post, Sam! I agree that this is really one of the strengths of x-phi as a field, both in terms of culture (as Josh pointed out) and in terms of directly taking up this issue in the ways you elucidated.

In line with Brent's suggestion, one thing that people in psych have started doing is running replication studies as part of methods courses. (Rebecca Saxe and Mike Frank have great courses that do this; I'm sure others do too.) I wonder what people who are teaching experimental philosophy would think about having their own students run studies which both involve a direct replication of some existing experiment in addition to some additional conditions that test out their own ideas. Seems like a nice way of having students directly contribute to x-phi while also giving them some room to make their own discoveries.

[Shen-yi Liao]

Jonathan: That's a great idea. It's something that's done in political science as well (cf. https://politicalsciencereplication.wordpress.com/2015/06/15/political-science-should-not-stop-young-researchers-from-replicating/ ), which has become increasingly empirical.

In other news, I know I've sung the praises of the Experimental Turk blog here before, but there's a new post that's highly relevant to replication in experimental philosophy. The headline is that replications with non-naive subjects on MTurk reduce effect sizes (for more, see https://experimentalturk.wordpress.com/2015/06/16/using-nonnaive-participants-can-reduce-effect-sizes/ ). So that's something to watch out for, in order to be fair to the replicatee.