Blog Coordinator

Knobe's X-Phi Page

X-Phi Grad Programs

« Final CFP: Buffalo Annual Experimental Philosophy Conference 2015 | Main | X-Phi at the Movies »

06/05/2015

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Brent Strickland

I agree that Xphi might be in a better position than some of the other sciences, but for a slightly different reason. All things being equal, you would expect the non-replicability rate in a given field to scale with the cost of running direct replications. If it is going to take a lot of time and money to directly replicate a study (such as in a large clinical study), it seems more likely that a non-replicable finding will make it to publication. Xphi is in a fairly unique position in that the cost of direct replication is extraordinarily low since 95% of the studies are run on-line. If you find a marginally significant effect, it's really easy to just replicate it internally. So I'd also expect that if someone were to do a large scale replicability type study, they'd find a very low non-replicability rate compared to other fields.

Shen-yi Liao

That's a great additional point, Brent. In this respect, perhaps the (stereotyped) simplicity of experimental philosophy turns out to be a strength---when it comes to replicability.

(By the way, for other readers of the blog, another great example of the self-critical and self-correctional culture of experimental philosophy is Brent Strickland and Aysu Suben's "Experimenter Philosophy: the Problem of Experimenter Bias in Experimental Philosophy" available at http://philpapers.org/rec/STREPT .)

Brent Strickland

Thanks for the shout-out. It would be really interesting to do something similar to the Brian Nosek-led massive replicability project but for Xphi studies. Has anything like this been attempted yet? Given how easy these studies are to replicate, it could be done very quickly and easily, and my bet would be that Xphi would come out of this in a very positive light.

Joshua Knobe

I completely agree with Shen-yi and Brent that replication seems to play a different role in experimental philosophy from the role it plays in social psychology. Some of the difference is doubtless just a matter of logistics (it is often easier to replicate x-phi studies than social psych studies), but I also think there is something a little bit deeper going on here. Specifically, it seems like there might be a real cultural difference at work. Folks in the experimental philosophy community show a very strong tendency not to see replication as some kind of aggressive act but just as a normal part of the way research gets done.

The lion's share of the credit for this should go to Jonathan Weinberg in particular. He has done an amazing job of fostering exactly the right sort of culture here, and I know that a lot of us really admire him for it.

Shen-yi Liao

I am not sure if this falls under logistics, but I also think there's an important difference in incentives of professional structures.

As many readers know, one big issue in psychology is the so-called "file drawer effect", which means that many null results and non-replications go unpublished, which in turn means that many researchers feel incentivized to produce positive results. The incentive structure seems very different in philosophy. For example, the Gettier case from Weinberg, Nichols, and Stich (2001) has now generated two publications that are primarily non-replications: Seyedsayamdost (2015) and Kim and Yuan (forthcoming).

there are now three publications of non-replications of

Jonathan  Scott  Phillips

Nice post, Sam! I agree that this is really one of the strengths of x-phi as a field, both in terms of culture (as Josh pointed out) and in terms of directly taking up this issue in the ways you elucidated.

In line with Brent's suggestion, one thing that people in psych have started doing is running replication studies as part of methods courses. (Rebecca Saxe and Mike Frank have great courses that do this; I'm sure others do too.) I wonder what people who are teaching experimental philosophy would think about having their own students run studies which both involve a direct replication of some existing experiment in addition to some additional conditions that test out their own ideas. Seems like a nice way of having students directly contribute to x-phi while also giving them some room to make their own discoveries.

Shen-yi Liao

Jonathan: That's a great idea. It's something that's done in political science as well (cf. https://politicalsciencereplication.wordpress.com/2015/06/15/political-science-should-not-stop-young-researchers-from-replicating/ ), which has become increasingly empirical.

In other news, I know I've sung the praises of the Experimental Turk blog here before, but there's a new post that's highly relevant to replication in experimental philosophy. The headline is that replications with non-naive subjects on MTurk reduce effect sizes (for more, see https://experimentalturk.wordpress.com/2015/06/16/using-nonnaive-participants-can-reduce-effect-sizes/ ). So that's something to watch out for, in order to be fair to the replicatee.

The comments to this entry are closed.

3QD Prize 2012: Wesley Buckwalter