Blog Coordinator

Knobe's X-Phi Page

X-Phi Grad Programs

« Free Will, Dualism, and Intentionality | Main | FSU Free Will Project »



Feed You can follow this conversation by subscribing to the comment feed for this post.

Joel Pust

Isn't this pretty much just attending to the distinctions which Kirk Ludwig (2007) and others have pointed out are part of traditional philosophy of language - between speaker meaning and sentence meaning for example?

Justin Sytsma

Hi Joel,

Actually, our paper focuses on a distinct ambiguity from the speaker’s reference / semantic reference ambiguity that Ludwig discusses (and that was, in fact, noted by Kripke himself in talking about the Gödel example). Thus, we note two distinct ambiguities that can be found in Machery et al.’s test question. Beyond the speaker’s reference / semantic reference ambiguity, there is an epistemic ambiguity in the test question. Specifically, the question used in their Gödel probe does not clearly indicate whether the (A) and (B) answer choices are to be read from the narrator’s epistemic perspective (the narrator relaying information of which John is ignorant) or rather from John’s epistemic perspective (as the speaker using the name ‘Gödel’). The reason that this is a problem is that the epistemic perspective that is adopted is relevant to deciding who the descriptions given in the answer choices denote, raising the possibility that different participants might associate the same description with different people from the story. While Machery et al. expect the descriptions to be read from the narrator’s perspective, the question might plausibly lead participants to instead adopt John’s perspective. The result is that it is possible for a participant to think that when John uses the name ‘Gödel’ he is talking about Gödel and nonetheless for that participant to legitimately answer (A) because she reads that description from John’s perspective as denoting Gödel.

The four studies that we ran to test the effect of ambiguity on participant responses focus on the epistemic ambiguity. Nonetheless, as our goal was simply to test whether Machery et al.’s results reflect ambiguity in their test question, we did not attempt to distinguish between these two types of ambiguity experimentally. It is therefore possible that in attempting to clarify the epistemic ambiguity, we also clarified the speaker’s reference ambiguity to some extent.

Jonathan Livengood

Also, even if we have made no conceptual advance with our work, we extended the dialectic in a way that Ludwig did not. Namely, we tested whether people *actually* find the probe to be ambiguous, as opposed to merely pointing out that people *might* find the probe to be ambiguous.

Insofar as there is something different between what is possible and what is actual, what we did is different from what Ludwig did.

Ron Mallon

This is a good paper. But study 4 only shows correlation, not causation. It relies on the eminently questionable assumption that subjects are introspectively accessing the logical form of their encoding of the original probe and using the knowledge they acquire to answer the study 4 probe.
That could be correct, but it seems to me that there’s plenty of reason to be skeptical. First, there are well known examples of failures of introspective access in psychological experiements. Second, the correlation may have another explanation. For example, subjects may want to choose a response to experiment 4 that rationalizes their response or “sounds like” their real reason. Third, the design offers subjects a forced choice between options that presupposes the theory that is being tested.
On the last two points, suppose for a moment that subjects do not find the original probe ambiguous. It never occurs to them that “who John thinks he is talking about” and “who John is actually talking about” might be different. Suppose the form in which they encode the the question is: “WHEN JOHN USES THE NAME “GÖDEL”, IS HE TALKING ABOUT…” And suppose, as Machery et al. suggest (without quite endorsing), subjects answer the original question using either a descriptive or causal historical theory of names.
If so, when they get the study 4 probe, they are offered a forced choice which doesn’t allow them a chance to report their actual reasons. In that case, you might expect subjects to then choose an answer which is related to their reason or response, either by “rationalizing” the response, or by “sounding more like” the reason (if they can access it). If so, you’d expect to get precisely the result S&L got, but their account of their result would be wrong.

Jonathan Livengood


Thanks for the comment. I have two questions. First, I'm not sure I understand how your "rationalizing" explanation predicts the result we saw in our Study 4. Could you elaborate a bit? I'm concerned here because the main reason I like that experiment is that we had a clear prediction, which was satisfied, and I couldn't convince myself that you guys had a plausible alternative that predicted the pattern of results we saw. Second, what do you make of the consilience argument: Studies 1-3 are all very similar ways of getting at the question and give an answer that supports our position; Study 4 gets at the question in a different way and at the very least can be interpreted as agreeing with the results of Studies 1-3; doesn't that combination lend more support to our conclusion? Shouldn't a challenge to our collection of studies give a joint reply by showing how they all fail for similar reasons? Or do you think that is an unreasonable demand?

Ron Mallon

Hi Jonathan!
If a rationalizing explanation were right, the important point would be that when a subject answers (1) they take it to rationalize (A) and when they read (2) they take it to rationalize (B).

Consider that your own perspective studies (John’s perspective and narrator’s perspective) show that subjects take (1) “John thinks he is talking about” to indicate (A)“the person who really discovered the incompleteness of arithmetic” while they take (2) “John is actually talking about” to indicate (B) “the person who got hold of the manuscript and claimed credit for the work.” On this skeptical line, all the fourth study does is ask them to reconstruct this backwards: it has them offer an answer to the original probe, and then they answer the second probe with the answer that would justify the first answer. And your own studies suggest that ordinary subjects do take there to be a rational connection (whether or not I can reconstruct it correctly).

Here’s one way in might go. So, suppose you’re doing the study and you get the first question and answer the question:
“When John uses the name “Gödel”, is he talking about:…”


“(A) the person who really discovered the incompleteness of arithmetic?”

Then you get the second question which (on this line of thought), raises an distinction that you perhaps haven’t considered, and the mere question raises the possibility that John is in error about who he is talking about.
So you think, well, gosh, I said John was talking about the person who “really discovered the incompleteness of arithmetic.” And that is who John thinks he is talking about – indeed, it nearly specifies the entire content of his beliefs about “Gödel.” Since my earlier answer commits me to “who John is thinking about” being who John is talking about, I answer (1).

In contrast, , suppose you answered (B):

(B) the person who got hold of the manuscript and claimed credit for the work?

And now you get this question that again implies John may actually be talking about someone different than the person he thinks he is talking about.
As with the alternate condition, now that you think about it, you think that John “thinks” he is talking about (A) (since A specifies the content of John’s beliefs). But you didn’t answer (A), you answered (B), so it must be that the right answer for you here is (2), since the only way for (B) to be right is for it to be not the person who John thinks he is talking about and the only other alternative is (2). (Moreover, your own narrarator’s perspective study suggests subjects will think that interpreting the question in way (2) would mean that (B) was the right answer.

In short, when study 4 introduces a new distinction to subjects, it can lead people to reinterpret the original question (which is a way of saying what your “John’s perspective" and "narrarator's perspective" probes show – viz. that subjects map on “John is thinking about” to “the person who really discovered the incompleteness of arithmetic” and “John is actually talking about” onto “the person who got hold of the manuscript…”). But that doesn’t mean that the distinction between who John thinks he was talking about and who John actually is talking about was lying implicit in the population all along, explaining the original result. It only means that the population can reconstruct that reasoning from answer to probe in just the way they construct it from probe to answer.

I don’t know any reason why that story couldn’t be the right one, and it doesn’t rely on introspecting one’s reasons for choosing in the first task in the way your explanation does. I know you (Jonathan) and Justin were skeptical of this line of reasoning at SPP (though I didn’t attempt to make it explicit in the time available there). It is, anyway, only one way of making out the rationalizing line (there could be others), which is (as I noted) only one way of making the argument that what you’ve got is correlation not causation.

With regard to your second question, I agree that your other studies do and should give pause to those who would like to explain the original result by appeal to distinct theories of reference. And that’s because you introduce a possible confounding factor and show that it can “move the subjects around” in ways you predict. And that’s pretty cool. And the fact that this study comes out to be consistent with that hypothesis counts in favor of the hypothesis for it could have been that Study 4 would not have come out the way you predicted. But I know that a number of philosophers are impressed by Study 4 especially, and my point in raising questions about it is only to note both that other empirical possibilities remain open and that study 4 itself relies on a specific assumption about introspective access that there is some empirical reason to doubt.

Now that I've gone on too long, I'll stop, and let you tell me if I've gone wrong somewhere!

Justin Sytsma

Dear Ron,

If I understand, the issue is what degree of additional support for our hypothesis the fourth study offers (if any). It sounds like you think it offers some additional support, but are skeptical about just how much. I think that that is something that Jonathan and I can live with. Nonetheless, we think that it offers more support than you do.

Some quick clarification is in order about what this study supports: The skeptical line could be taken with regard to whether the fourth study supports our line that epistemic ambiguity explains much of the variation in the original Gödel probe results or the skeptical line could be taken with regard to whether the fourth study provides support for our claim that the original Gödel probe doesn’t get at semantic intuitions. I think that the fourth study is strong support for the latter, weaker support for the former. The latter, however, is what we primarily wanted to show, so I’ll focus on that version of the skeptical line in what follows.

It seems that the skeptical line might also be extended to our first three studies (and considering this might cast light on why Jonathan and I are skeptical about the latter type of skeptical line for the fourth study). That is, it might be argued that we have not shown that participants’ responses to the original Gödel probe question do not by and large reflect their semantic intuitions, but that they can be pushed around, as you put it, by adding further cues to the question that suggest an alternative way of reading the question. The problem, here, I think is that the skeptical line would predict that people would only be pushed in one direction: When we add the clarifications emphasizing John’s perspective, the percentage of (B) answers would be predicted to decrease; but, when we add the clarifications emphasizing the narrator’s perspective, it is not clear why these would be taken to suggest an alternative way of reading the question if the original question was actually getting at people’s semantic intuitions, so it does not seem that an increase in the percentage of (B) answers would be predicted.

This is more plausible for the fourth study, however, as on the second page participants saw both reformulations of the test question and the comparison with the John’s perspective reformulation could suggest the alternative way of reading the original question. (Note, though, that we also ran the probes from the first three studies within-subjects and that the numbers were largely the same, which indicates that just seeing the John’s perspective formulation does not make a difference.) So, the question is whether seeing the John’s perspective formulation on the second page leads people to read the question in a new way, rationalizing the answer that they gave on the first page in terms of the new reading of the question. If this is correct, then it seems to me that the issue is not so much whether subjects rationalize in giving their answers to the second question—as chances are that they hadn’t fully thought out their reasons for answering the way they did on the first question—but what that rationalizing reflects.
Now if we take the skeptical line to be such that you are committed to the view that the original question did reliably get at participants’ semantic intuitions, then we can assume that each interpretation takes a stance with regard to what is driving participants’ responses to the original question: Their semantic intuitions or ambiguity (and specifically the epistemic perspective that they take in reading the question).

Doing this, it strikes me that our interpretation is quite a bit more plausible. The reason is that if people’s answers to the original question reflect their semantic intuitions, then it isn’t clear to me why the second question would push them to read the test question in a different way in rationalizing their answer: It seems that the (2) response would be good regardless of how they answered the first question. I take this to push the advocate of the skeptical line to have to say that people’s answers to the first question reflected their semantic intuitions, but that nonetheless they weren’t very clear on that fact or didn’t really know why they answered how they did; the second question then pushes them to reflect on their answer to the first question and when they do so they tend to rationalize their answer in terms of a different reading of the first question than they actually used in answering it. The issue for me, then, is that the advocate of the skeptical line needs to say that given the rather minimal prompts in the second question, people quickly pick up on the alternative reading of the first question. But, if that is the case, then given that the original question can also be read in either way, it seems natural to expect that many people would read it in the way that we suggest.

Put another way, if people’s responses to the original Gödel probe question do track their semantic intuitions, then there should be some presumption that they will continue to understand the question in that way when they read the second question, since the second question can be read in a manner fully consistent with that and since the prompts that might suggest an alternative reading are rather minimal. Thus, on the skeptical line we would expect to see no difference in the likelihood that a participant answers (1) or (2) on the second question conditional on their response to the first question. But, that isn’t what we found. Looking just at those people who answered (A) on the original question, MMNS propose to take these people to be descriptivists. But, we see that for those people the response, “I thought I was being asked about who John thinks he is talking about,” is about three times more likely than the alternative. If the subjects who answered (A) initially really are descriptivists, then it isn’t clear to us why they should change their reading of the original question and, thus, why they should prefer the “John thinks” response.

The comments to this entry are closed.

3QD Prize 2012: Wesley Buckwalter