Two LSAC responses worthy of philosophers (photo credit: Todd Lapin)
Ever see one of those “item removed from scoring” notices on an LSAT preptest? That means the question had some kind of flaw. The LSAC doesn’t release information on why the questions were flawed, but presumably more than one answer was right, or the stimulus didn’t lead to the credited answer.
You can challenge a question yourself if you think it’s flawed. Back when I was a teacher for one of the major LSAT companies, they had an ongoing project that searched for potentially flawed questions.
I’ve challenged a couple myself. In both cases, LSAC has sent me a multi-page letter explaining why the question is not, in fact, flawed. One response convinced me I was wrong. The second response I’m not certain about, but I didn’t think the question was flawed in a major way.
LSAT Preptest 43, Section III, Question 26
The first question I challenged was from preptest 43. I wrote this on March 29th, 2011:
Hello,
I’m an LSAT tutor, and have been working with the LSAT for several years. I’m writing you because I believe I have found an error in one of your questions. In general I find the logical consistency and correctness of your questions quite remarkable, which is why this particular question struck me as off.Here is the text of question 26:26.[Graeme’s note: I have removed the text to avoid infringing copyright. Please find a copy of Preptest 43]E is the credited response. A common enemy is not mentioned in the stimulus. Presumably it is an unstated assumption that allies will face a common enemy. I do not believe this assumption is warranted. Many alliances have existed without common enemies. NATO presently has no common enemy, for example. During the cold war it was also difficult to identify a single and consistent common enemy for each block of allied countries. USA vs. USSR is the cliched answer, but the actual situation amongst and between NATO and Warsaw pact countries was considerably more nuanced and fluid.As such, I don’t believe such an unstated assumption is warranted. It may seem like nitpicking, but usually I’ve found LSAT logical reasoning questions to be absolutely above reproach, and I don’t think this meets that standard.Further, there is some merit to answer choice C. I believe it is a warranted assumption that there is a military commitment between countries which are allies. According to general usage the word of alliance almost invariably refers to military matters when used to refer to nations. In the context of the Cold War, it is an irrebutable assumption.The second part of answer E“enabling them to ignoreeconomic tensions that would otherwise beproblematic.”is of course much stronger than the second half of answer choice C. But I believe that the first half of the answer choice makes the answer problematic. Though I can clearly see the intended reasoning, I do not feel it makes for a valid question, on par with other LSAT questions. I recognize that the question stem merely says “conforms most closely”, but I do not feel this resolves the issue either.This is surely a small matter, and I apologize if my explanation has been rather lengthy. But I know the LSAC takes it’s questions seriously, and has pulled questions in the past. I hope you find this information useful, and that you will consider reviewing the question.Sincerely,Graeme Blake
LSAC Reply, Preptest 43
They sent me the following response on May 11, 2011. I find it convincing. It’s an interesting view into how the LSAT uses outside knowledge:
Dear Mr. Blake,
This is in response to your inquiry about Question 26 in Section 3, of The Official LSAT PrepTest 43 (Form 5LSN67). Members of the Test Development staff at LSAC have carefully reviewed this question and determined that the credited response is the best answer. We are flattered that you believe that we maintain high standards of logical consistency and correctness, but we must disagree with you that this question does not meet those standards. The instructions for the Logical Reasoning sections of the LSAT indicate: “For some questions, more than one of the choices could conceivably answer the question. However, you are to chose the best answer; that is, the response that most accurately and completely answers the question.”
Question 26 reads:
In the aftermath of the Cold War, international relations between Cold War allies became more difficult. Leaders of previously allied nations were required to conduct tactful economic negotiations in order not to arouse tensions that had previously been overlooked.
The situation described above conforms most closely to which one of the following propositions?
The credited response is:
(E) A common enemy contributes to a strengthened bond between nations, enabling them to ignore economic tensions that would otherwise be problematic.
The passage of Question 26 describes a particular state of affairs following the Cold War and each of the responses is a general statement. You are asked to identify the general proposition to which the particular case most closely conforms. The situation is that, following the end of the Cold War, leaders of nations that had been allied in the Cold War had to conduct careful economic negotiations to avoid tensions that had been previously (during the Cold War) overlooked.
This situation in the passage conforms closely to the generalization in (E). The nations are described in the stimulus as Cold War allies. As such they were united in opposition to the opposing coalition, either the East Bloc or the West Bloc and its dominant power. In each case that opponent was the common enemy that was the raison d’être for the alliance. The passage suggests that while the Cold War was going on that alliance allowed the allied countries to ignore economic tensions that otherwise might have arisen. However, following the Cold War, the former allies were no longer so strongly united by rivalry with their former common enemy. In that case the economic tensions among the former allies could no longer be overlooked. Thus, (E) provides a reasonable account of the situation described in the passage.
You argue that response (E) is not a satisfactory answer because the stimulus does not mention a common enemy. For this reason you believe that the credited response requires the assumption that an alliance must have a common enemy and you argue that this assumption is unwarranted. However, this question is not about any or every alliance. It is specifically about Cold War allies. And it is common knowledge that the Cold War was a bilateral rivalry between two major powers and their respective alliances. Since this rivalry was generally hostile and sometimes escalated to the brink of violence, it is fair to characterize the members of each bloc as sharing a common enemy. You claim that “during the cold war it was also difficult to identify a single and consistent common enemy for each block of allied countries.” But this seems to us to mischaracterize the situation and is at odds with the way it is commonly described. Writing and discussion about the cold war ordinarily takes for granted this presumption that each side regards the other as a common enemy. You aver that identifying the US or the USSR as the common enemy is clichéd and that the actual alliances were more nuanced and fluid. But, while it may be the case that the memberships of the alliances might shift and that many countries might drift in and out of the orbit of one or the other major powers, playing one against the other, this does not undermine the fact that the Cold War was a confrontation between two major powers, the US and the USSR, and their military, political, and economic alliances. Thus, it is both commonplace and sensible to describe the members of each Cold War alliance as being united against a common enemy, even though the level of hostility might heat up or cool down over the course of the years and the relationships of many nations to the two alliances and the leading nations in them might often be complicated.
As a result, the credited response to Question 26 does not depend on an unstated assumption that every alliance has a common enemy. Rather it trades on the common understanding that the Cold War was a period of rivalry and confrontation between two opposing blocs, and that it is fair to say that the members of each bloc regarded and treated the opposing bloc as a common enemy. Moreover, as you acknowledge, the second half of the credited response fits the situation described in the stimulus very well. So, the credited response is the best answer to the question.
You further argue that response (C) has some merit as an answer. Response (C) reads:
(C) When there is a military commitment between countries, fundamental agreement between them on economic matters is more easily reached.
You argue that “it is a warranted assumption that there is a military commitment between countries which are allies.” We do not dispute this. However, as you concede “the second part of answer E…is of course much stronger than the second half of answer choice C.” Indeed, the second part of response (C) is not really conformed to by the situation in the passage. The situation in the passage is not about fundamental agreement, but about economic tensions that can no longer be overlooked. The passage does not indicate that the Cold War allies were in or negotiated to reach fundamental agreement on economic matters, but only that they overlooked economic tensions, which is a much different state of affairs. Indeed, overlooked economic tensions would seem to be incompatible with fundamental economic agreement between parties. The passage indicates that after the Cold War the allies “were required to conduct tactful economic negotiations in order not to arouse tensions…” This does not indicate that they reached or were trying to reach fundamental economic agreement. So there is nothing in the passage to indicate that they were trying to or able to reach fundamental economic agreement either during or after the Cold War. There is, then, considerable disparity between the general statement in response (C) and the case described in the stimulus. On the other hand, the situation in the passage conforms very closely to the credited response (E), given only the general and commonplace understanding of the Cold War as a bilateral rivalry between opposing blocs of allies the members of which regarded the opposing bloc as a common enemy.
We hope that this addresses your concerns regarding this question. Thank you for your high regard for the standards of the LSAT and for endeavoring to help us maintain them.
Sincerely,
Stephen W. Luebke
Principal Test Specialist
LSAT Preptest 64, Question 25, Section 3
My second (and final) challenge was to question 25 of preptest 64, section III. In this case, I am still not entirely convinced by the response. But, I don’t think the issue I raised is particularly damning to the question, either. Whether you accept my view or the LSAC’s view is a matter of tendencies the test has shown, nothing more. There’s nothing per se incorrect about their reasoning. I just didn’t think they’ve done things like this in the past.
My original question is below. You can find more details on this thread I made at Top Law Schools: http://www.top-law-schools.com/forums/viewtopic.php?f=6&t=209540
Original question:
Hello,
I believe I have found a flaw in LSAT 64, section III, question 25. The one about the trade union strike. The stimulus says arbitration would avert a strike. This is a sufficient condition.
The argument then shows that arbitration is unlikely. So it has proven that one sufficient condition for averting the strike is unlikely.
However, the argument then concludes that the strike is unlikely. This is incorrect. The planned strike could still be averted for other reasons, such as government intervention, an excellent offer on the part of the employer, etc.
The credited answer, D, parallels the stimulus in its probabilistic conclusion. However, the argument in D is a good argument. It has shown that Lopez is unlikely to be hydrated by his team. Thus he is unlikely to win.
I do not believe that a good argument can parallel a flawed argument. Thus I believe the question is flawed, and should be withdrawn from future prints of LSAT 64.
Please let me know whether you agree with my reasoning.
LSAC Reply, preptest 64
The LSAC sent a very detailed response. You can see it in it’s original format in this pdf:
https://lsathacks.com/wp-content/uploads/2013/06/LSAC-Response-LSAT-64.pdf
Here’s the text of the reply:
Dear Mr. Blake:
This is in response to your email communication dated May 13, 2013 regarding question 25 in Section 3 of The Official LSAT PrepTest 64. Members of the Test Development staff at LSAC have carefully reviewed the question and did not find it to be defective.
Question 25 reads:
Journalist: The trade union members at AutoFaber Inc. are planning to go on strike. Independent arbitration would avert a strike, but only if both sides agree to accept the arbitrator’s recommendations as binding. However, based on past experience, the union is quite unlikely to agree to this, so a strike is likely.
Which one of the following arguments exhibits a pattern of reasoning most similar to that exhibited by the journalist’s argument?
The credited response is option (D):
(D) Lopez will run in tomorrow’s marathon. Lopez will win the marathon only if his sponsors do a good job of keeping him hydrated. But his sponsors are known to be poor at keeping their athletes hydrated. So it is probable that Lopez will not win the marathon.
While you appear to agree that option (D) is the best answer to the extent that, compared with the other options, the reasoning in (D) is the most similar to that in the stimulus argument, you contend that the stimulus argument is flawed whereas the argument in (D) is not, rendering the two arguments fundamentally non- parallel. Thus, you believe that the item itself is flawed.
Your contention that the stimulus argument is flawed appears to be based on the assumption that the journalist’s argument can be adequately analyzed as simply a deductive argument. You suggest that whereas the argument demonstrates that one sufficient condition (arbitration) for a particular outcome (no strike) has been shown unlikely to hold, the argument does not establish that the outcome itself is unlikely since there may be other conditions under which that outcome could hold. Thus, you maintain, the argument’s conclusion does not follow deductively from the given premises (it is not a valid modus tollens argument, as you take the argument in (D) to be).
However, the journalist’s argument cannot be adequately analyzed as simply a deductive argument. It is an example of a reasonable informal, nondeductive argument that draws a conclusion about the probability of an outcome based largely on probabilistic reasoning. In analyzing this argument, there are certain features that should be taken note of. First, the argument is made by a journalist. In the context of journalism, it is a reasonable application of the “principle of charity” in argument interpretation to presume that the information provided by the journalist constitutes a relatively complete picture of the relevant facts. While there might be other conditions under which the strike could be averted, the journalist apparently does not think them important or likely enough to mention. The argument is not about them. Second, the argument begins with the statement: “The trade union members at AutoFaber Inc. are planning to go on strike.” In informal, everyday English, an unqualified assertion of the form ‘X is planning to do Y’—Sam is planning to leave work at 4:00 today, The teacher is planning to give an exam next week, The mayor is planning to run for reelection—invites the inference ‘It is likely that X will do Y’. Accordingly, in the case of the journalist’s argument, it is reasonable to infer from the first statement in the argument that a strike is likely. Note that this inference is reinforced by
subsequent use of the word “avert” in relationship to the strike: ‘to avert an outcome X by doing Y’ suggests that X is likely unless some positive action Y is taken to prevent X.
The argument identifies a sufficient condition under which a strike would be averted: a strike would be averted by independent arbitration, provided that a necessary condition holds, namely, that both sides agree to accept the arbitrator’s recommendations as binding. It is argued, however, based on past experience, that this necessary condition is unlikely to hold. Thus, the argument concludes, a strike is likely. The pattern of reasoning in the argument can be characterized as follows:
S is likely.
If A, then not S.
A only if B.
B is unlikely.
(So, A is unlikely.) Therefore, S is likely.implication of first statement
[based on past experience]
implicit inference, intermediate conclusion
S = union members will strike
A = arbitration
B = both sides accept arbitrator’s recommendations as binding
Notice that the journalist’s argument is not invalid or a bad argument. It does not, as you suggest, require one to infer that a strike is likely merely from the claim that arbitration is unlikely (a sufficient condition doesn’t hold). The conclusion that a strike is likely is inferred from this claim together with the fact that union members are planning to strike. The journalist’s argument contains a pattern of reasoning like a deductive modus tollens argument (lines 3 through 5), but its conclusion does not follow from that reasoning alone. It is based on an additional non-deductive inference that is reasonably well-supported.
Your observation that a strike might be averted for some other reason—”government intervention, an excellent offer on the part of the employer, etc.”—has no bearing on the issue of whether the argument, as given, is a cogent one. The argument is not strictly deductive and the mere possibility of other conditions sufficient to avert the strike does not by itself undermine the nondeductive evidence that a strike is likely.
Finally notice that the pattern of reasoning in lines 3 through 5 of our analysis of the journalist’s argument is highly similar to the pattern of reasoning in response (D). The argument in (D) concludes that Lopez is unlikely to win the marathon based on the observation that a necessary condition for Lopez’s winning is unlikely to hold:
A only if B.
B is unlikely. [based on past experience] So, not-A is likely. (A is unlikely.)Although it is true that response (D) does not contain all the elements of reasoning found in the journalist’s argument, it does exhibit a pattern of reasoning that is highly similar to the reasoning exhibited in a central portion of the journalist’s argument. No other response exhibits a pattern of reasoning as similar to that exhibited in the journalist’s argument. Thus, (D) is the best answer.
We hope that this answers your concerns. Thank you for your interest in the LSAT.
Sincerely,
Stephen W. Luebke Principal Test Specialist
I found the bit about the journalist interesting. The LSAC is asking us to apply outside knowledge of what a good journalist would do. I find this specific application tenous, but the “principle of charity” is good advice. I sometimes see LSAT students twist arguments into crazy interpretations. Don’t do that. Assume the speaker isn’t a complete idiot. Take the sensible interpretation of what they say. You should be precise about what they say, but don’t give their words a silly interpretation.
Don’t take these responses too seriously
I posted these just for your own curiousity. A few important points:
1. If you think a question is wrong, it is *waaaaay* more likely that you are wrong. Check with other people first before writing LSAC.
2. You *don’t* need to think at this level of detail to get questions right. Don’t worry if the LSAC responses don’t make sense, particularly the second. I’m posting these for your own interest, and also to show you what depth is present in even a single LSAT question.
Hi, I'm Graeme Blake
Founder of LSAThacks; I scored a 177 on the LSAT.
I created the LSAT Mastery Seminars, which are part of the Ultimate LSAT Prep Bundle
If you like the explanations you're reading I'm 100% confident you'll enjoy the courses. Satisfaction guaranteed.
---------
Updates:
For updates, sign up for my email list. I update whenever I have new posts.
Wow–yes, I think you were right–I find their discussion of different types of reasoning challenging. We have to guess whether the source is using strict reasoning or probabilistic based on our reasonable assumptions? So different logics are used inconsistently and we need to use our intuition to guess which applies?
This is fantastic, well done sir.
I just took PT 64, and had exactly your same concern on question 25.
I said to myself “the journalist’s argument is flawed — he deduces that a sufficient condition is unlikely, and uses that to determine that the outcome is unlikely, whereas answer choice E is not flawed — it states that a necessary condition is unlikely, and uses that to determine the outcome is unlikely”.
And from there, I said how can a flawed argument really be particularly similar to an unflawed one?
I mean, I know the question asks “which pattern of reasoning is *most similar*”, not which one is “*exactly the same*” but still — flawed vs unflawed seems like a pretty big difference.
I ended up debating between B and E, as I saw both of them as flawed (B deduces a necessary condition will occur, and takes that to mean the outcome will happen, E fails to take into account that the situation regarding instructor availability could change by the spring). I didn’t think either was a good match to the pattern of reasoning, but went with B.
I can see where the LSAC is coming from — the pattern does match more with D, but that feels like a superficial similarity given the underlying difference in validity of arguments.
I think it’s arguable that their answer is the best, but only arguable — not indisputable. So I’d say this puts the lie to the claim I’ve seen in a few places that there is always one clearly correct response and 4 clearly incorrect ones. And serves as a good reminder not to waste too much time on a question, because the question could be flawed (rare, but not impossible). I ended up with no time for Q26 because I lost track of time on Q25, and missed them both (and Q26 while long was pretty straightforward — the answer match what I would’ve expected).
Anyway, I googled this question to see if the various sites were discussing this issue. You seem to be the only one who did.
Thanks! Glad you appreciated it. I wouldn’t say their reply completely convinced me for that question. However, I can say their discussion of the principle of charity *greatly* informed my interpretation of LSAT questions, and has proven very useful in getting through answers faster.
Note: This is an old comment but I wanted to address the point.
At first I disagreed with you on the second question, based upon the fact that “only if” language precedes a consequent, not an antecedent. But then I recognized the primary difference between the stimulus and the credited response. The stimulus can be represented by the phrase “If independent arbitration averts a strike, then both sides agree to accept the arbitrator’s recommendations as binding.” Modus Tollens only gets you “Independent arbitration will not avert a strike.” In contrast, the credited answer can be represented as “If Lopez wins the marathon then his sponsors do a good job of keeping him hydrated.” Modus tollens gets you, “Lopez will not win the marathon.” There’s a big difference between “Independent arbitration will not avert a strike” and “there will not be a strike.” That being said, I don’t have immediate access to the alternative answer choices. My guess would be that this is a question testing modus tollens and the operation of “only if” as preceding the consequent as opposed to the antecedent.
Per LSAC they actually don’t analyze it purely as a deductive argument. They say this: “Your contention that the stimulus argument is flawed appears to be based on the assumption that the journalist’s argument can be adequately analyzed as simply a deductive argument”
Their full explanation taught me a lot about how they use relevant authority.
Note: This is an old comment but I wanted to clarify the point.
Thanks for posting these. Interesting stuff!
I do find the consistency of the LSAT to be ridiculous, I can’t imagine how whoever rights these things functions in the real world when they have to be so perfectly consistent in their own formulation of the questions. Every time I get a LR wrong I go back and see that the correct answer just fits better. To be that logical and then read the average post… ah what a curse.
I’ve noticed what you’ve mentioned in your analysis of the second response a few times. It seems often you have to take some basic assumptions into account that in the moment I end up thinking ‘well really we have to assume this likely fact too’, but generally these get sorted out by the whole “best answer thing”. I’ve seen several ‘non-perfect’, but easily best, answers.
Some of the parallels I’ve done aren’t as fundamentally different (depending on your interpretation) as the one you posted about, but have different numbers of components or are otherwise different in a way that always makes me second guess myself haha.
Again really cool that they actually take the time to do this. Maybe I should challenge them on every question under different names and addresses until I can publish an LSAT test book entirely composed of LR Analysis (3 page per question of course)…