- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates how LLMs handle mathematical word problems containing impossible quantities such as negative physical quantities. The authors argue that models may recognize a contradiction but still introduce assumptions or adjustments that make the problem make sense.
They construct a benchmark with three conditions: violating (V), same-legal (S), and transfer (T), distinguishing problems where repair is permitted from those where no valid repair exists.
Edit: Following the author response, i have updated my score because my concerns where largely addressed
Critical strengths
-
The paper distinguishes recognizing a contradiction from responding appropriately to it. This distinction is potentially important for evaluating LLM reliability beyond ordinary answer accuracy.
-
The paper includes extensive evaluation
Critical weaknesses
-
L028 suggests that newer generations do not close the gap, but the Sep 2026 experiments omit several models that were available by experiment date, including GPT-5.6, Kimi K3, GLM-5.3, and stronger Qwen3.8 variants. Notably, GPT-5.6 is used as a judge but not evaluated as a solver. Comparisons among older, size-matched models do not adequately support conclusions about the strongest contemporary systems
-
L085–L087 controls do not isolate the proposed failure mechanism. Changing an input to make a problem valid can allow shortcuts that produce correct answers without checking intermediate computations (see figure 20 which has skipped intermediate computation). The transfer control also changes the question and potentially the reasoning strategy. Consequently, differences in accuracy may reflect reasoning or computation differences rather than sensitivity to impossible quantities alone. Figure 19 also shows an additional failure mode not controlled for and which is mislabeled as unlicensed when the model completely misunderstood the question.
-
The main behavioral classifications rely on a single LLM judge despite only moderate agreement with human consensus for resolution strategies (0.492).
-
The paper does not adequately motivate the practica signifance of the proposed failure mode
Questions for authors
see weaknesses.
clarify the compensation offered to human annotators and how they where recruited
8 comments
Response to Reviewer (Part 2/5)
W2. "L085–L087 controls do not isolate the proposed failure mechanism. Changing an input to make a problem valid can allow shortcuts that produce correct answers without checking intermediate computations (see figure 20 [...]). The transfer control also changes the question and potentially the reasoning strategy. Consequently, differences in accuracy may reflect reasoning or computation differences rather than sensitivity to impossible quantities alone. Figure 19 [...] is mislabeled as unlicensed when the model completely misunderstood the question."
A2. The gap persists among responses that explicitly report the critical intermediate value: regime-I accuracy is 95.2% on same-legal items, 65.1% on transfer items and 22.8% on violating items. Of the errors on violating items, 79.3% occur in responses that explicitly acknowledge the conflict.
The two controls. The same-legal item holds the scenario, the quantity and the constraint fixed and changes one numerical input. The transfer item holds the critical intermediate value fixed and changes what it represents (Table 1). We therefore checked whether the accuracy differences persist when responses explicitly report that value.
Explicit intermediate computation. The critical step is the step whose value is impossible in the violating item, and we checked whether its value appears in the response. Among correct same-legal responses, 89.2% explicitly report it. Accuracy among the responses that report the value:
| Same-legal | Transfer | Violating | |
|---|---|---|---|
| Regime I | 95.2% | 65.1% | 22.8% |
| Regime R | 93.1% | 90.6% | 78.8% |
Among these responses, the regime-I gap is 72.4 percentage points (pp) to the same-legal control and 42.3 pp to the transfer control, against 63.2 pp and 30.3 pp over all responses. The gap therefore does not come from shortcuts that skip the intermediate computation.
Where the errors occur. Error rates, as a share of all scored responses of the condition:
| Violating items, all errors | after acknowledging the conflict | without acknowledging it | Same-legal items, all errors | |
|---|---|---|---|---|
| Regime I | 76.9% | 64.1% | 12.8% | 13.7% |
| Regime R | 21.1% | 13.2% | 7.8% | 9.3% |
The excess errors are concentrated in responses that acknowledge the conflict. These account for 64.1% of all regime-I violating responses. Errors without such acknowledgment account for 12.8%, compared with 13.7% total errors on same-legal items.
Figures 19 and 20. Neither response acknowledges the conflict. Figure 19 is labeled PREMISE_REWRITE for its altered reading of the problem, and its monitoring field reads NONE, which separates it from the sequence in which a model acknowledges the conflict and then repairs. Neither example enters the conditional repair estimate of 94.8% or the numerator of the 60.7% full-sequence estimate. Both remain in the regime-I denominator and in the accuracy statistics. Figure 20 shows the shortcut concern addressed above: the gap persists among responses that explicitly report the critical intermediate value.
Together, these analyses strengthen the paper's finding: the gap persists when the critical intermediate value is explicitly reported, and most violating-item errors occur after the response acknowledges the conflict. We will add them to the final version and replace Figure 19 with an example that explicitly states the conflict.
Response to Reviewer (Part 3/5)
W3. "The main behavioral classifications rely on a single LLM judge despite only moderate agreement with human consensus for resolution strategies (0.492)."
A3. The behavioral findings persist under two further judges from other providers and under independent human annotation. On the same 4,703 responses (covering all 17 models and all 50 families), the three judges estimate the two regime-I proportions of Section 7 at 86.9% to 94.8% (unlicensed repair among incorrect responses that acknowledged the conflict) and 58.7% to 62.7% (full sequence). The annotators' labels give 79.9% and 57.4%.
0.492. The κ of 0.492 concerns exact agreement across 13 resolution classes. The findings use a coarser label built from these classes, the unlicensed-repair group of Table 11, on which the paper's judge gives the annotators' label for 88.1% of the annotated responses. We assess this label directly below with two further judges and with the annotators' labels.
1. Does the finding depend on one judge? The findings do not depend on one judge. We re-scored 4,703 responses from the paper's 17 models with two further judges from other providers, Qwen3.8-Max and Claude Opus 5.5, using the paper's rubric.
Table 2. Agreement between the three judges on these responses. Same label: share of responses on which the two judges give the same label. Ranking: Spearman ρ between the two judges' per-model results over the 17 models.
| Paper's judge and Qwen3.8-Max | Paper's judge and Claude Opus 5.5 | Qwen3.8-Max and Claude Opus 5.5 | |
|---|---|---|---|
| Same label: answer correctness | 91.2% | 93.1% | 93.5% |
| Same label: conflict acknowledgment | 92.9% | 93.2% | 96.1% |
| Same label: unlicensed repair, yes or no | 90.3% | 90.7% | 91.3% |
| Ranking: accuracy on violating items | 0.96 | 0.97 | 0.97 |
| Ranking: gap S−V in regime I | 0.88 | 0.82 | 0.86 |
| Ranking: full-sequence rate | 0.98 | 0.97 | 0.94 |
Table 3. The findings under each judge, on the same responses. (1): unlicensed repair among incorrect responses that acknowledged the conflict. (2): full sequence among violating responses. Gap: S−V in percentage points (pp).
| Paper's judge | Qwen3.8-Max | Claude Opus 5.5 | |
|---|---|---|---|
| (1), regime I | 94.8% | 94.1% | 86.9% |
| (2), regime I | 62.7% | 59.8% | 58.7% |
| Gap, regime I | +71.1 pp | +61.3 pp | +71.6 pp |
| Gap, regime R | +12.9 pp | +13.1 pp | +12.2 pp |
| Models with gap I > gap R | 17 of 17 | 16 of 17 | 17 of 17 |
Any two judges give the same label on 90.3% to 96.1% of the responses. Under all three, the regime-I proportions are at least 86.9% for (1) and 58.7% for (2), and the gap is at least +61.3 pp in regime I against at most +13.1 pp in regime R. Per-model results are in Part 5.
2. Does independent human judgment support the finding? The annotators' labels alone reproduce the finding. Three independent annotators labelled 400 responses. The same two proportions, computed from their labels alone:
Table 4. Proportions (1) and (2), defined as in Table 3, from the annotators' labels, beside the paper's values. Annotated responses are weighted as in Appendix G.
| Annotators | Paper's judge | |
|---|---|---|
| (1), regime I | 79.9% | 94.8% |
| (2), regime I | 57.4% | 60.7% |
| (1), regime R | 24.0% | 49.8% |
| (2), regime R | 5.0% | 6.5% |
The annotators' labels alone reproduce the finding: both proportions are high in regime I and several times lower in regime R (79.9% against 24.0%, and 57.4% against 5.0%).
We will add these results to the final version.
Response to Reviewer (Part 1/5)
Thank you for your valuable feedback. We answer the four weaknesses and the question on annotators in turn: W1 here, W2 in Part 2, W3 in Part 3, W4 and the question in Part 4. Part 5 is a supplementary table.
W1. "L028 suggests that newer generations do not close the gap, but the Sep 2026 experiments omit several models that were available by experiment date, including GPT-5.6, Kimi K3, GLM-5.3, and stronger Qwen3.8 variants. [...] Comparisons among older, size-matched models do not adequately support conclusions about the strongest contemporary systems."
A1. Unlicensed repair remains an open problem for most frontier models. Four of the five newly evaluated frontier models keep a large regime-I gap, even with reasoning enabled: +27.8 to +74.4 percentage points (pp), versus +0.5 to +8.3 pp in regime R. GPT-6 Astra nearly closes it (+0.5 pp).
We evaluated the newest API-accessed models of five providers, with reasoning on, on all 50 families in both languages, with the paper's prompt and its frozen judge, gpt-5.6-luna.
Table 1. Newest models, reasoning on. V and S: accuracy on violating and same-legal items (%). Gap: S−V in pp, shown separately for the two regimes. In regime R the problem licenses a repair of the impossible value, such as rounding a fractional count up. In regime I no repair is licensed, and the correct response is to report that the premises admit no consistent answer. A gap near zero means the model is as accurate when the computed value is impossible as when it is legal, and a large gap means it is not. Full sequence: share of regime-I violating responses that state the conflict, answer incorrectly, repair without license and commit to an alternative (%).
In the four models with a large gap, 28% to 69% of the regime-I responses go through the full sequence. Strong performance on legal items does not ensure appropriate handling of inconsistent premises: Claude Opus 5.5 achieves 99.2% same-legal accuracy while retaining a +27.8 pp regime-I gap.
GPT-6 Astra nearly closes the gap under the original prompt, which shows that correct handling is achievable. The other four models, all among the strongest contemporary systems, have not achieved it.
We will add these results to the final version and release the raw inputs and outputs of all API-accessed and open-weight models.