acceptodds
Under review as a conference paper at ICLR 2027

Unlicensed Repair: When Large Language Models Turn Impossible Quantities into Plausible Answers

Abstract

When solving word problems, large language models can explicitly identify an intermediate value they compute as impossible, such as a negative count of pallets, yet still produce plausible-looking answers. We study unlicensed repair: responding to a conflict between a computed value and an implicit commonsense constraint on its quantity with a modification the problem does not license. We construct an English–Chinese corpus in which every violated constraint is implicit, spanning multiple quantity types and distinguishing problems that license a repair from those admitting no consistent answer. Each violating problem has two controls: one changes an input so the value becomes legal, and the other keeps the value and changes what it represents. We evaluate models on model–prompt pairs with an LLM judge, checked against three independent annotators and a rule-based scorer. Pooled accuracy is lower on violating problems than on either control. Across all models, the gap to the first control is larger when no repair is licensed: against percentage points pooled. Most incorrect responses explicitly acknowledge the conflict. When no repair is licensed, almost all of those that acknowledge it go on to perform exactly this repair: they alter premises or introduce unsupported assumptions, then commit to a plausible alternative. No lever we tested closes the gap: not more parameters, a newer model generation, a prompt to check, or reasoning. A plausible final answer can therefore conceal an unresolved contradiction that the model has explicitly identified.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reasoning Reliability, Metacognition, Unlicensed Repair

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_aGBMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how LLMs handle mathematical word problems containing impossible quantities such as negative physical quantities. The authors argue that models may recognize a contradiction but still introduce assumptions or adjustments that make the problem make sense.

They construct a benchmark with three conditions: violating (V), same-legal (S), and transfer (T), distinguishing problems where repair is permitted from those where no valid repair exists.

Edit: Following the author response, i have updated my score because my concerns where largely addressed

Critical strengths
  1. The paper distinguishes recognizing a contradiction from responding appropriately to it. This distinction is potentially important for evaluating LLM reliability beyond ordinary answer accuracy.

  2. The paper includes extensive evaluation

Critical weaknesses
  1. L028 suggests that newer generations do not close the gap, but the Sep 2026 experiments omit several models that were available by experiment date, including GPT-5.6, Kimi K3, GLM-5.3, and stronger Qwen3.8 variants. Notably, GPT-5.6 is used as a judge but not evaluated as a solver. Comparisons among older, size-matched models do not adequately support conclusions about the strongest contemporary systems

  2. L085–L087 controls do not isolate the proposed failure mechanism. Changing an input to make a problem valid can allow shortcuts that produce correct answers without checking intermediate computations (see figure 20 which has skipped intermediate computation). The transfer control also changes the question and potentially the reasoning strategy. Consequently, differences in accuracy may reflect reasoning or computation differences rather than sensitivity to impossible quantities alone. Figure 19 also shows an additional failure mode not controlled for and which is mislabeled as unlicensed when the model completely misunderstood the question.

  3. The main behavioral classifications rely on a single LLM judge despite only moderate agreement with human consensus for resolution strategies (0.492).

  4. The paper does not adequately motivate the practica signifance of the proposed failure mode

Questions for authors

see weaknesses.

clarify the compensation offered to human annotators and how they where recruited

8 comments
AuthorsOpenReview ↗

Response to Reviewer (Part 1/5)

Thank you for your valuable feedback. We answer the four weaknesses and the question on annotators in turn: W1 here, W2 in Part 2, W3 in Part 3, W4 and the question in Part 4. Part 5 is a supplementary table.

W1. "L028 suggests that newer generations do not close the gap, but the Sep 2026 experiments omit several models that were available by experiment date, including GPT-5.6, Kimi K3, GLM-5.3, and stronger Qwen3.8 variants. [...] Comparisons among older, size-matched models do not adequately support conclusions about the strongest contemporary systems."

A1. Unlicensed repair remains an open problem for most frontier models. Four of the five newly evaluated frontier models keep a large regime-I gap, even with reasoning enabled: +27.8 to +74.4 percentage points (pp), versus +0.5 to +8.3 pp in regime R. GPT-6 Astra nearly closes it (+0.5 pp).

We evaluated the newest API-accessed models of five providers, with reasoning on, on all 50 families in both languages, with the paper's prompt and its frozen judge, gpt-5.6-luna.

Table 1. Newest models, reasoning on. V and S: accuracy on violating and same-legal items (%). Gap: S−V in pp, shown separately for the two regimes. In regime R the problem licenses a repair of the impossible value, such as rounding a fractional count up. In regime I no repair is licensed, and the correct response is to report that the premises admit no consistent answer. A gap near zero means the model is as accurate when the computed value is impossible as when it is legal, and a large gap means it is not. Full sequence: share of regime-I violating responses that state the conflict, answer incorrectly, repair without license and commit to an alternative (%).

ModelVSGap, regime RGap, regime IFull sequence
GLM-5.354.598.2+8.3+74.469.1
Qwen3.8-Max62.598.2+1.2+64.857.5
Kimi-K3.1-Pro78.099.0+5.5+34.528.3
Claude Opus 5.584.099.2+0.5+27.828.2
GPT-6 Astra97.298.2+1.6+0.53.2

In the four models with a large gap, 28% to 69% of the regime-I responses go through the full sequence. Strong performance on legal items does not ensure appropriate handling of inconsistent premises: Claude Opus 5.5 achieves 99.2% same-legal accuracy while retaining a +27.8 pp regime-I gap.

GPT-6 Astra nearly closes the gap under the original prompt, which shows that correct handling is achievable. The other four models, all among the strongest contemporary systems, have not achieved it.

We will add these results to the final version and release the raw inputs and outputs of all API-accessed and open-weight models.

AuthorsOpenReview ↗

Response to Reviewer (Part 2/5)

W2. "L085–L087 controls do not isolate the proposed failure mechanism. Changing an input to make a problem valid can allow shortcuts that produce correct answers without checking intermediate computations (see figure 20 [...]). The transfer control also changes the question and potentially the reasoning strategy. Consequently, differences in accuracy may reflect reasoning or computation differences rather than sensitivity to impossible quantities alone. Figure 19 [...] is mislabeled as unlicensed when the model completely misunderstood the question."

A2. The gap persists among responses that explicitly report the critical intermediate value: regime-I accuracy is 95.2% on same-legal items, 65.1% on transfer items and 22.8% on violating items. Of the errors on violating items, 79.3% occur in responses that explicitly acknowledge the conflict.

The two controls. The same-legal item holds the scenario, the quantity and the constraint fixed and changes one numerical input. The transfer item holds the critical intermediate value fixed and changes what it represents (Table 1). We therefore checked whether the accuracy differences persist when responses explicitly report that value.

Explicit intermediate computation. The critical step is the step whose value is impossible in the violating item, and we checked whether its value appears in the response. Among correct same-legal responses, 89.2% explicitly report it. Accuracy among the responses that report the value:

Same-legalTransferViolating
Regime I95.2%65.1%22.8%
Regime R93.1%90.6%78.8%

Among these responses, the regime-I gap is 72.4 percentage points (pp) to the same-legal control and 42.3 pp to the transfer control, against 63.2 pp and 30.3 pp over all responses. The gap therefore does not come from shortcuts that skip the intermediate computation.

Where the errors occur. Error rates, as a share of all scored responses of the condition:

Violating items, all errorsafter acknowledging the conflictwithout acknowledging itSame-legal items, all errors
Regime I76.9%64.1%12.8%13.7%
Regime R21.1%13.2%7.8%9.3%

The excess errors are concentrated in responses that acknowledge the conflict. These account for 64.1% of all regime-I violating responses. Errors without such acknowledgment account for 12.8%, compared with 13.7% total errors on same-legal items.

Figures 19 and 20. Neither response acknowledges the conflict. Figure 19 is labeled PREMISE_REWRITE for its altered reading of the problem, and its monitoring field reads NONE, which separates it from the sequence in which a model acknowledges the conflict and then repairs. Neither example enters the conditional repair estimate of 94.8% or the numerator of the 60.7% full-sequence estimate. Both remain in the regime-I denominator and in the accuracy statistics. Figure 20 shows the shortcut concern addressed above: the gap persists among responses that explicitly report the critical intermediate value.

Together, these analyses strengthen the paper's finding: the gap persists when the critical intermediate value is explicitly reported, and most violating-item errors occur after the response acknowledges the conflict. We will add them to the final version and replace Figure 19 with an example that explicitly states the conflict.

AuthorsOpenReview ↗

Response to Reviewer (Part 3/5)

W3. "The main behavioral classifications rely on a single LLM judge despite only moderate agreement with human consensus for resolution strategies (0.492)."

A3. The behavioral findings persist under two further judges from other providers and under independent human annotation. On the same 4,703 responses (covering all 17 models and all 50 families), the three judges estimate the two regime-I proportions of Section 7 at 86.9% to 94.8% (unlicensed repair among incorrect responses that acknowledged the conflict) and 58.7% to 62.7% (full sequence). The annotators' labels give 79.9% and 57.4%.

0.492. The κ of 0.492 concerns exact agreement across 13 resolution classes. The findings use a coarser label built from these classes, the unlicensed-repair group of Table 11, on which the paper's judge gives the annotators' label for 88.1% of the annotated responses. We assess this label directly below with two further judges and with the annotators' labels.

1. Does the finding depend on one judge? The findings do not depend on one judge. We re-scored 4,703 responses from the paper's 17 models with two further judges from other providers, Qwen3.8-Max and Claude Opus 5.5, using the paper's rubric.

Table 2. Agreement between the three judges on these responses. Same label: share of responses on which the two judges give the same label. Ranking: Spearman ρ between the two judges' per-model results over the 17 models.

Paper's judge and Qwen3.8-MaxPaper's judge and Claude Opus 5.5Qwen3.8-Max and Claude Opus 5.5
Same label: answer correctness91.2%93.1%93.5%
Same label: conflict acknowledgment92.9%93.2%96.1%
Same label: unlicensed repair, yes or no90.3%90.7%91.3%
Ranking: accuracy on violating items0.960.970.97
Ranking: gap S−V in regime I0.880.820.86
Ranking: full-sequence rate0.980.970.94

Table 3. The findings under each judge, on the same responses. (1): unlicensed repair among incorrect responses that acknowledged the conflict. (2): full sequence among violating responses. Gap: S−V in percentage points (pp).

Paper's judgeQwen3.8-MaxClaude Opus 5.5
(1), regime I94.8%94.1%86.9%
(2), regime I62.7%59.8%58.7%
Gap, regime I+71.1 pp+61.3 pp+71.6 pp
Gap, regime R+12.9 pp+13.1 pp+12.2 pp
Models with gap I > gap R17 of 1716 of 1717 of 17

Any two judges give the same label on 90.3% to 96.1% of the responses. Under all three, the regime-I proportions are at least 86.9% for (1) and 58.7% for (2), and the gap is at least +61.3 pp in regime I against at most +13.1 pp in regime R. Per-model results are in Part 5.

2. Does independent human judgment support the finding? The annotators' labels alone reproduce the finding. Three independent annotators labelled 400 responses. The same two proportions, computed from their labels alone:

Table 4. Proportions (1) and (2), defined as in Table 3, from the annotators' labels, beside the paper's values. Annotated responses are weighted as in Appendix G.

AnnotatorsPaper's judge
(1), regime I79.9%94.8%
(2), regime I57.4%60.7%
(1), regime R24.0%49.8%
(2), regime R5.0%6.5%

The annotators' labels alone reproduce the finding: both proportions are high in regime I and several times lower in regime R (79.9% against 24.0%, and 57.4% against 5.0%).

We will add these results to the final version.

Reviewer_AnJkOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates Unlicensed Repair, a phenomenon in which LLMs recognize violations of commonsense constraints in mathematical word problems but still produce answers by making unjustified modifications or additional assumptions. The authors compare constraint-violating problems (V), controls with modified numerical values (S), and controls with altered quantity meanings (T), while distinguishing between cases where a repair is justified (R) and cases with inconsistent premises (I). Across 17 models, the pooled accuracy is 49.4% for V, 88.3% for S, and 69.8% for T, with a much larger S−V gap in I (63.2 percentage points) than in R (11.8 percentage points). Furthermore, 79.3% of incorrect V responses are classified as acknowledging a contradiction or relevant constraint, highlighting the difference between recognizing a contradiction and responding appropriately.

Critical strengths
  1. I appreciate that the paper evaluates not only whether a model recognizes a contradiction, but also whether its subsequent repair is justified by the problem conditions. In particular, R requires a justified repair, while I requires reporting the inconsistency, making it possible to distinguish appropriate behavior from simply abstaining.
  2. Table 3 shows that the S−V gap is larger for I than for R across all 17 models, and the large gap persists in the sensitivity analyses in Table 19. This makes the main findings more convincing, as they do not appear to depend on a particular model or grading method.
Critical weaknesses
  1. As shown in Table 12, R and I largely involve different constraint types, and V and its controls in I also require different answer formats. Therefore, it is difficult to attribute the observed gap specifically to whether a repair is licensed, and the conclusions should be stated more cautiously.
  2. In Appendix G, the unlicensed repair rate among incorrect responses acknowledging a contradiction differs substantially between human annotations (68.3%) and LLM grading (87.8%). Although the denominators are not identical, the discrepancy remains even for cases identified by both methods, so a separate human-based estimate for I would help strengthen the findings.
Questions for authors
  1. Is it possible to compare R and I within the same constraint types or similar scenarios using the existing data? If not, I think it would be more appropriate to limit the conclusions in Section 5 and Figure 3 to the observed differences between the two groups in this corpus.
  2. Could the authors use the existing human annotation data to estimate the two key proportions reported for I-V in Section 7? Specifically, these are (1) among incorrect I-V responses that recognized the inconsistency, the proportion that performed an unlicensed repair (reported as 94.8%); and (2) among all I-V responses, the proportion that recognized the inconsistency, produced an incorrect answer, performed an unlicensed repair, and committed to an alternative answer (reported as 60.7%). It would be helpful to report the denominator and uncertainty estimates for each proportion. If the available human-annotated sample is too small to support reliable estimates, please discuss how this limitation affects the conclusions.
7 comments
AuthorsOpenReview ↗

Response to Reviewer (Part 1/3)

Thank you for the careful reading and for two precise questions. We answer W1 with Q1 here and W2 with Q2 in Part 2. Part 3 is a supplementary table.

W1 and Q1. R and I "largely involve different constraint types, and V and its controls in I also require different answer formats. Therefore, it is difficult to attribute the observed gap specifically to whether a repair is licensed". "Is it possible to compare R and I within the same constraint types or similar scenarios using the existing data?"

A1. The larger regime-I gap persists within Temporal, across all eight scenario domains, and under both scoring rules. The prompt experiment further shows that models can produce the required answer when explicitly instructed.

1. Same constraint type (Section 5, lines 314–317). Temporal is the one constraint type with families in both regimes. Inside it the legality gap S−V is +67.5 percentage points (pp) where no repair is licensed and +2.9 pp where one is, and it is larger in 16 of 17 models. With the constraint type held the same, the contrast between the regimes remains.

2. Similar scenarios. The corpus was written in eight scenario domains that cut across quantity types, and every domain contains families of both regimes. We compared R and I within each domain, using the original responses and scoring.

Table 1. Legality gap S−V by scenario domain and regime, pooled over 17 models.

DomainS−V, regime RS−V, regime IModels with I > R
Warehousing+12.8+65.917/17
Manufacturing+13.4+50.715/17
Retail+12.5+69.717/17
Education+12.5+63.617/17
Healthcare+14.9+65.617/17
Food service+10.3+59.717/17
Transport+11.8+65.617/17
Office+6.4+69.617/17

In every domain the gap is several times larger where no repair is licensed. The contrast is not tied to particular constraint types either: 26 of the 27 regime-I families have a larger gap than every regime-R family. The types of the two regimes are also of similar difficulty when the value is legal: same-legal accuracy is 90.7% in R and 86.3% in I, 4.4 pp apart, against 55.8 pp on the violating items.

3. Answer format. In regime I the correct response is to report that the premises admit no consistent answer. The rubric scores the final commitment, not whether a number appears: it also accepts a clearly conditional calculation that withholds an unconditional real-world answer. The gap does not depend on where this line is drawn: it is +63.2 pp under the judge and +62.2 pp under the rule-based scorer, which uses no model and requires an inconsistency statement and no committed numerical answer (Tables 17 and 19).

An additional prompt experiment. To check that the required answer is one the models can give, we presented the regime-I violating items to all 17 models of the paper (27 configurations, counting reasoning off and on) under three prompts, scored as in the paper: (a) the paper's prompt; (b) the contradiction stated in the problem, for example "Together the three orders took more pallets than the depot held when it opened"; (c) one added sentence, "If the conditions of the problem cannot all hold at once, say so instead of giving a number". We also ran same-legal items, to check that the added sentence does not simply make the models decline to answer.

Accuracy on regime-I violating items under the paper's judge rubric (per configuration: Table 3 in Part 3):

Models(a) Paper's prompt(b) Contradiction stated in the problem(c) Added sentenceSame-legal accuracy, (a) → (c)
API-accessed models (10 configurations)30.2%21.6%98.8%96.9% → 89.9%
Open-weight models (17 configurations)8.7%10.3%71.5%83.1% → 77.5%
All 2716.7%14.4%81.7%88.2% → 82.0%

The models can give the answer that regime I requires: with the added sentence, accuracy on these items rises from 30.2% to 98.8% for the API-accessed models and from 8.7% to 71.5% for the open-weight models. They do not simply decline to answer: under the added sentence they still solve 89.9% and 77.5% of the same-legal items. Stating the contradiction in the problem does not have this effect (21.6% and 10.3%).

Together, these analyses strengthen the paper's finding that the legality gap is larger where no repair is licensed: the regime-I gap persists within the same constraint type and within every scenario domain, and it exceeds 62 pp under both scorers. We will add them to Section 5 and Figure 3 in the final version.

AuthorsOpenReview ↗

Response to Reviewer (Part 2/3)

W2 and Q2. The unlicensed-repair rate "differs substantially between human annotations (68.3%) and LLM grading (87.8%) [...] the discrepancy remains even for cases identified by both methods". "Could the authors use the existing human annotation data to estimate the two key proportions reported for I-V in Section 7? [...] report the denominator and uncertainty estimates for each proportion."

A2. Human annotations confirm both findings of Section 7: in regime I, unlicensed repair is the dominant response among the incorrect responses that acknowledged the inconsistency (79.9%, 95% CI 71.9–87.2), and 57.4% of all I-V responses go through the full sequence (50.0–63.7). We computed both proportions from the existing annotation data (Table 2). The 68.3% in Appendix G pools both regimes, and regime I alone gives 79.9%.

Method. Of the 400 responses labelled by three annotators, 200 are regime-I violating responses, weighted to the 21,129 I-V responses of the annotation sampling frame (Appendix G). The annotators' label is the majority of the three. Intervals are 95% bootstrap intervals, with responses resampled within strata and the weights kept. As a check, the judge's labels on the same weighted sample give 96.2% and 61.6%, close to its full-pool values.

Table 2. Human estimates of the two Section 7 proportions, with regime R as a reference. (1) Unlicensed repair among incorrect responses that explicitly acknowledged the inconsistency. (2) The full sequence (explicit acknowledgment, incorrect answer, unlicensed repair, commitment to an alternative real-world answer) among all violating responses.

Regime / proportionJudge (%)Annotators (%) [95% CI]Annotated nWeighted denominator
I: (1)94.879.9 [71.9–87.2]9715,169
I: (2)60.757.4 [50.0–63.7]20021,129
I: (1), shared subset95.985.1 [75.9–93.0]5712,720
R: (1)49.824.0 [11.1–41.9]773,951
R: (2)6.55.0 [2.1–9.3]20018,772

Annotated n counts the responses in each human-estimate denominator. The Weighted denominator is the sum of their sampling weights. The shared subset contains the responses that both methods classify as acknowledging the inconsistency and answering incorrectly. Judge values are the paper's full-pool estimates, except in the shared-subset row, where both estimates use the same 57 responses and weights.

Under the annotators' labels both proportions are far higher in regime I than in regime R, and for (1) the contrast between the regimes is larger than under the judge's labels (55.9 pp against 45.0 pp). The full-sequence estimate is close to the judge's (57.4% against 60.7%). The conditional repair rate is lower than the judge's and is still four in five. The discrepancy the reviewer points to therefore changes the level of the first proportion and leaves both findings in place.

The cases identified by both methods. On the responses that both methods identify, unlicensed repair is the large majority under either label: 85.1% for the annotators and 95.9% for the judge. The discrepancy concerns repair classification within responses both methods consider incorrect.

Sample size. The estimates rest on 97 and 200 annotated responses. This sample is sufficient for both conclusions: the intervals are well separated between the regimes for both proportions (71.9–87.2 against 11.1–41.9, and 50.0–63.7 against 2.1–9.3).

We will report these human estimates beside the judge's values in the final version.

We would appreciate your reconsideration in light of these additional analyses and welcome any remaining questions.

AuthorsOpenReview ↗

Response to Reviewer (Part 3/3, supplementary table for A1)

Table 3. The prompt experiment of Part 1, per configuration: accuracy on regime-I violating items under the paper's judge rubric. The last column gives same-legal accuracy under the paper's prompt → with the added sentence.

Model (reasoning off unless noted)(a) Paper's prompt(b) Contradiction stated in the problem(c) Added sentenceSame-legal accuracy, (a) → (c)
GPT-5.541.2%30.6%100.0%99.5% → 91.0%
GPT-5.5, reasoning on71.8%65.1%98.6%98.5% → 96.5%
DeepSeek-V4-Flash7.4%6.9%100.0%93.0% → 89.5%
DeepSeek-V4-Flash, reasoning on59.0%48.3%97.2%98.5% → 98.0%
Qwen3.8-Flash11.1%1.0%96.3%96.9% → 92.1%
Qwen3.8-Flash, reasoning on42.9%32.7%100.0%99.0% → 96.3%
GLM-5.211.1%6.8%99.1%91.5% → 53.0%
GLM-5.2, reasoning on11.7%6.5%100.0%97.5% → 95.5%
Kimi-K2.623.0%8.9%97.2%95.9% → 90.5%
Kimi-K2.6, reasoning on20.5%8.7%100.0%99.0% → 97.9%
API-accessed models, pooled (10 configurations)30.2%21.6%98.8%96.9% → 89.9%
Gemma-2-9B27.3%46.8%80.0%90.5% → 76.5%
Llama-3-8B4.4%10.8%67.1%75.8% → 72.9%
Mistral-7B16.7%31.9%71.7%53.0% → 34.5%
DeepSeek-LLM-7B12.7%23.8%53.7%51.5% → 38.5%
GLM4-9B26.9%28.3%62.8%62.3% → 58.5%
OLMo-3-7B6.9%6.2%88.0%88.3% → 83.8%
Qwen2.5-7B2.3%3.2%69.9%88.5% → 86.0%
Qwen3-0.6B5.6%3.3%41.1%66.0% → 47.5%
Qwen3-0.6B, reasoning on3.2%3.1%19.7%81.4% → 75.6%
Qwen3-4B2.8%2.3%80.1%92.5% → 90.5%
Qwen3-4B, reasoning on0.6%1.1%89.4%94.9% → 93.4%
Qwen3-8B6.1%3.7%69.0%95.5% → 95.0%
Qwen3-8B, reasoning on1.7%2.6%75.5%95.9% → 94.0%
Qwen3-14B10.2%2.3%85.2%94.5% → 85.0%
Qwen3-14B, reasoning on5.6%1.4%91.7%93.0% → 96.0%
Qwen3-32B9.4%0.5%86.1%94.5% → 94.0%
Qwen3-32B, reasoning on1.4%0.9%81.9%95.0% → 96.0%
Open-weight models, pooled (17 configurations)8.7%10.3%71.5%83.1% → 77.5%
All 27 configurations16.7%14.4%81.7%88.2% → 82.0%

The result of Part 1 holds for each model, not only on average. With the added sentence (c), accuracy on regime-I violating items rises in all 27 configurations. Stating the contradiction (b) does not have this effect: in every configuration accuracy stays below the level under (c). Under (c) the models do not simply decline to answer: same-legal accuracy moves by less than 10 pp in 22 of the 27 configurations.

Reviewer_mcqaOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

It shows that reasoning models will accept unrealistic intermediate statements, such as using fractions in place of whole counting numbers, when pushed to do so by unrealistic premises. This explains some cases where a model uses sloppy thinking internally.

Critical strengths

I really love how this uses implicit knowledge as constraints. More papers need to be doing this. The examples of integer-values and positive-values are good examples of this.

The corpus is valuable.

The core phenomenon under investigation is very important to investigate.

Critical weaknesses

The abstract uses the word "licensed" but does not define it clearly, which hurts clarity.

I believe this issue of unlicenced repair is widespread in real-life reasoning traces. The paper would benefit from identifying more cases of it.

The appendix should include a taxonomy of implicit knowledge types that are used in the corpus. "Real numbers cannot be negative" is an example, as is "whole numbers must be integers". How broad is this taxonomy? This makes a significant difference between the corpus being large and the corpus being truly broad.

The V, S, T, I nomenclature is hard to follow and I suggest you use the full words: "Violating", "Same-legal", and "Transfer" throughout the paper. This would make it wordier but clearer so this is just personal preference.

If "the correct response is to report that the premises admit no consistent answer", was this clearly communicated to the LLM? Sometimes in an eval setting, it is not clear to them that they can do this.

Figure 2 is somewhat confusing. What is the "Legality gap"? that term should be better defined.

Table 4 is a wall of numbers. Could it be a chart or graph instead?

Questions for authors

How broad is the corpus? How many distinct violation types do you include? If there is a lot of diversity here, I would be much more likely to promote this paper.

Please think about how you could present your numerical results more clearly. I think a clear explanation of the repair gap would go a long way.

I recommend you include more examples in the main text of the paper, to help the reader understand what is going on in practice.

10 comments
AuthorsOpenReview ↗

Response to Reviewer (Part 1/3)

Thank you for your encouraging and constructive feedback. Part 2 gives the taxonomy with representative examples (W3 and Q1). This part starts with the terms and the presentation (W4, W1, W6, W7 and Q2), since they fix the wording used in the rest of this reply. Part 3 answers the prompt question (W5) and the request for more cases (W2 and Q3).

W4 (abbreviations). "The V, S, T, I nomenclature is hard to follow and I suggest you use the full words: 'Violating', 'Same-legal', and 'Transfer' throughout the paper."

A1. Thank you for this suggestion. This reply already uses the full words, and we will replace the letters with them throughout the paper (text, tables and captions) in a revised version before the end of the discussion period.

Table 1. The abbreviations of the paper and the full words that replace them. The first three names identify the item conditions. The last two identify whether a repair is licensed.

In the paperFull wordsMeaning
Vviolatingan item in which a computed value is impossible for the quantity it stands for
Ssame-legalthe same scenario with one input changed, so that the value is legal
Ttransferthe same computed value, standing for a quantity for which it is legal
Rrepair licensedthe task conditions and the quantity's meaning justify a specific repair
Ino repair licensedthe premises admit no consistent answer, so the response should report the inconsistency

W1 (the word "licensed"). "The abstract uses the word 'licensed' but does not define it clearly, which hurts clarity."

A2. "Licensed" means authorized by the problem itself. A repair is licensed when the task conditions and the quantity's meaning justify a specific adjustment. It is unlicensed when the model introduces a change that those conditions do not support.

For example, unloading all 187 cartons at 30 per indivisible dock appointment requires 7 appointments, and a reading of 31:00 on a 24-hour clock wraps to 07:00 the next day. Both adjustments are licensed. When 412 kg are unloaded from a container that held 309 kg, answering that 0 kg remain is unlicensed, and so is changing a listed weight from 181 to 98 grams so that the ingredients fit the tub. Nothing in the problem supports either change, even when the model announces it. The paper calls the whole behavior unlicensed repair: the model acknowledges the conflict, makes such a change, and commits to the answer.

The risk of unlicensed repair is that such an answer looks like a normal result, so the error goes unnoticed. In settings such as multi-turn conversations and agent workflows, later steps then build on the result of the unlicensed repair, and the error becomes hard to trace back to its source.

We will define the term in the abstract and illustrate it with these examples in the Introduction of the final version.

W6, W7 and Q2 (legality gap, Figure 2, Table 4). "What is the 'Legality gap'? that term should be better defined." "Table 4 is a wall of numbers. Could it be a chart or graph instead?" "I think a clear explanation of the repair gap would go a long way."

A3. The legality gap is same-legal accuracy minus violating accuracy, in percentage points (pp). Pooled over the 17 models it is 88.3% − 49.4% = +38.9 pp: accuracy is 38.9 points lower on violating items than on their same-legal controls.

How to read the gap. A gap near zero means similar accuracy in the two conditions. A large positive gap means substantially lower accuracy on violating items. The gap is +11.8 pp where a repair is licensed and +63.2 pp where none is. Section 7 characterizes the failures separately: where no repair is licensed, 60.7% of the violating responses show the full unlicensed-repair sequence.

Figure 2. In panel (a), the horizontal distance between violating and same-legal accuracy is the legality gap. Panel (b) shows that gap separately for items with and without a licensed repair. In the final version we will state this definition in the caption and open Section 5 with a short reading guide.

Table 4. Thank you for your valuable suggestion. We will replace Table 4 with a chart, in the same revised version as the full words, before the end of the discussion period. Each robustness check is one row on a common axis in percentage points. The gaps with and without a licensed repair are the two highlighted points, so their separation is visible at a glance, and the other two contrasts appear in lighter marks. We will move Table 4 to the appendix.

AuthorsOpenReview ↗

Response to Reviewer (Part 2/3)

W3 and Q1 (how broad is the corpus?). "The appendix should include a taxonomy of implicit knowledge types that are used in the corpus. [...] How broad is this taxonomy?" "How broad is the corpus? How many distinct violation types do you include? If there is a lot of diversity here, I would be much more likely to promote this paper."

A4. The corpus covers seven categories of implicit quantity constraints, instantiated in 50 scenario families across eight application domains. The two examples the reviewer mentions, integer counts and non-negative quantities, are two of the seven.

1. Taxonomy of implicit knowledge.

Table 2. The seven categories, the implicit knowledge each relies on, and one example from the corpus.

CategoryImplicit knowledgeExample from the corpusFamilies
Part–wholeparts drawn from a fixed whole cannot exceed itthree consignments occupy 424 cubic metres in a fixed 357-cubic-metre hold8
Signa physical stock cannot be negative412 kg unloaded from a container holding 309 kg: −103 kg left8
Bounded fractiona share of a fixed whole cannot exceed 100%191 pallets for 150 rack positions: an occupancy share of 127%6
Relationquantities tied by conservation or by a complete partition must agree347 staff, each in exactly one of three centres, and the centres list 3444
Integer countindivisible objects come in whole numbers187 cartons at 30 per dock appointment: 6.23 appointments before rounding up8
Divisibilitya quantity moves only in whole packs13 bottles per shop when bottles leave only in cases of 68
Temporalclock time is cyclic, and the time left in a fixed window cannot be negativea shift ending at 31:00, and 126 days used of a 117-day returns window8

The categories require different knowledge. For example, 13 bottles is a whole number and still violates the pack constraint, so divisibility is a different check from integrality. The model must infer the relevant constraint from the quantity's meaning and the stated conditions.

2. Is the corpus broad, or only large? The 50 scenario families span eight application domains. Within a domain every family belongs to a different category, so each domain covers five to seven of the seven categories (Table 3). Eight numerical variants per family in English and Chinese then yield 2,400 items within these categories.

3. Does the finding hold across this diversity? The finding holds in every one of the eight domains. The legality gap is +50.7 to +69.7 percentage points where no repair is licensed, against +6.4 to +14.9 where one is.

Table 3. The eight scenario domains: categories covered and the legality gap in percentage points, pooled over the 17 models.

DomainCategories coveredGap, repair licensedGap, no repair licensed
Warehousing6 of 7+12.8+65.9
Manufacturing7 of 7+13.4+50.7
Retail5 of 7+12.5+69.7
Education7 of 7+12.5+63.6
Healthcare6 of 7+14.9+65.6
Food service7 of 7+10.3+59.7
Transport6 of 7+11.8+65.6
Office administration6 of 7+6.4+69.6

We will expand Table 12 into this taxonomy, with the underlying knowledge and a representative example for each category, and summarize the coverage in the main text of the final version.

AuthorsOpenReview ↗

Response to Reviewer (Part 3/3)

W5 (was this clearly communicated to the LLM?). "If 'the correct response is to report that the premises admit no consistent answer', was this clearly communicated to the LLM? Sometimes in an eval setting, it is not clear to them that they can do this."

A5. The paper studies the models' default behavior, so its prompts do not tell the model that it may report an inconsistency. We also tested what happens when they do: with one added sentence, accuracy on violating items with no licensed repair rises from 16.7% to 81.7% across the 17 models, and from 30.2% to 98.8% for the API-accessed models.

The experiment. We presented the same items to all 17 models (27 configurations, counting reasoning off and on) under the paper's prompt and with one added sentence: "If the conditions of the problem cannot all hold at once, say so instead of giving a number". Each configuration answered 216 violating items with no licensed repair and 200 same-legal items, which measure what the sentence costs where a number is the correct answer.

Table 4. Accuracy on violating items with no licensed repair under the paper's judge rubric, and same-legal accuracy, without and with the added sentence. Per-configuration results are in Table 5.

ModelsPaper's promptWith the added sentenceSame-legal accuracy, without → with
API-accessed models (10 configurations)30.2%98.8%96.9% → 89.9%
Open-weight models (17 configurations)8.7%71.5%83.1% → 77.5%
All 27 configurations16.7%81.7%88.2% → 82.0%

The added sentence raises accuracy by 63 to 69 percentage points in every row. The low accuracy under the paper's prompt is therefore the models' default choice and not a limit of what they can do: once permitted, most of them report the inconsistency. The sentence also has a cost, since same-legal accuracy falls from 88.2% to 82.0%.

W2 and Q3 (more cases and examples). "I believe this issue of unlicensed repair is widespread in real-life reasoning traces. The paper would benefit from identifying more cases of it." "I recommend you include more examples in the main text of the paper."

A6. The paper's data contain 12,988 responses with the full sequence of unlicensed repair, from all 17 models, all 27 families without a licensed repair and all eight domains. Three of them are shown below, and we will add more examples to the appendix and release all raw model outputs.

Among these 12,988 responses, 57.9% are classified as an arbitrary cap, 31.0% as a rewritten premise and 11.1% as a clamp to zero. One example of each, from API-accessed models under the paper's prompt:

  • An arbitrary cap (GPT-5.5). Three inbound loads take 191 positions in racking that has 150. The model notes that the racking cannot hold more than 150, sets the occupied share to 100%, and reports the full insurance cost.
  • A rewritten premise (DeepSeek-V4-Flash). Three legs cover 355 km of a route stated to be 326 km long. The model notes that this is impossible, assumes that the stated length is a mistake and takes the route to be 355 km, and reports the full subsidy.
  • A clamp to zero (Kimi-K2.6). Three departments draw 281 boxes from a store that started with 270. The model notes that more was drawn than was in stock, treats the store as empty, and reports that nothing is written back.

In each case the model acknowledges the inconsistency and then commits to an answer based on an adjustment that the problem does not support.

In the final version the appendix will carry more examples for every category, and the Introduction will carry the examples of A2. We will also release the raw inputs and outputs of all API-accessed and open-weight models, including the five newest models that we evaluated for Reviewer aGBM (GLM-5.3, Qwen3.8-Max, Kimi-K3.1-Pro, Claude Opus 5.5 and GPT-6 Astra).

Thank you again for your encouraging assessment of our work and for the suggestions that make the paper clearer. We hope you find our responses helpful, and we warmly welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.