- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors test whether models follow rules that differ from familiar math and science rules. Some questions can be solved using the supplied information. Others have missing or conflicting information, so the model should say it cannot give a definite answer. More accurate models do better on the solvable questions, but they still sometimes give textbook answers when they should decline to answer.
Critical strengths
-
The paper tests whether models can follow new rules and recognise when information is missing. A model could do one well and still struggle with the other.
-
The authors report how often models decline to answer and whether their wrong answers match textbook answers. This gives a clearer picture of the mistakes than overall accuracy alone.
Critical weaknesses
-
An answer close to the textbook answer could also come from a calculation mistake. I need more evidence before accepting the authors' explanation. The example that names a method without showing a calculation also does not establish that checking the steps would miss the error.
-
The main analysis of questions without a definite answer uses different sets of completed questions for different models. The authors include some checks, but I still want this analysis repeated on the same questions for every model.
-
I found the definitions and results difficult to follow. A complete example would help explain the scoring. The authors should also make clear what each percentage counts.
-
I want to know whether the wording of the questions affects the results. Models might behave differently if they are explicitly told that the supplied rules replace ordinary textbook rules.
Questions for authors
- How do you distinguish a model using textbook rules from one making a calculation error? Do the conclusions change if you require closer numerical matches?
- Does the main result hold when every model is evaluated on the same questions?
- Could you explain the scoring through a worked example and clarify what each percentage measures?
- Do the results change when the prompt explicitly tells models to follow the supplied rules over textbook knowledge?
3 comments
Thank you for the additional experiments. They substantially address my main concerns, and I am raising my recommendation to weak acceptance. The tasks are still narrow, but they show that solving problems correctly and knowing when not to answer are different abilities and to approach them soundly.
Thank you for the follow-up and for raising your recommendation. We are glad the additional experiments addressed your main concerns. We agree that the task scope could be made wider. We will further expand the worlds and tasks in the camera-ready version.
Response to Reviewer KkXg
Response to Reviewer KkXg
Thank you for the clear and constructive review. We address each point below. All changes are in the revised PDF.
W1 & Q1: Textbook rules versus calculation errors
We agree that matching the prior is not enough on its own, so we added three pieces of evidence.
First, in a new analysis, we manually checked reasoning traces behind all 142 prior-matching answers of Qwen3-30B-A3B. In 128 of them the trace uses the prior directly. In the other 14 it does not say which value was used (new manual check, end of Section 4.3.2). We have included the full list of the 142 checked questions and their responses to our rebuttal version of the supplementary material, under the name
Qwen3-30B-A3B-Instruct_manual_check.xlsx.Second, in a new experiment we conducted, we measured prior preference from token probabilities (details can be viewed at Appendix D of rebuttal pdf). At the point where the model starts its final answer, we read off the probability it gives to the prior and to two decoy numbers next to it. A model that reaches the prior through an arithmetic slip has no reason to favour it over its neighbours. Yet on the questions where Qwen3-4B gave the prior, it already favours the prior over both decoys before any reasoning on 82% of them, against 42% on its other questions. Qwen2.5-Math-7B shows the same pattern (63% against 47%). The two reasoning models build the preference up while reasoning: Qwen3-4B-Think favours the prior over ABSTAIN on none of the unanswerable questions before reasoning, and on 34% once 60% of its trace is written.
Third, in a new experiment, we used tighter numerical tolerances (Appendix C). Halving the tolerance keeps 91.5% of prior-leaking answers across all 26 models (1,568 of 1,714), and 82.4% equal the prior exactly. Therefore out conclusions are unchanged.
On the example that names a method without a calculation: we agree it does not show that checking the steps would miss the error. We no longer rest the claim on it, and we have softened the section title to “Stated reasoning traces do not fully reveal leakage” (Section 4.3.1).
W2 & Q2: The same questions for every model
We repeated the analysis on the 330 questions completed by all 26 models (new analysis, Appendix B), and the main result holds. Between 10% and 90% accuracy, answerable leakage falls from 14.5% to 1.5% (r = −0.71). Unanswerable leakage does not fall: 14.9% to 19.5% (r = +0.20), and the slope’s 95% CI includes zero. The full-set values are 12.6% to 1.9% and 12.1% to 15.5%. Therefore, the conclusion drawn from this common question set is almost identical to the one we have made. All runs draw from the same shuffled list of 700 questions, so shorter runs are also a fair random draw across the seven worlds (Section 3).
W3 & Q3: A worked example and what each percentage counts
The box in Section 4.3.1 now walks through the full scoring of one question. It shows the question, how aext and aint are computed, the tolerance τ(a)=max(0.02,0.01∣a∣) with the two windows it gives, and how real replies to this question are classified into each failure type. The tolerance is also stated in the Evaluation Protocol (Section 3). The per-model table (Appendix E) defines each column’s denominator: accuracy is over completed questions, leakage is over committed answers on prior-distinguishing questions, and failure to abstain is over unanswerable questions.
W4 & Q4: Telling models that the supplied rules replace textbook rules
In a new experiment we conducted, we tested this directly (Section 4.2, Figure 8 in Appendix J). We reran the 210 unanswerable questions on four models, replacing the closing sentence with one of two stronger instructions. The first tells the model to use only the information given, even where it contradicts the real world. The second also forbids any value or default assumption from prior knowledge, and names filling in a missing quantity explicitly. The first instruction has no effect. The second lowers failure to abstain from 52.6% to 46.9%, yet 31.8% of the answers models still commit are the prior, against 36.9% under the original prompt. Prior gap-filling is not mainly an artifact of weak wording, while weak wording can have some slight confounding effects.