- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper measures the extra completion tokens that reasoning models spend when two equally trusted passages in the context disagree, and asks what removes this cost. Across six open model configurations, an unresolved conflict costs 2.0–57× the tokens of matched agreeing passages on 32 constructed notes and 23 Wikipedia contradictions (≥176× in the cap-censored K2-Horizon-7B read), and almost all of the excess is reasoning. The central claim is that the cost comes from not being told what to report. One sentence that names the passage to answer from removes most of the cost, even when it explicitly disclaims authority, while an assertion that one passage is operative does not substitute for that decision. A factorial experiment on 47 further Wikipedia contradictions separates the grammatical form of the decision (0.93–1.05× the command) from replacing the decision with an operative-source claim (1.55–3.71×), in three models and two wordings. Because a fixed answer overrides evidence that settles the question, the paper also proposes an evidence-conditional stopping rule (answer CONFLICT only when the other documents leave the answer open) and tests its transfer on 150 new unsettleable items.
Critical strengths
-
Clean causal isolation. The factorial in Section 6 separates the grammatical form of the decision (declarative vs. command, 0.93–1.05×) from replacing the decision with an operative-source claim (1.55–3.71×). The result holds in all three models and both wordings. This is the evidence that makes the main claim ("a decision, not authority") credible rather than a wording artifact, and it carries most of my recommendation.
-
Careful measurement and statistics. Every estimate is a ratio with a 95% interval, every reported ratio is computed by two independent implementations, and the cap-censored K2-Horizon-7B result (≥176×) is reported as a lower bound. The stopping rule is tested for transfer on 150 unsettleable items that no model had seen: it removes 46–88% of the cost and keeps or improves settleable accuracy on two of three models. The measurements are more trustworthy than is usual for prompt studies of this kind.
Critical weaknesses
-
The tasks are highly controlled, which limits the general claim. The two passages differ in one value, every call is a single-turn prompt, and the known-answer worlds are small synthetic document sets. There is no realistic multi-step RAG or agent setting, and the natural-text evidence is small (23 + 47 Wikipedia articles). The closing claim that response policy is "a first-class lever on the inference cost of retrieval" therefore reaches further than the evidence. I see this as a scope limit, not a defect in the evidence, but the claim should be scoped to match.
-
The stopping rule only helps models that already tell settled from unsettled conflicts. In Study 2, Olmo-3-7B-Think answers CONFLICT on only 24–32 of 150 unsettleable items under the rule and saves nothing (1.13–1.15× no instruction). In the transfer test, gpt-oss-20b loses 6.7 points of settleable accuracy [4.3, 9.0], mostly through CONFLICT answers. The paper states this scope, but it gives no way to make the rule work for weaker models.
Questions for authors
-
(Clarification) Do you have any result beyond two passages with one differing value, for example three or more sources, several conflicting values, or a multi-turn exchange? Such a result would change how far the general claim reaches. If there is none, I would read this as future work and ask that the claim be scoped.
-
(Clarification) For a model such as Olmo-3-7B-Think, which does not separate settled from unsettled conflicts, did you try a variant of the stopping rule, for example an explicit check step before the answer, that recovers part of the saving?
3 comments
Response to the authors
Thank you for the response and the additional experiments.
Your answers address my concerns.
- W1: the revision removes "first-class lever", changes the title to "Removes Most of the Cost", and limits the conclusion to one disputed quantity. The claim now matches the evidence.
- Q1: you added five passages with distractors, a third source, a third value and a second turn on the 47 extension articles. You report that the directive brings the cost back within 1.42 times the agreeing cost (Appendix Table 24). This answers my question on settings beyond two passages with one differing value.
- W2 / Q2: you ran the check-step variant. You report that on Olmo-3-7B-Think the check is right on 49% of settleable and 58% of unsettleable items, and that the variant does not recover the saving (Appendix Table 33). Section 3.5 now states which models the stopping rule needs. This answers my question.
I keep my rating at 3 (weak acceptance).
Thank you for your response, we're glad that we addressed all your concerns. Happy to clarify anything else if needed.
Thank you for your review. You asked us to scope the claim, to show results beyond two passages with one differing value, and whether an explicit check step helps weaker models.
W1 (The "first-class lever" claim reaches further than the evidence): We scoped the claim to match. We removed "first-class lever" and changed the title to say "Most of the Cost". The conclusion now limits the claim to one disputed quantity, mostly with two passages in one turn. We also keep apart what each corpus shows. Accuracy comes from the synthetic known-answer worlds, where the other documents either settle the conflict or leave it open. The Wikipedia articles measure token cost, since they have no known answer.
Q1 (Results beyond two passages with one differing value): Yes. On the 47 extension articles, we added more sources, several conflicting values and a multi-turn exchange: five passages with three distractors, a third source repeating one value, a third source adding a new value, and a second turn. Conflicts raise cost in every format, and the directive brings cost back within 1.42 times the agreeing cost (Appendix Table 24). Even when two sources outvote the assigned passage, the directive still reports that passage on 99 to 100% of calls (Appendix Table 25). On the closed models with three values or two turns, the directive leaves 1.08 to 2.00 times agreeing (Appendix Table 23).
W2 / Q2 (Weaker models, and an explicit check step before the answer): We ran your check-step variant, which first writes CHECK: yes or no. On Olmo-3-7B-Think the check is right only about half the time, on 49% of settleable and 58% of unsettleable items, and the variant does not recover the saving (Appendix Table 33). The stopping rule depends on exactly this judgment, so check accuracy shows which models it needs. On two more models, Nemotron-3-Nano-30B and Olmo-3-32B-Think, the check is right on only 3% and 8.3% of unsettleable items, and the rule adds settleable failures on both (Appendix Table 34). On gpt-oss-20b, almost all of the 6.8 points of settleable failures that the rule adds are abstentions. Of the 53 failing responses, 51 answer CONFLICT, and wrong answers fall from four to two (Appendix Table 16). The paper states this condition directly in Section 3.5.
Thank you again. Your suggestion of an explicit check step showed us which models the stopping rule needs, and your note on scope made our claims more precise. We would be glad to answer any further questions.