acceptodds
Under review as a conference paper at ICLR 2027

Telling Reasoning Models What to Report Removes the Cost of Conflicting Sources

Abstract

Retrieval-augmented and agentic systems routinely place disagreeing sources in one context, imposing large inference costs on reasoning models. Across 23 Wikipedia contradictions and 32 constructed documents, six open model configurations use 2.0–57× more completion tokens for conflicting than matched agreeing passages (≥176× in one generation-capped case), with 89.2–99.8% of conflict tokens spent on reasoning. We show that this cost largely disappears once the model is told what to report: a single passage-selection directive removes most of the premium without requiring authority. Across 47 additional contradictions, arbitrary directives were cheaper than authoritative rules in all four models (0.72–0.93×), while permission to report either value cost 1.3–2.8× more; within that frame, adding a command reduced cost by 19–60%, whereas merely asserting which passage was operative cost 1.4–3.0× more. A factorial experiment confirms that the key factor is specifying the response decision, not grammatical form or authority: declarative reporting instructions cost 0.93–1.05× commands, while claims about which passage was operative cost 1.55–3.71× more. Blindly fixing the answer saves tokens but sacrifices accuracy, failing 66–97% of conflicts resolvable by other documents. By contrast, an evidence-conditional stopping rule removes 46–88% of the cost on 150 previously unseen, unsettleable conflicts, returns CONFLICT on 94–100%, preserves or improves accuracy in two of three models, and repays its checking overhead when just 4.1–12.9% of conflicts are unsettleable. Response policy is therefore a first-class lever on the inference cost of retrieval.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Machine Learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_2XqoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper measures the extra completion tokens that reasoning models spend when two equally trusted passages in the context disagree, and asks what removes this cost. Across six open model configurations, an unresolved conflict costs 2.0–57× the tokens of matched agreeing passages on 32 constructed notes and 23 Wikipedia contradictions (≥176× in the cap-censored K2-Horizon-7B read), and almost all of the excess is reasoning. The central claim is that the cost comes from not being told what to report. One sentence that names the passage to answer from removes most of the cost, even when it explicitly disclaims authority, while an assertion that one passage is operative does not substitute for that decision. A factorial experiment on 47 further Wikipedia contradictions separates the grammatical form of the decision (0.93–1.05× the command) from replacing the decision with an operative-source claim (1.55–3.71×), in three models and two wordings. Because a fixed answer overrides evidence that settles the question, the paper also proposes an evidence-conditional stopping rule (answer CONFLICT only when the other documents leave the answer open) and tests its transfer on 150 new unsettleable items.

Critical strengths
  1. Clean causal isolation. The factorial in Section 6 separates the grammatical form of the decision (declarative vs. command, 0.93–1.05×) from replacing the decision with an operative-source claim (1.55–3.71×). The result holds in all three models and both wordings. This is the evidence that makes the main claim ("a decision, not authority") credible rather than a wording artifact, and it carries most of my recommendation.

  2. Careful measurement and statistics. Every estimate is a ratio with a 95% interval, every reported ratio is computed by two independent implementations, and the cap-censored K2-Horizon-7B result (≥176×) is reported as a lower bound. The stopping rule is tested for transfer on 150 unsettleable items that no model had seen: it removes 46–88% of the cost and keeps or improves settleable accuracy on two of three models. The measurements are more trustworthy than is usual for prompt studies of this kind.

Critical weaknesses
  1. The tasks are highly controlled, which limits the general claim. The two passages differ in one value, every call is a single-turn prompt, and the known-answer worlds are small synthetic document sets. There is no realistic multi-step RAG or agent setting, and the natural-text evidence is small (23 + 47 Wikipedia articles). The closing claim that response policy is "a first-class lever on the inference cost of retrieval" therefore reaches further than the evidence. I see this as a scope limit, not a defect in the evidence, but the claim should be scoped to match.

  2. The stopping rule only helps models that already tell settled from unsettled conflicts. In Study 2, Olmo-3-7B-Think answers CONFLICT on only 24–32 of 150 unsettleable items under the rule and saves nothing (1.13–1.15× no instruction). In the transfer test, gpt-oss-20b loses 6.7 points of settleable accuracy [4.3, 9.0], mostly through CONFLICT answers. The paper states this scope, but it gives no way to make the rule work for weaker models.

Questions for authors
  1. (Clarification) Do you have any result beyond two passages with one differing value, for example three or more sources, several conflicting values, or a multi-turn exchange? Such a result would change how far the general claim reaches. If there is none, I would read this as future work and ask that the claim be scoped.

  2. (Clarification) For a model such as Olmo-3-7B-Think, which does not separate settled from unsettled conflicts, did you try a variant of the stopping rule, for example an explicit check step before the answer, that recovers part of the saving?

3 comments

Thank you for your review. You asked us to scope the claim, to show results beyond two passages with one differing value, and whether an explicit check step helps weaker models.

W1 (The "first-class lever" claim reaches further than the evidence): We scoped the claim to match. We removed "first-class lever" and changed the title to say "Most of the Cost". The conclusion now limits the claim to one disputed quantity, mostly with two passages in one turn. We also keep apart what each corpus shows. Accuracy comes from the synthetic known-answer worlds, where the other documents either settle the conflict or leave it open. The Wikipedia articles measure token cost, since they have no known answer.

Q1 (Results beyond two passages with one differing value): Yes. On the 47 extension articles, we added more sources, several conflicting values and a multi-turn exchange: five passages with three distractors, a third source repeating one value, a third source adding a new value, and a second turn. Conflicts raise cost in every format, and the directive brings cost back within 1.42 times the agreeing cost (Appendix Table 24). Even when two sources outvote the assigned passage, the directive still reports that passage on 99 to 100% of calls (Appendix Table 25). On the closed models with three values or two turns, the directive leaves 1.08 to 2.00 times agreeing (Appendix Table 23).

W2 / Q2 (Weaker models, and an explicit check step before the answer): We ran your check-step variant, which first writes CHECK: yes or no. On Olmo-3-7B-Think the check is right only about half the time, on 49% of settleable and 58% of unsettleable items, and the variant does not recover the saving (Appendix Table 33). The stopping rule depends on exactly this judgment, so check accuracy shows which models it needs. On two more models, Nemotron-3-Nano-30B and Olmo-3-32B-Think, the check is right on only 3% and 8.3% of unsettleable items, and the rule adds settleable failures on both (Appendix Table 34). On gpt-oss-20b, almost all of the 6.8 points of settleable failures that the rule adds are abstentions. Of the 53 failing responses, 51 answer CONFLICT, and wrong answers fall from four to two (Appendix Table 16). The paper states this condition directly in Section 3.5.

Thank you again. Your suggestion of an explicit check step showed us which models the stopping rule needs, and your note on scope made our claims more precise. We would be glad to answer any further questions.

Reviewer_2XqoOpenReview ↗

Response to the authors

Thank you for the response and the additional experiments.

Your answers address my concerns.

  • W1: the revision removes "first-class lever", changes the title to "Removes Most of the Cost", and limits the conclusion to one disputed quantity. The claim now matches the evidence.
  • Q1: you added five passages with distractors, a third source, a third value and a second turn on the 47 extension articles. You report that the directive brings the cost back within 1.42 times the agreeing cost (Appendix Table 24). This answers my question on settings beyond two passages with one differing value.
  • W2 / Q2: you ran the check-step variant. You report that on Olmo-3-7B-Think the check is right on 49% of settleable and 58% of unsettleable items, and that the variant does not recover the saving (Appendix Table 33). Section 3.5 now states which models the stopping rule needs. This answers my question.

I keep my rating at 3 (weak acceptance).

Thank you for your response, we're glad that we addressed all your concerns. Happy to clarify anything else if needed.

Reviewer_bjK7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper highlights instances where the systems struggle more when they have to decide between two equally trusted passages that have conflicting information. The main contribution of the work is to show that the system works better when you directly tell it which answer to give, instead of explaining which information it should trust. The work explores different metrics that cover empirical evidence to validate it.

The authors conducted experiments to show how unresolved conflicts between documents increase LLM reasoning costs. They find that a single sentence fixes the answer and can eliminate most of the overhead cost. They also propose a stopping rule that reduces unnecessary reasoning costs by 46–88% of unsettleable cost on 150 new items, while maintaining accuracy on some models.

Critical strengths
  1. The paper focuses on reducing LLM reasoning cost, which is potentially a major practical problem. They mention that the reasoning segment carries the major cost on conflicting items. Fixing the answer with a simple strategy substantially reduces the overhead cost.

  2. The authors also explored an evidence-conditional rule that provides substantial savings while preserving accuracy for models that can distinguish settled from unsettled conflicts.

Critical weaknesses
  1. Although the paper shows that a single sentence fixes the answer while reducing the underlying costs, the reason why it works is still unclear. It might be because it reduces uncertainty or it is preventing repeated deliberation?

  2. The work is missing details on the correctness of the response. Reducing token consumption does not necessarily imply correct answers. For example, an extended discussion is needed to evaluate whether the selected passage remains reliable and does not contain misleading or incorrect information.

  3. Since the stopping rule does not generalise, there is a need to evaluate the approach under different conflict distributions and complex multi-source settings.

Questions for authors

In addition of the issues raised under Weaknesses: Since the central claim is cost reduction, could you include and report some absolute costs for a couple of closed-weight models?

3 comments

Thank you for your review. You asked why one sentence works, whether lower token consumption still means correct answers, how the stopping rule behaves under different conflict distributions, and what conflicts cost in absolute terms.

W1 (Reducing uncertainty or preventing repeated deliberation): Our evidence points to preventing repeated deliberation.

If the sentence worked only by reducing uncertainty about which source is right, then naming the operative passage would be enough. It is not. A sentence that names the operative passage but does not say what to report leaves a premium 1.39 to 2.98 times that of a command to report that passage (Appendix Table 13). Telling the model what to report does cut the cost. When we hold the rest of the sentence fixed and put the decision in place of a neutral clause, it removes a further 46 to 79% of the remaining premium (Appendix Table 31).

The reasoning traces show the repeated deliberation directly. Unresolved traces switch between the two values 9.1 to 57.2 more times per trace than directive traces. Against a length-matched neutral clause, the decision cuts 4.8 to 8.7 switches per 1,000 reasoning tokens (Appendix Tables 21 and 36). Inserting the decision in the middle of a trace cuts the remaining tokens to 0.35 to 0.85 times those after a neutral insert (Appendix Table 32).

W2 (Correctness of the response, and whether the selected passage is reliable): We measure correctness alongside cost, and we match the instruction to the conflict. The directive names the passage to report and says the choice is arbitrary, so it makes no claim that the selected passage is reliable. It is for conflicts that a policy must decide, such as which of two trusted sources to quote. When the other documents can decide, we use the stopping rule instead. A fixed choice would override that evidence, and in the known-answer test the directive fails 66 to 96% of settleable pairs (Appendix Table 11).

The stopping rule answers from the evidence when the evidence decides and answers CONFLICT otherwise. In the transfer test, it keeps settleable accuracy on Qwen3.8-27B and stays within 7 points of unresolved on the two gpt-oss models (Table 2). On gpt-oss-20b, it gives fewer wrong answers than unresolved, two against four, and 51 of the 53 failing responses are CONFLICT abstentions (Appendix Table 16). We grade all accuracy with markdown emphasis stripped from answers (Appendix Table 37).

W3 (Different conflict distributions and complex multi-source settings): The stopping rule pays once a modest share of conflicts are unsettleable, and its saving tracks how often it stops. Across conflict distributions in the transfer test, the rule costs less than unresolved once 4.1 to 12.9% of conflicts are unsettleable, depending on the model (Figure 7).

In multi-source settings with three different values, five passages or two turns, the rule answers CONFLICT on 79 to 95% of calls and costs 0.14 to 0.99 times unresolved (Appendix Table 24). When two of three sources repeat one value, models mostly treat the majority as the answer, and the rule answers CONFLICT on only 18 to 48% of calls. For that pattern the directive is the cheaper tool, at 1.10 to 1.37 times agreeing (Appendix Tables 24 and 25).

Q1 (Absolute costs for a couple of closed-weight models): An unresolved conflict doubles the dollar cost on Opus and Sonnet, and a decision removes it. In the closed-model tests at high effort, a sentence stating what to report brings cost back to the agreeing level or below (Appendix Table 22).

USD per 1,000 callsAgreeingUnresolvedDirectiveDecision
Opus 5.55.8411.886.365.66
Sonnet 5.51.963.941.761.66

Dollar figures are list-price estimates for input and output tokens, using per-call means. Batch pricing halves them (Appendix Table 22).

Thank you again. Your question about why one sentence works led us to test it directly, and the reasoning traces now answer it. We would be glad to answer any further questions.

Thank you for clearing up the major concerns. I want to keep the same rating of 3.

Thank you for your response. We're glad we cleared up your major concerns.

Reviewer_R8UCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper mainly investigates why large language models incur substantial additional reasoning overhead when confronted with contradictory information sources, and how such overhead can be reduced. The authors find that in RAG or agent systems, when two documents with similar credibility supply conflicting information, reasoning models tend to repeatedly analyze and compare different sources, leading to a marked rise in reasoning token consumption. Experiments show that conflicting information can multiply token consumption several-fold or even tens of times compared with scenarios where information is consistent.

The authors further explore the root of this extra overhead, drawing a key distinction between two explanations: “the model does not know which source to trust” and “the model does not know which answer to report in the end”. By designing varied prompts including authority assertions, arbitrary source assignment and free choice settings, they discover that explicitly instructing the model to report the answer from a designated source — even without guaranteeing that source is correct — can substantially cut reasoning costs. In contrast, simply letting the model pick answers freely or declaring a source authoritative usually yields weaker results. The authors therefore argue that uncertainty in answer decision-making is a major driver of higher reasoning overhead under conflicting conditions.

That said, while directly specifying a source saves tokens, it may overlook other evidence and degrade answer accuracy. To address this, the authors propose an Evidence-Conditional Stopping Rule: the model responds normally when available evidence suffices to determine the correct answer; when evidence cannot resolve the conflict, it outputs CONFLICT directly to avoid futile further reasoning. Experiments demonstrate that this strategy reduces token consumption by roughly 46%–88% on new undecidable conflict samples, though accuracy varies across models.

The core contribution of this paper lies not in a new model architecture or training algorithm, but in identifying that extra reasoning cost induced by conflicting information is tightly coupled with answer decision-making, alongside a simple conditional stopping strategy. In terms of novelty, the central debate surrounding the work is whether the authors have truly uncovered the underlying mechanism behind rising reasoning costs, or merely verified experimentally that explicit output instructions reduce model reasoning.

Critical strengths
  1. The finding that the key to reducing reasoning costs amid conflicts may lie not in identifying which information source is correct, but in clarifying what the model should ultimately report. The authors show that explicitly assigning a source for the answer can substantially cut reasoning tokens, even when the model is warned that the assigned source may be wrong. By contrast, reasoning costs remain high even if the model is allowed to freely choose between the two conflicting sources. Follow-up factorial experiments reveal that this effect stems primarily from whether the reporting decision itself is fixed, rather than from imperative phrasing. This offers a new behavioral-level explanation for overthinking in reasoning models.

  2. The clear trade-off between reasoning efficiency and answer correctness. Directly fixing the answer source drastically shortens reasoning, yet risks making the model overlook evidence that could resolve the conflict, leading to severe accuracy degradation. To address this, the authors propose an evidence-conditional stopping rule: the model answers normally when evidence is sufficient, and outputs a conflict declaration when evidence is inconclusive. Experiments demonstrate that this strategy yields substantial overhead reductions on some models while preserving or even improving answer accuracy. This moves the paper beyond mere observational analysis and toward practically useful mitigation strategies.

Critical weaknesses

While the paper includes extensive control experiments, it lacks a dedicated, self-contained methodology section. Critical details including prompt setups, data construction, control conditions and statistical procedures are largely relegated to appendices. The main text prioritizes results over consolidated explanations of key design choices. In particular, the paper’s central conclusions hinge on subtle semantic distinctions between different instructions, yet the main text does not systematically explain how these prompts isolate distinct causal factors. This raises barriers to understanding and evaluating the validity of the experiments.

The paper’s central claim is that extra reasoning overhead induced by conflicting information arises primarily from uncertainty over the reporting decision, rather than source authority per se. Although the authors provide behavioral evidence through diverse prompt comparisons and factorial experiments, these results mainly demonstrate that instructions alter token consumption. They do not directly uncover the internal mechanisms that generate additional reasoning. For instance, assigning an output source may reshape the model’s task interpretation, reasoning strategy, or instruction-following behavior. Accordingly, attributing the observed pattern to reporting decision uncertainty as the dominant cause requires deeper mechanistic analysis or targeted intervention validation.

The paper only cites 10 recent papers (2024‑2026). Focusing exclusively on recent LLM literature is acceptable, and the most foundational benchmarks, datasets and overthinking‑attack papers in this space are well‑covered. Even so, several highly relevant recent empirical papers studying reasoning‑model behaviour under conflicting evidence and prompt‑based conflict mitigation in RAG are missing.

Questions for authors
  1. Could the authors provide more direct mechanistic analyses, such as reasoning trajectory analysis, internal representation probing, or causal interventions, to further verify that uncertainty in the reporting decision is indeed the key driver of extra reasoning overhead, rather than merely a behavioral difference induced by prompt phrasing?

  2. Current experiments focus primarily on single-turn question answering and factual conflicts between two information sources. Practical systems, however, often involve multiple conflicting sources, varying degrees of source reliability, and complex multi-hop reasoning. Have the authors validated whether fixing the reporting decision still effectively reduces reasoning costs under conflicts involving multiple documents, latent contradictions, or sequential multi-turn decisions?

  3. The proposed Evidence-Conditional Stopping Rule essentially guides the model via prompts to output CONFLICT when evidence is insufficient. It bears similarities to existing refusal mechanisms, uncertainty estimation techniques, and early-stopping strategies for reasoning. Could the authors further clarify the novelty of this strategy relative to prior approaches and provide direct baseline comparisons?

3 comments

Thank you for your review. We answer each point in order. The interventions under W2 / Q1 are the core of this response.

W1 (A dedicated, self-contained methodology section): Section 2 is now that section. It covers the prompt setups, data construction, control conditions and statistical procedures, and it defines every condition in the text (exact sentences in Appendix Table 3). On how the prompts isolate distinct causal factors, each comparison changes one element of the instruction. Replacing a decision with a source claim changes two elements at once, so we vary them separately in the factorial and the crossed test (Appendix Table 5).

W2 / Q1 (Mechanistic analysis and targeted intervention validation): We ran two targeted interventions, the crossed test and the trace intervention, and a reasoning trajectory analysis. We did not probe internal representations, so our claim is behavioral: stating what to report reduces reasoning tokens.

Prompt phrasing. The crossed test holds the phrasing fixed. Every condition uses the sentence "If the two passages disagree, X, and either value counts as a full answer", and only X changes. With no sentence, the conflict premium is 2.93 to 15.58 times agreeing across the three open models. A neutral X of the same token length lowers it to 1.44 to 2.53 times. Replacing that neutral X with the decision, "the value from passage k is the one to report", lowers it further to 1.13 to 1.50 times. This last step removes 46 to 79% of the remaining log premium, and every interval for the decision-to-neutral ratio is below 1 (Appendix Table 31).

Trace intervention. We took the first four unresolved traces per article on 45 extension articles and cut each one at the sentence where the second value first appears. We inserted either the decision, "The value from passage k is the one to report.", or a neutral sentence, "I have now read both passages and the question.", and let the model continue. After the decision, the remaining completion tokens were 0.35 to 0.85 times those after the neutral sentence, with every interval below 1 on all three open models (Appendix Table 32).

Instruction-following and task interpretation. In the factorial, the declarative decision says "the value from passage k is the one to report", and the source claim says "only passage k states the operative fact". Both name the same passage, and models give that passage's value on 97.3 to 100% of responses under both. The source claim still costs 1.55 to 3.71 times as much as the decision, so instruction-following does not explain the gap (Appendix Table 19).

Reasoning trajectories. Unresolved traces switch between the two values 9.1 to 57.2 more times per trace than directive traces (Appendix Table 21). Longer traces allow more switches, so we also normalized by length. Against the length-matched neutral clause, the decision cuts 4.8 to 8.7 switches per 1,000 reasoning tokens (Appendix Table 36).

W3 (Missing recent empirical papers): We now cite 28 works. They include recent studies of knowledge conflicts, conflict resolution in retrieval, efficient reasoning and abstention (Section 4).

Q2 (Multiple conflicting sources, source reliability, multi-hop reasoning and multi-turn decisions): On the 47 extension articles, we added five passages, a third source, a third value and a second turn. Conflicts raise cost in every setting, and the directive brings cost back within 1.42 times the agreeing cost (Appendix Table 24). Our stated rule tests varying source reliability directly, because it tells the model which passage is authoritative. The directive calls the choice arbitrary and still leaves a smaller premium, 0.72 to 0.93 times the stated rule's (Appendix Table 13). Multi-hop reasoning and latent contradictions are outside the scope of this paper.

Q3 (Novelty relative to refusal mechanisms, uncertainty estimation and early-stopping strategies): Uncertainty-based methods ask whether the model is confident. The stopping rule asks whether the other documents settle the conflict, and it uses no confidence estimate, sampling or training. For uncertainty estimation, a five-sample self-consistency vote catches only 18% of unsettleable conflicts on gpt-oss-20b, because its samples agree on one value even when the documents cannot decide. The stopping rule catches 94% at a tenth of the cost (Appendix Table 35). For early stopping, low reasoning effort is cheaper than the directive on both gpt-oss models but adds 34.7 and 28.7 points of settleable failures (Appendix Table 26). For refusal, an abstention prompt is cheaper on settleable items but catches only 43 to 63% of unsettleable items, while the stopping rule catches 90 to 100% (Appendix Table 29).

Thank you again. Your request for targeted intervention validation led to the crossed test and the trace intervention. We would be glad to answer any further questions.

Reviewer_R8UCOpenReview ↗

Thank you for your response.

Thank you for your response. I now have a clearer understanding of your project. Would you be able to provide the experimental details? I will consider increasing my score.

Thank you for your response. Below are the experimental details. We first cover the four areas from your first weakness, then the analyses from W2 / Q1. Section 2 and the appendices give the full details.

Prompt setups: Each call has a system message and a user message. The system message is a fixed instruction, "You are a helpful assistant. Answer using only the two passages below. Give the final line as ANSWER: <value>.", followed by one condition sentence. The user message holds the two passages and the question. Only the condition sentence changes between conditions. Appendix Table 4 lists every condition sentence word for word.

Data construction: We started from 1,052 Wikipedia articles tagged as containing a contradiction. Each item uses two passages from one article that give different values for the same fact. Two blind readers kept 54 of 88 candidates, and a third reader kept 47. For each conflicting item, we also build two agreeing versions in which both passages give the same value. For the stopping rule, we generate synthetic document sets with known answers and check each one by exhaustive enumeration.

Control conditions: Every comparison uses the same article, passages and question, and changes only the condition sentence. In the extension and the crossed test, we also run every condition on the two agreeing versions. This lets us measure the extra cost of the conflict itself, separate from any cost the sentence adds. In the unresolved condition, the sentence is neutral filler.

Statistical procedures: We treat each article as one unit. For each article, we average completion tokens over repetitions and divide the conflicting cost by the average cost of its two agreeing versions. We average these ratios over the 47 articles on the log scale and report 95% t intervals. Comparisons between two conditions are paired within article. We also report Bonferroni simultaneous intervals.

Models and sampling: We run the open models with vLLM 0.28.0, with reasoning on and a cap of 126,976 completion tokens. Qwen3.8-27B (INT4) samples at temperature 0.6. Both gpt-oss models sample at temperature 1.0 with high reasoning effort. Appendix Table 6 gives every setting.

Crossed test: This test holds the wording fixed. Every condition uses the same sentence, "If the two passages disagree, X, and either value counts as a full answer", and only X changes. X is the decision, the source claim, both together, or one of two neutral clauses matched in token length. For example, the short neutral clause is "the text is printed in a standard font throughout this exercise", which has 11 tokens, like the decision. A sixth condition has no sentence. We ran Qwen3.8-27B and both gpt-oss models on all 47 articles, with every condition and its agreeing versions in one run, 8 times each (Appendix Table 31).

Trace intervention: We took four reasoning traces per article from the unresolved condition. We cut each trace right after the sentence where the second value first appears. At that point, we inserted the decision, the source claim, a neutral sentence or nothing, and let the model continue. We then compared how many tokens the model spent after the cut. With nothing inserted, the continued traces are about as long as the originals (0.94 to 1.20 times). We used 45 articles per model. On the other two, every trace either never mentions both values or has fewer than 50 reasoning words after the cut (Appendix Table 32).

Factorial: This test separates the form of the sentence from whether it states what to report. We used four sentences: a command ("give the value from passage k"), a declarative decision ("the value from passage k is the one to report"), the command with the source claim added, and the source claim alone. Each comes in two wordings. We ran them on the 47 conflicting articles, 16 times on Qwen3.8-27B and 8 times on each gpt-oss model (Appendix F).

Reasoning trajectories: We counted value switches in the existing traces, without generating new text. A switch is a change from mentioning one value to mentioning the other. We report switches per trace and per 1,000 reasoning tokens, averaged over articles, with 95% bootstrap intervals (Appendix Tables 21 and 36).

Code: The supplement contains the tasks and analysis code for every experiment. For the factorial, the trace intervention and the known-answer tests, a second, independent implementation reproduces the estimates. We will release all model responses with the camera-ready version.

Happy to clarify anything else.

Reviewer_Ao4KOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the phenomenon where reasoning-capable large language models (LLMs) incur a significant overhead in additional reasoning tokens when encountering conflicting retrieval sources within retrieval-augmented systems. Through a series of three progressive studies—conducted using various open-source models, custom-curated conflicting documents, and real-world contradictory examples from Wikipedia—the authors demonstrate that models do not require explicit information about which source is factually correct. Instead, providing a directive on the desired output content is sufficient to substantially reduce the reasoning overhead caused by conflicts; simply asserting the factual validity of a specific source does not yield the same benefit. Furthermore, the authors propose an "evidence-conditional stopping rule," which outputs "CONFLICT" when a contradiction cannot be resolved—a strategy that trades a small number of additional verification tokens for a better balance between token expenditure and task accuracy. This work clarifies the underlying mechanism by which conflicting evidence triggers "overthinking" in LLMs and quantifies both the benefits and side effects of various prompting interventions.

Critical strengths
  1. Rigorous controlled experimental design with a clear approach to hypothesis elimination. Rather than merely presenting a new method and reporting performance metrics, the study employs a series of controlled interventions using various prompts. Through a factorial experimental design, it disentangles two confounding variables—"reported decisions" and "source-based factual assertions"—and systematically rules out competing alternative hypotheses. This provides robust empirical evidence for understanding how models reason when faced with conflicting evidence, representing the core contribution of the paper.
  2. Comprehensive, transparent statistical analysis and a commitment to reproducibility. All results are reported with 95% bootstrap confidence intervals and adjusted for multiple comparisons using the Bonferroni correction; findings were verified through independent implementations using two separate codebases. The prompts and experimental setups are fully disclosed, with the appendix documenting edge cases—such as output truncation or repetitive loops—thereby facilitating replication and lowering the barrier for future researchers to verify the findings.
  3. Practical engineering implications. Addressing the issue of "reasoning token explosion" caused by conflicting information from multiple sources in Retrieval-Augmented Generation (RAG) systems, the paper proposes a low-cost, prompt-based solution. It further quantifies the trade-off between accuracy and token usage, identifying the "break-even" point. These insights offer valuable guidance for designing response strategies in real-world AI agents and retrieval-augmented systems.
Critical weaknesses
  1. The writing and narrative structure lack readability, creating a high barrier to entry that is unfriendly to the average reader. The paper is organized around three "Studies," which intermingle tasks, interventions, ablation studies, and statistical analyses, rather than following the conventional "problem–method–ablation–experiment" structure. The main text is cluttered with numerous ratios and confidence intervals for various versions, while many key concepts and formulas are relegated to the appendices. This makes it difficult for researchers new to the field to quickly grasp the core narrative, as they must frequently flip between tables and appendices to verify custom terminology (e.g., "directive," "assertion," "permission"), imposing an unnecessary cognitive burden.
  2. The experimental scenarios are limited, with insufficient validation of the findings' generalizability to real-world applications. All experiments are restricted to single-turn prompt settings; they fail to account for multi-turn dialogues, longer document contexts, or RAG pipelines that incorporate real-world noise. In practical production systems, inputs often contain a mix of irrelevant text and conflicting segments. It remains unclear whether the observed token-cost dynamics and intervention effects would hold true in such environments, which limits the engineering applicability of the conclusions.
  3. There is a bias in the selection of models; the study lacks testing on smaller models or closed-source reasoning models. The experiments rely exclusively on open-source reasoning models (ranging from 7B to 120B parameters) and omit mainstream closed-source alternatives. Furthermore, for certain models, the proposed intervention strategies proved largely ineffective; however, the paper fails to analyze the underlying model characteristics that led to these failures, leaving the boundaries of the models' capabilities insufficiently explored.
  4. The evaluation of accuracy is constrained by the artificial classification of conflicts into "settleable" and "unsettleable" categories, which may not reflect the distribution of conflicts encountered in real-world scenarios. The study relies heavily on manually constructed datasets with known answers to categorize conflicts. In reality, retrieval-based conflicts are rarely so neatly binary; they often stem from incomplete information or partial contradictions. Consequently, the "break-even" ratios derived from these synthetic datasets cannot be directly applied to real-world operational contexts.
  5. The paper lacks a comparative analysis against existing baselines—specifically, established methods for conflict-aware prompting and RAG-based conflict resolution. The paper focuses primarily on internal comparisons of prompt variants, yet it largely overlooks benchmarking against existing methods—such as those for conflict detection, evidence weighting, and conflict-avoidance prompting—proposed in the literature. Consequently, it fails to demonstrate the relative merits of the authors' proposed prompting strategies compared to established baselines, which weakens the positioning of the work.
Questions for authors
  1. For the Olmo-3-7B-Think model, the benefits of both "directive" and "stopping rule" interventions are significantly diminished; the model itself frequently produces errors on "settleable" samples even without any intervention. What intrinsic characteristics of the model do the authors believe cause these intervention strategies to fail? Could you provide simple, observable, a priori indicators that might help predict whether these strategies will be effective for a given model? (Clarification question; relevant to assessing the boundaries of the conclusions.)
  2. All experiments in this paper are conducted in a single-turn setting. If these decision-making instructions were embedded into a multi-turn dialogue—where conflicting information might persist in the historical context—would the effectiveness of token reduction diminish? Have any preliminary tests been conducted to explore this? (Clarification question.)
  3. The synthetic "known-answer" dataset relies on a distinction between "settleable" and "unsettleable" instances—a ground-truth label that is typically unavailable in real-world RAG systems. How, then, can the evidence-based stopping rule be implemented in the absence of such ground-truth labels? Are there any lightweight, heuristic-based alternatives? (Clarification question.)
  4. Why was there no comparative analysis against existing baselines for conflict-aware RAG prompting? Is this due to dataset incompatibility, or some other reason? It is suggested that such a comparison be included in future work. (Suggestion for future work.)
1 comment

Thank you for your review. We ran a new experiment or revised a section for each weakness and question you raised, and we answer them in order.

W1 (Readability and narrative structure): The paper now follows a problem, method, results order. We replaced the three "Studies" with named experiments. Section 2 is a self-contained method section that defines every custom term, including directive and permission, in plain text before any result uses it. What we called an assertion is now a source claim. Appendix Table 3 gives every exact sentence, and Table 1 lists each experiment with its data and models, so the main text no longer depends on flipping to the appendix.

W2 / Q2 (Multi-turn dialogue, irrelevant text and longer contexts): The saving holds in all three. We tested them on the 47 extension articles. In a two-turn dialogue, the directive keeps cost within 1.42 times the agreeing cost (Appendix Table 24). The conflict still costs extra on the second turn when the model writes its own first reply, so the problem persists in dialogue. With three irrelevant passages mixed in, the directive keeps cost within 1.34 times agreeing. With longer contexts of 10 to 40 unrelated excerpts, the directive still removes 56 to 68% of the premium on Claude Sonnet 5.5, on the log scale (Appendix Table 23).

W3 (Closed-source and smaller models): We added three mainstream closed models, Claude Haiku, Sonnet and Opus 5.5, and they pay the same premium. At every effort level, an unresolved conflict costs them 2.00 to 2.88 times the agreeing cost. A single sentence stating what to report brings them to 0.74 to 1.04 times agreeing (Appendix Table 22). Our smallest open models have 7B parameters. We answer your question about the model characteristics behind the failures under Q1.

W4 / Q3 (Ground-truth labels in real RAG systems): The stopping rule needs no settleable label at inference. The model itself decides whether the other documents settle the conflict. We use labels only to score accuracy and to measure check accuracy on a small sample. As a lightweight alternative without labels, a team can count CONFLICT answers and tokens saved on its own traffic. On our 47 unlabelled Wikipedia articles, the rule costs 0.16 to 0.73 times unresolved (Appendix Table 27). The break-even curves in Figure 7 then turn a measured CONFLICT rate into an expected saving, assuming our measured rates.

W5 / Q4 (Comparison against established baselines): We added the comparisons, and each baseline leaves more cost. For RAG-based conflict resolution, we ran MADAM-RAG from its official code. It pays its own conflict premium of 2.11 to 6.45 times agreeing on the extension articles, and on our synthetic items it costs 14 to 46 times the stopping rule. It also fails 74 to 97% of settleable cases, while the stopping rule fails at most 9.3% (Appendix Table 30). We did not run Astute RAG, because its internal-knowledge step has nothing to draw on in our fictional worlds. For conflict-aware prompting, we tested brevity, token-budget, conflict-handling and abstention prompts. Each costs more than the directive on all three open models (Appendix Table 26). A five-sample self-consistency vote catches 18% of unsettleable conflicts on gpt-oss-20b, and the stopping rule catches 94% at a tenth of the cost (Appendix Table 35).

Q1 (Simple, observable, a priori indicators): We found two. The first is check accuracy. We asked each model to write CHECK: yes or no before answering. gpt-oss-20b, where the rule works, gets this check right on 98% of settleable and 75% of unsettleable items. Olmo-3-7B-Think, where the rule fails, gets it right on only 49% and 58% (Appendix Table 33). So its failure goes with a measurable trait. It cannot reliably tell whether the other documents settle the conflict. The second indicator is cost. Where the rule works, the unresolved model already spends 3.50 to 18.09 times as much on conflicts the documents cannot decide as on those they settle. Olmo-3-7B-Think spends about the same on both, 1.06 times (Appendix Table 12). Both indicators need only a small labelled sample of 40 settleable and 30 unsettleable items. We then ran two more models, Nemotron-3-Nano-30B and Olmo-3-32B-Think. Their checks are right on only 3% and 8.3% of unsettleable items, and the stopping rule does not pay on either (Appendix Table 34).

Thank you again. Your questions led directly to the multi-turn, closed-model and baseline experiments, and the paper is stronger for them. We would be glad to answer any further questions.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Changes & Results

We thank the reviewers for careful and constructive reviews. The revised PDF makes these changes:

  • Title. We changed the title from "Telling Reasoning Models What to Report Removes the Cost of Conflicting Sources" to "Telling Reasoning Models What to Report Removes Most of the Cost of Conflicting Sources", so that it matches the measured effect. We also removed the claim that response policy is "a first-class lever on the inference cost of retrieval", and revised the abstract and conclusion to match.

  • Method and organization. A self-contained method section (Section 2) defines every condition in the text, and Table 1 maps each experiment to its data, models and results.

  • New experiments. We ran the main comparisons in harder settings (more passages, three sources, three values, two turns) and on three closed models. We compared the directive and the stopping rule against MADAM-RAG, self-consistency, four prompt baselines and low reasoning effort. We tested why the directive works with a crossed test and a trace intervention, ran a check-step variant, and ran the stopping rule on two more models.

  • Scoring. We now strip markdown emphasis from every answer before grading; Appendix Table 37 lists every value this changed. Token results are unchanged.

Our three main results are:

  1. Conflicts are expensive. Across 32 constructed notes and 23 Wikipedia contradictions, two equally trusted passages that disagree cost reasoning models 2.0 to 57 times the tokens of matched agreeing passages, almost all of it in reasoning (Appendix Table 7). We call this extra cost the conflict premium.

  2. Telling the model what to report removes most of the premium. On 47 further Wikipedia contradictions, a directive, one sentence that names the passage to report and says the choice is arbitrary, removes 81 to 99% of the premium on the log scale (Section 3.2). It needs no claim that either source is right.

  3. An evidence-conditional stopping rule keeps the saving where evidence cannot decide. The rule answers from the evidence when the other documents settle the conflict and answers CONFLICT otherwise. On 150 new unsettleable items it cuts cost by 46 to 88% and answers CONFLICT on 94 to 100% (Table 2).

Our reply to each review answers its points in order and names the experiment and table behind each claim.

Area_Chair_UrDsOpenReview ↗

Please engage in the discussion with authors

Dear Reviewers,

Please engage in the discussion with the authors, thank you.

AC

We thank the reviewers and the area chair for their time and feedback.

Reasoning models pay a steep price for conflicting sources. When we give a model two equally trusted passages with different values, it uses 2.0 to 57 times as many tokens as it does on matched passages that agree. Almost all of the extra goes to reasoning. We call this extra cost the conflict premium, and we measure it on 32 constructed notes and 23 Wikipedia contradictions. On Claude Opus 5.5 at high effort, a conflict doubles the cost, from 5.84 to 11.88 USD per 1,000 calls.

One sentence in the prompt removes most of the premium. It names the passage to report and says the choice is arbitrary, so it takes no side on which source is right. On 47 further Wikipedia contradictions, it removes 81 to 99% of the premium on the log scale. On Opus 5.5, it cuts the cost from 11.88 to 6.36 USD per 1,000 calls. A declarative wording of the same decision brings it to 5.66 USD, below the 5.84 USD for passages that agree.

The saving comes from settling what to report. Telling the model which source is authoritative does much less. In our factorial test, a sentence saying that one passage states the operative fact costs 1.55 to 3.71 times as much as a sentence saying that passage's value is the one to report. Models give the same answer under both on 97.3 to 100% of calls. When we insert the reporting decision into the model's own reasoning, right where the second value first appears, the model finishes with 0.35 to 0.85 times the tokens it uses after a neutral sentence.

When the other documents can decide, a fixed choice would override them, so we also give a stopping rule. The model answers from the evidence when it settles the conflict and answers CONFLICT otherwise. On 150 new items where the evidence cannot decide, the rule answers CONFLICT on 94 to 100% of them and cuts cost by 46 to 88%. On gpt-oss-20b, a five-sample self-consistency vote flags only 18% of these conflicts, and the rule flags 94% at a tenth of the cost. The rule needs a model that can judge whether the evidence settles the conflict, and a small labelled sample shows which models can.

Three of the four reviewers now recommend acceptance. R8UC, who raised the strongest soundness concerns and asked for mechanistic evidence, raised their rating after our crossed test, trace intervention and experimental details. 2Xqo said our response addressed their concerns, and bjK7 said it cleared up their major concerns. Ao4K, the only reviewer who does not recommend acceptance, has not responded to our rebuttal. Our response to Ao4K answers each of their points with a new experiment or a revised section. It adds closed models, multi-turn dialogue, longer contexts and comparisons with established baselines, and it restructures the paper in a problem, method, results order.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.