acceptodds
Withdrawn from ICLR 2027

When Evidence Changes, Does Reasoning Follow? Revising Language Model Conclusions Under Changing Evidence

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reasoning establishes relationships between premises and interpretations, which may be reconsidered when the premises change. We study how language models respond when relationships change while maintaining earlier reasoning in context. We change a fact that affects the correct answer while leaving the earlier reasoning in the context, then test whether models use the updated evidence or repeat an outdated conclusion. The experiments cover controlled arithmetic, logic, and navigation tasks, GSM8K, FinQA, CRUXEval, and Wikidata problems, and the STALEFLOW tool-action test, which examines whether updated information governs a subsequent action. We compare retaining the full history with dependency-based pruning: removing reasoning steps that depend on a changed premise, including their downstream conclusions, while preserving unaffected reasoning. This comparison tests whether stale reasoning obstructs revision and whether selectively removing it improves accuracy. Across these studies, we evaluate Qwen3-1.7B, 4B, 8B, 14B, and 32B; Qwen2.5-3B, 7B, and 72B; Llama-3.1-8B and 70B; Phi-4; Mistral-7B-Instruct-v0.3; and OLMo-2-7B, including base checkpoints, together with five OpenAI models. On the controlled problems, removing only the invalidated steps lowers Qwen3-4B-Thinking's rate of repeating the outdated answer from 0.969 to 0.070, and under immediate answering the effect appears in all 17 models tested and in all 40 model–dataset cells on the benchmark problems. A correction notice fails when models answer immediately but works when they reason first, so the better remedy depends on how the model answers. Across 1,800 multi-turn sessions, dependency-based pruning improves accuracy by 27.3 percentage points compared with retaining the full history. In STALEFLOW, Qwen2.5-7B uses an outdated value in 62.5% of immediate-action cases. These findings frame revision as maintaining relationships between evidence and conclusions: checking what has lost support, preserving what remains valid, and computing a new answer where needed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reasoning Revision, Belief Revision, Chain-of-Thought Reasoning, Context Management, Dependency Tracking, Multi-Turn Reasoning, Tool-Using Agents

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_WPzmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether large language models (LLMs) can appropriately revise their conclusions when the underlying evidence changes while previously generated reasoning remains in context. The authors identify a phenomenon in which LLMs tend to repeat outdated conclusions despite receiving updated evidence, and study how stale intermediate reasoning contributes to this behavior.

The paper introduces a dependency-based perspective on reasoning revision, where steps invalidated by an evidence update and their downstream consequences are selectively removed while unaffected reasoning is preserved. The authors construct controlled arithmetic, logic, and navigation tasks and extend their evaluation to GSM8K, FinQA, CRUXEval, Wikidata, and a tool-calling benchmark (STALEFLOW). Experiments across 17 LLMs demonstrate that retaining stale reasoning substantially increases the likelihood of repeating outdated answers, while selectively removing invalidated reasoning significantly reduces this effect. Further analyses investigate the influence of reasoning budgets and response procedures, examine the underlying mechanisms through attention-head ablation and hidden-state interventions, and explore revision-oriented training. Finally, experiments on 1,800 multi-turn sessions show that dependency-based pruning improves accuracy by 27.3 percentage points compared with retaining the full reasoning history.

Critical strengths
  1. Rigorous experimental design with well-controlled comparisons. The paper provides a systematic investigation of how stale reasoning affects LLMs' ability to revise conclusions. Carefully designed controls, including random step deletion, selective pruning, and coherent revised derivations, help isolate the impact of stale reasoning from other factors such as context length. These experiments provide convincing evidence for the paper's central findings.

  2. Comprehensive empirical validation and analysis. The paper evaluates the phenomenon across 17 LLMs and multiple reasoning tasks, demonstrating its prevalence. Further analyses of reasoning budgets, correction strategies, and internal mechanisms provide additional insights into the observed behavior. Experiments on tool-calling tasks and multi-turn conversations further highlight the practical relevance of the findings.

Critical weaknesses
  1. Limited novelty and insufficient differentiation from prior work. The core problem of revising conclusions under changing evidence has been extensively studied in belief revision and knowledge conflict research. Related studies, such as Belief-R, ReviseQA, and TRACK, have already investigated how LLMs update conclusions and propagate changes through multi-step reasoning. Although this paper provides more detailed empirical analyses of stale reasoning, its fundamental research question overlaps considerably with existing work. The paper does not sufficiently clarify its distinct contributions relative to these closely related studies, which weakens its novelty.

  2. Limited practical applicability of dependency-based pruning. The proposed dependency-based pruning relies on identifying invalidated reasoning steps and their downstream dependencies. However, the paper does not provide a concrete method for automatically constructing dependency graphs or identifying invalidated steps from arbitrary natural language reasoning. The experiments largely rely on known or identifiable dependencies in controlled settings. Consequently, while the results demonstrate the benefits of removing stale reasoning, the feasibility and effectiveness of this approach in realistic, complex reasoning scenarios remain unclear.

Questions for authors
  1. Novelty relative to existing research. How does this work fundamentally differ from existing studies on belief revision and reasoning under changing evidence, such as Belief-R, ReviseQA, and TRACK? Could the authors clarify the specific new insights beyond demonstrating that LLMs struggle to revise conclusions when evidence changes?

  2. Construction and applicability of reasoning dependency graphs. How are reasoning dependencies and invalidated steps identified in the experiments, particularly in the 1,800 multi-turn sessions? Are these dependencies obtained from ground-truth information or automatically inferred from reasoning traces? If ground-truth dependencies are used, to what extent can the reported improvements be expected to generalize to realistic scenarios where such dependencies are unavailable?

  3. Limited theoretical justification. The paper provides extensive empirical evidence for the influence of stale reasoning and the effectiveness of dependency-based pruning. However, the proposed dependency-graph formulation is not accompanied by a deeper theoretical analysis. Could the authors provide a formal characterization or theoretical explanation of why stale reasoning leads to outdated conclusions and under what conditions dependency-based pruning is expected to be effective? Such an analysis would help establish the generality of the findings beyond the evaluated settings.

Reviewer_Kr8mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether language models revise conclusions supported by earlier reasoning when the evidence changes. Across controlled arithmetic, logic, and navigation tasks, as well as mathematical, financial, code reasoning, and factual question-answering benchmarks, the authors compare retaining stale reasoning, removing invalidated steps, and providing correction notices. They further investigate tool actions, internal mechanisms, and training interventions. The results show that stale reasoning substantially increases the old-answer rate, while removing invalidated downstream reasoning mitigates this effect. The effectiveness of interventions also depends strongly on whether the model answers through immediate readout or generates a new solution. The authors therefore argue for maintaining context according to the dependencies between evidence and conclusions.

Critical strengths
  1. This paper examines whether models can maintain the validity of their conclusions during interactions in which facts change, extending the focus beyond solving static reasoning problems. This question is directly relevant to multi-turn conversations, agent memory, and tool use. The authors further distinguish detection, propagation, and execution, and report old-answer rate and revision accuracy separately.

  2. This paper reveals an important relationship between revision and the response procedure. The authors show that revision depends not only on whether models receive updated evidence, but also on how they process it. Consequently, a context intervention cannot be evaluated independently of the response procedure; its effectiveness must be assessed alongside the opportunity to rederive the answer and the available reasoning budget.

Critical weaknesses
  1. The explanation of failures in dependency maintenance remains incomplete. The authors’ ablation experiments show that the identified attention heads read both correct and stale conclusions. These results therefore primarily locate a general mechanism for reading stated answers, without directly establishing where models fail to detect premise changes or propagate their consequences.

  2. This paper has not established the end-to-end feasibility of dependency-based pruning in natural conversations. The controlled experiments can obtain dependency information from the task construction, but the authors do not evaluate pruning based on dependencies automatically inferred from natural-language transcripts. It therefore remains unclear whether the reported gains would persist when dependency identification involves omissions or errors.

  3. The multi-turn pruning experiments are insufficiently documented. Although the paper reports substantial accuracy gains, it provides limited detail on session construction, dependency acquisition, and results broken down by model and task. Comparisons with other context-cleaning strategies under matched context budgets are also insufficient. These limitations make the results harder to reproduce and the source of the gains harder to assess.

Questions for authors

see weaknesses

Reviewer_dAc1OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents an analysis study of “stale context” in an agent’s trajectory. In particular, they study the effect of evidence changes on reasoning and correctness. They assemble a benchmark of datasets across GSM8K, FinQA, CRUXEval, Wikidata, and STALEFLOW in which they intentionally edit lines of evidence and observe statistically significant drops in performance when evidence changes without the corresponding removal of invalid reasoning.

Specifically, they claim contributions around a benchmark of matched reasoning trace edits across five datasets. Experimentally, they establish that “stale reasoning” has a harmful effect, and that certain prompt-based or decoding methods cannot fix this effect. Through mechanistic analysis, they introduce a training objective that better detects when evidence changes to account for it. Finally, they mine 473 examples from WildChat as real-world examples where revisions occur.

The paper is a collection of 12 loosely-connected experiments about this topic of “stale evidence” that supports the above claims. The WildChat subset is not mentioned in the main paper except in the “contributions” section. There is a section in the appendix.

Overall review: While the methods and conclusions look reasonable, I admit I do not understand what the authors did, and the results are not too surprising either. I suggest rejection due to the narrow scope and lack of clarity in the contributions and research questions. See the last part of the review.

Critical strengths

The paper addresses a timely question: “What happens to our reasoning models when information is updated? Can we reuse our reasoning trace? What if the wrong trace remains in context?” The experiments establish that a performance drop will exist and naive methods do not work to recover the loss.

The resources released (set of edits matched between impactful and un-impactful) over a common set of reasoning datasets, along with mined real-world examples, could also be valuable for future research.

Critical weaknesses

First, the problem is somewhat unmotivated. In practice, models do not need to reuse their reasoning trace upon seeing new evidence: they can re-run the CoT reasoning with the updated evidence or question. If this is in a strictly multi-turn setting, wouldn’t the evidence change be a turn on its own and therefore detection should be an easy problem?

The experiments section is unclear, specifically what was the exact methodology being run, and what is the high-level conclusion of the results? The numbers can stay in a table, when they are in-text it is difficult to read or follow (even better would be a figure). The overall writing style in pages 3-7 are extremely dense and do not clearly correspond to the claims in the introduction. Overall, the organization of the paper could be improved significantly.

Questions for authors

Authors -- please address the weaknesses, you do not need to address every one of these questions in your rebuttal. I have many more questions but only picked out some of them

Concrete examples about the writing:

[Section 2]

  • L112: What is FF? Why is there an update tt? What is the difference between VV and L\mathcal{L}? What is C\mathcal{C}? Most of this notation is not referenced again.
  • L123 suggests that we care about detection, propagation, and execution, yet these terms are also not revisited later in the paper. I think the issue of detection would be quite interesting, the rest of the paper seems to be closer to propagation/execution. I’m not convinced that the “old-answer” rate is the same as detection; the model can select the old answer even if it knows the evidence has changed, maybe because it is too convinced by the reasoning?
  • L131: it would be better to introduce each dataset closer to what is in each dataset. That way, we can see whether it is better/worse at navigation or math when we look at the data tables
  • L139: the “immediate-readout study” and “silent-edit” study have not been defined yet. The point of the methodology section is to clearly define these studies. I admit I am lost in this section.

[Section 3]

  • Each of these paragraphs is describing a new experiment, but does not clearly explain what the experiment is. Each of the paragraphs is also describing a result, but does not explicitly state what the conclusion is. The methods need to be inferred, while the conclusions are often deep within the paragraph and also need to be inferred from the numbers.
  • “The effect” is used throughout this section, but it is never clear what “the effect” is. It feels like there should be an overarching hypothesis that your paper is claiming (i.e. “stale traces lead to wrong answers”) and you can then reference this claim throughout.
  • L249: I am quite interested in the details of the benchmark revisions, yet they don’t make sense to me. GSM8K is open-ended mathematical QA, so there should not be any “answer options”? So I must be misunderstanding something.
  • L368: was this a from-scratch 111M transformer? Or an existing model?

[Section G]

  • Why is this in the supplemental material?

Other suggestions and questions:

  • Did the model overfit to this dataset before? How come random deletion (in table 1) lowers the score so much?

  • This work cites several statistical tests (Holm, Wilson) - can you please include a citation to those works, as the reader may not be familiar with them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.