- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether large language models (LLMs) can appropriately revise their conclusions when the underlying evidence changes while previously generated reasoning remains in context. The authors identify a phenomenon in which LLMs tend to repeat outdated conclusions despite receiving updated evidence, and study how stale intermediate reasoning contributes to this behavior.
The paper introduces a dependency-based perspective on reasoning revision, where steps invalidated by an evidence update and their downstream consequences are selectively removed while unaffected reasoning is preserved. The authors construct controlled arithmetic, logic, and navigation tasks and extend their evaluation to GSM8K, FinQA, CRUXEval, Wikidata, and a tool-calling benchmark (STALEFLOW). Experiments across 17 LLMs demonstrate that retaining stale reasoning substantially increases the likelihood of repeating outdated answers, while selectively removing invalidated reasoning significantly reduces this effect. Further analyses investigate the influence of reasoning budgets and response procedures, examine the underlying mechanisms through attention-head ablation and hidden-state interventions, and explore revision-oriented training. Finally, experiments on 1,800 multi-turn sessions show that dependency-based pruning improves accuracy by 27.3 percentage points compared with retaining the full reasoning history.
Critical strengths
-
Rigorous experimental design with well-controlled comparisons. The paper provides a systematic investigation of how stale reasoning affects LLMs' ability to revise conclusions. Carefully designed controls, including random step deletion, selective pruning, and coherent revised derivations, help isolate the impact of stale reasoning from other factors such as context length. These experiments provide convincing evidence for the paper's central findings.
-
Comprehensive empirical validation and analysis. The paper evaluates the phenomenon across 17 LLMs and multiple reasoning tasks, demonstrating its prevalence. Further analyses of reasoning budgets, correction strategies, and internal mechanisms provide additional insights into the observed behavior. Experiments on tool-calling tasks and multi-turn conversations further highlight the practical relevance of the findings.
Critical weaknesses
-
Limited novelty and insufficient differentiation from prior work. The core problem of revising conclusions under changing evidence has been extensively studied in belief revision and knowledge conflict research. Related studies, such as Belief-R, ReviseQA, and TRACK, have already investigated how LLMs update conclusions and propagate changes through multi-step reasoning. Although this paper provides more detailed empirical analyses of stale reasoning, its fundamental research question overlaps considerably with existing work. The paper does not sufficiently clarify its distinct contributions relative to these closely related studies, which weakens its novelty.
-
Limited practical applicability of dependency-based pruning. The proposed dependency-based pruning relies on identifying invalidated reasoning steps and their downstream dependencies. However, the paper does not provide a concrete method for automatically constructing dependency graphs or identifying invalidated steps from arbitrary natural language reasoning. The experiments largely rely on known or identifiable dependencies in controlled settings. Consequently, while the results demonstrate the benefits of removing stale reasoning, the feasibility and effectiveness of this approach in realistic, complex reasoning scenarios remain unclear.
Questions for authors
-
Novelty relative to existing research. How does this work fundamentally differ from existing studies on belief revision and reasoning under changing evidence, such as Belief-R, ReviseQA, and TRACK? Could the authors clarify the specific new insights beyond demonstrating that LLMs struggle to revise conclusions when evidence changes?
-
Construction and applicability of reasoning dependency graphs. How are reasoning dependencies and invalidated steps identified in the experiments, particularly in the 1,800 multi-turn sessions? Are these dependencies obtained from ground-truth information or automatically inferred from reasoning traces? If ground-truth dependencies are used, to what extent can the reported improvements be expected to generalize to realistic scenarios where such dependencies are unavailable?
-
Limited theoretical justification. The paper provides extensive empirical evidence for the influence of stale reasoning and the effectiveness of dependency-based pruning. However, the proposed dependency-graph formulation is not accompanied by a deeper theoretical analysis. Could the authors provide a formal characterization or theoretical explanation of why stale reasoning leads to outdated conclusions and under what conditions dependency-based pruning is expected to be effective? Such an analysis would help establish the generality of the findings beyond the evaluated settings.