acceptodds
Under review as a conference paper at ICLR 2027

Stateful Reasoning via Insight Replay

Abstract

Chain-of-Thought (CoT) reasoning has become a foundation for eliciting multi-step reasoning in large language models, but recent studies show that its benefits do not scale monotonically with chain length: while longer CoT generally enables a model to tackle harder problems, on a given problem, accuracy typically increases with CoT length up to a point, after which it declines. We identify a major cause of this phenomenon: as the CoT grows, the model's attention to critical insights produced earlier in the trace gradually weakens, making those insights progressively less accessible when they are most needed. Therefore, we propose InsightReplay, a stateful reasoning approach in which the model periodically extracts critical insights from its reasoning trace and replays them near the active generation frontier, keeping them accessible as the reasoning scales. Extensive experiments on a 2×3×4 benchmark grid, covering model scales 8B,30B, model families Qwen3.5,DeepSeek-R1-Distill-Qwen,Gemma-4, and reasoning benchmarks AIME,HMMT,GPQA Diamond,LiveCodeBench v5, show that 3-round InsightReplay yields accuracy gains across all 24 settings, with an averaged improvement of +1.65 points over standard CoT, and a largest single-setting gain of +9.2 points on R1-Distill-32B's LiveCodeBench v5 subset. Our results suggest that the effectiveness of test-time scaling depends not only on how much a model reasons, but also on whether critical intermediate insights remain accessible throughout long reasoning trajectories.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language model reasoning, Long chain-of-thought, Test-time scaling, Inference-time intervention, Attention decay, Context management, Reasoning models, Reinforcement learning with verifiable rewards

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_34KdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why LLM accuracy can decline as chain-of-thought (CoT) reasoning grows longer. The paper attributes this decline partly to reduced attention to important conclusions reached earlier in the trace, making them harder to use in subsequent reasoning.

The proposed InsightReplay method periodically extracts key conclusions from the reasoning trace and replays them near the current generation position. The core method retains the full reasoning trace, allowing the model to continue with both the original context and the extracted insights. The paper also develops a theoretical model of insight accessibility and derives conditions under which replay can shift the optimal reasoning length rightward and increase peak accuracy.

Experiments span three model families, two size tiers, and four reasoning benchmarks. Three-round InsightReplay reports gains over standard CoT in all 24 settings, with a macro-average improvement of 1.65 percentage points. The paper further presents attention diagnostics, reasoning-length analyses, GRPO post-training experiments, and an agentic history-compression variant evaluated on SWE-Bench Verified.

Critical strengths

S1. Insight accessibility offers a useful perspective on performance degradation in long CoT.

The paper goes beyond the relationship between reasoning length and accuracy to examine how models access earlier intermediate conclusions. The experiments provide evidence that full reasoning traces and extracted insights are complementary, while attention scores to critical insights decline with token distance. These findings offer useful empirical motivation for explicit reasoning-state management.

S2. The method is simple, training-free, and shows consistent gains across models and tasks.

InsightReplay requires neither architectural changes nor additional training at inference time. Experiments across multiple model families and reasoning benchmarks show broadly consistent accuracy improvements, with particularly notable gains on coding tasks. The GRPO experiments also report higher peak validation accuracy and improved late-training stability.

Critical weaknesses

W1. Insufficient comparisons with relevant alternatives under matched inference costs.

The main experiments compare InsightReplay with standard CoT and Verify-Only. Comparisons with summarize-and-restart methods such as InftyThink and PENCIL are missing, so the benefit of retaining the full reasoning trace remains unclear. It is also unclear whether insight replay uses additional inference compute more effectively than other approaches. IR3 uses more tokens than Verify-Only, so the accuracy difference cannot be attributed solely to replay.

Comparisons with representative context-management and test-time scaling methods at matched end-to-end costs would clarify whether InsightReplay offers a practical advantage.

W2. The contributions of different operations to the reported gains are not sufficiently isolated.

InsightReplay combines several operations, including insight extraction, summarization, replay near the generation position, question restatement, and further verification. Although the attention analysis shows that attention scores decline with distance, it does not directly establish that bringing conclusions closer causes the reported accuracy improvements.

The paper lacks controlled comparisons that isolate these operations. For example, replaying the same insights at different positions, comparing verbatim conclusions with regenerated summaries, or using length-matched control content under identical prompts and continuation budgets would help identify the source of the gains.

Questions for authors

Q1. What is the full inference cost of InsightReplay, and do its gains persist under matched total costs?

Table 2 reports completion tokens, while Appendix I provides throughput-amortized batch time for IR1, including insight extraction and continuation. However, individual-request latency, complete token-processing costs, and IR3 end-to-end runtime remain unclear.

Could the authors report total input/output tokens, cached input tokens, and end-to-end latency or GPU compute cost across all model calls for IR1 and IR3? Could they also compare accuracy with Verify-Only and representative test-time scaling methods under matched total compute budgets?

Q2. How statistically robust are the positive IR3 gains across all 24 settings?

Table 2 reports positive IR3 gains across all settings, but some improvements are as small as 0.1–0.2 percentage points. Appendix J provides independent replication and paired-bootstrap confidence intervals for only six IR1 settings, leaving the statistical robustness of the main IR3 results unclear.

Could the authors report problem-level paired-bootstrap 95% confidence intervals and win/tie/loss counts for IR3 versus Base across all 24 settings, using the existing paired trajectories? This would help determine whether the consistently positive point estimates represent statistically robust improvements.

Q3. How do the different history-retention strategies in core InsightReplay and its agent variant fit the paper's motivation?

Section 2.1 motivates retaining the original reasoning trace alongside extracted insights, and the core method in Section 3.1 follows this design. However, the SWE-Bench variant in Appendix E replaces the interaction history with a summary of accumulated findings, introducing a different history-management mechanism.

Why does the agent variant replace the history rather than retain it? Is this primarily due to context-window constraints? Could the authors compare full-history retention and summary replacement on SWE-Bench while keeping the model, workflow, and prompts otherwise identical? Such a comparison would clarify the role of history compression in the agentic variant and its relationship to the core InsightReplay mechanism.

Reviewer_X7KEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the problem of performance degradation in long Chain-of-Thought reasoning. The authors argue that, as reasoning becomes longer, models gradually pay less attention to important intermediate conclusions, which may reduce their ability to use previously derived information. To address this, the paper proposes InsightReplay, which extracts key conclusions from earlier reasoning and reintroduces them near the current generation position. The authors also provide a theoretical analysis and evaluate the method across six models and four reasoning benchmarks. Additional experiments investigate whether this reasoning strategy can improve reinforcement learning during post-training.

Critical strengths

1. An interesting problem and motivation. I think the paper addresses a meaningful question: why longer reasoning does not always lead to better performance. The analysis of how attention to earlier insights changes with reasoning length provides an interesting perspective on this problem.

2. Relatively broad experimental coverage. The authors evaluate multiple model families and reasoning tasks. I also appreciate the inclusion of a Verify-Only baseline, which helps distinguish the benefit of replaying intermediate conclusions from simply encouraging the model to continue reasoning.

Critical weaknesses

1. Limited methodological novelty and unclear implementation of the proposed method.

My main concern is that InsightReplay appears to be a relatively simple combination of existing ideas, particularly prompting models to continue reasoning and summarizing intermediate conclusions. Similar ideas have already been explored in previous works. I do not see a sufficiently clear methodological contribution that distinguishes this work from existing approaches.

I also found it difficult to understand what exactly constitutes InsightReplay across different experiments. Section 3.1 describes a periodic process that alternates between reasoning and insight extraction. However, Appendix D describes an inference procedure that first completes the original reasoning, then uses a separate model call to extract insights before continuing. More importantly, the main post-training experiment in Section 5 appears to use only a fixed continuation prompt, as described in Appendix G, without the explicit insight extraction used in the inference experiments.

These are substantially different implementations, yet their results are presented as evidence for the same method. The theoretical analysis also seems somewhat limited. Its main conclusions largely follow from assuming that replayed insights remain more accessible than earlier ones. While this is a reasonable intuition, I am not convinced that the analysis provides much additional understanding of the actual reasoning process.

2. Limited empirical gains and insufficient evaluation of computational cost.

I am not convinced that the reported improvements justify the additional computation.

In Table 2, IR1 improves micro-averaged accuracy from 64.82% to 65.47%. However, the much simpler Verify-Only baseline already achieves 65.18%. This means that the additional improvement from insight extraction and replay is only 0.29 percentage points.

IR3 achieves a larger gain, but it also uses approximately 36% more completion tokens than the baseline. Although some individual settings show encouraging improvements, the overall gains remain relatively modest, especially considering the additional extraction and continuation steps.

Furthermore, the reported completion-token counts do not fully characterize the inference cost. Appendix I explains that extracting insights requires separate model calls, which introduce additional processing and prefill overhead. The paper reports wall-clock measurements for IR1, but not for IR3. Without complete end-to-end measurements, it is difficult to assess the practical efficiency of the proposed approach.

I also find the baseline comparisons insufficient. The experiments mainly compare against standard CoT and a simple continuation prompt. Since the paper discusses related methods such as InftyThink, PENCIL, and TokenSkip, I would expect a more comprehensive comparison under matched computational budgets.

Additional concerns about presentation and reproducibility.

The writing and organization could be significantly improved. Important implementation details are scattered across different appendices, making the method unnecessarily difficult to follow. Some figures also have noticeable formatting problems.

I noticed that the manuscript uses numerical citations instead of the author-year format specified in the ICLR template. Since the main text already fills nine pages, I think the formatting compliance should be checked.

Finally, I could not find an anonymous code repository in the provided materials. Given the multiple inference passes and custom training procedures, releasing the implementation would be important for reproducibility.

Overall, I find the problem interesting, but the limited novelty, unclear methodological consistency, and modest gains relative to the additional computation significantly weaken the contribution.

Questions for authors
  1. Novelty and method design: What is the main methodological difference between InsightReplay and existing approaches such as reasoning continuation and intermediate-state summarization? What evidence shows that replaying the actual insights, rather than simply adding a verification prompt, is responsible for the improvements?

  2. Consistency between inference and training: Could the authors clarify why the post-training experiment in Section 5 uses a fixed continuation cue, while the inference method involves explicit insight extraction and replay? Does the main RL experiment actually perform the same InsightReplay procedure?

  3. Accuracy and efficiency: Given the small improvement of IR1 over Verify-Only in Table 2, how do the authors justify the additional extraction overhead? Could the authors report the full end-to-end latency of IR3 and compare it against a multi-round Verify-Only baseline under matched computational budgets?

  4. Experimental comparisons and reproducibility: Could the authors provide direct comparisons with relevant summarization or reasoning-management methods under comparable budgets? Is the implementation available for reproducing the inference and training experiments?

  5. Presentation and experimental details: Could the authors clarify the identity of the external LLM judge referred to as "Claude Fable 5" in Appendix K.1? I would also appreciate clarification regarding the use of numerical citations and whether the paper follows the official ICLR 2027 formatting requirements.

Reviewer_8wBQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes InsightReplay, which extracts intermediate conclusions from a reasoning trace, appends them near the generation frontier, and resumes reasoning while retaining the original trace. It motivates this procedure through answer-probability and attention diagnostics and a conditional theoretical model. Across six models and four benchmarks, three-round replay improves reported accuracy over standard CoT in all 24 settings, with a macro-average gain of 1.65 percentage points.

Critical strengths
  1. Practical gains supported by broad evaluation and a paired rerun. The intervention is simple and requires no additional training in the main inference setting. Table 2 spans six models and four benchmarks, and Appendix J adds an independent, problem-level paired replication. In that rerun, the IR1 gains over Base on LiveCodeBench are +3.2 pp for Qwen3.5-35B-A3B and +4.9 pp for R1-Distill-32B, with positive 95% confidence intervals. This provides substantive evidence that the complete intervention can improve reasoning outcomes, which is the strongest argument for acceptance.

  2. Useful analysis of retained reasoning and replay risks. Table 1 separates retaining the reasoning trace from replacing it with insights, and supports their complementarity. The automated audit in Appendix K distinguishes extraction distortions from inherited reasoning errors; it also acknowledges that replay can reinforce mistakes. Appendices I and L clarify serving costs, the limited LSTM analogy, and differences between the inference and agentic variants. These analyses make the proposal more informative and its limitations easier to assess.

Critical weaknesses
  1. My main concern is the attribution of the improvement to replay. Two comparisons matter here:
  • IR3 uses 23,032 average completion tokens versus 18,987 for Verify-Only (VO), about 21% more. Given this difference, Section 4.2's attribution of more than 60% of the gain to extraction and replay rather than additional thinking time seems too strong.

  • IR1 is closer to VO in token count, but the macro-average advantage is 0.29 pp. The paired intervals in Appendix J compare IR1 with Base, so they do not resolve this incremental comparison.

Appendix L.3 acknowledges the budget limitation. I find its caution difficult to reconcile with the stronger claim in the main text: the full intervention may be useful, but its added value beyond further verification and computation remains less clear.

  1. The practical results are more convincing to me than the explanation. The filler and random-span experiments suggest distance sensitivity; they do not establish that attention decay causes the reasoning failures or explains the gains. Table 1 changes the original trace rather than only the replay block, while Appendix K audits insight quality without isolating its effect on continuation. The realized-length curves are descriptive, as Appendix L.3 acknowledges. These caveats are welcome, but the abstract and conclusion still go further than the evidence supports. This matters because the mechanism is a central part of the claimed contribution.
Questions for authors
  1. What evidence supports the more-than-60% attribution in Section 4.2, given the different IR3 and VO token totals? Could the existing paired outputs provide a problem-level confidence interval for IR1 versus VO, or could a budget- and round-matched comparison be reported for one representative setting?

  2. Which existing result distinguishes restored access to earlier conclusions from structured re-verification? A replay-content control retaining the original trace and verification cue would help. Otherwise, would the authors narrow the causal and optimal-length claims to hypotheses supported by diagnostics? I do see practical value in the method. A clearer account of what the experiments establish could change my recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.