- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies why LLM accuracy can decline as chain-of-thought (CoT) reasoning grows longer. The paper attributes this decline partly to reduced attention to important conclusions reached earlier in the trace, making them harder to use in subsequent reasoning.
The proposed InsightReplay method periodically extracts key conclusions from the reasoning trace and replays them near the current generation position. The core method retains the full reasoning trace, allowing the model to continue with both the original context and the extracted insights. The paper also develops a theoretical model of insight accessibility and derives conditions under which replay can shift the optimal reasoning length rightward and increase peak accuracy.
Experiments span three model families, two size tiers, and four reasoning benchmarks. Three-round InsightReplay reports gains over standard CoT in all 24 settings, with a macro-average improvement of 1.65 percentage points. The paper further presents attention diagnostics, reasoning-length analyses, GRPO post-training experiments, and an agentic history-compression variant evaluated on SWE-Bench Verified.
Critical strengths
S1. Insight accessibility offers a useful perspective on performance degradation in long CoT.
The paper goes beyond the relationship between reasoning length and accuracy to examine how models access earlier intermediate conclusions. The experiments provide evidence that full reasoning traces and extracted insights are complementary, while attention scores to critical insights decline with token distance. These findings offer useful empirical motivation for explicit reasoning-state management.
S2. The method is simple, training-free, and shows consistent gains across models and tasks.
InsightReplay requires neither architectural changes nor additional training at inference time. Experiments across multiple model families and reasoning benchmarks show broadly consistent accuracy improvements, with particularly notable gains on coding tasks. The GRPO experiments also report higher peak validation accuracy and improved late-training stability.
Critical weaknesses
W1. Insufficient comparisons with relevant alternatives under matched inference costs.
The main experiments compare InsightReplay with standard CoT and Verify-Only. Comparisons with summarize-and-restart methods such as InftyThink and PENCIL are missing, so the benefit of retaining the full reasoning trace remains unclear. It is also unclear whether insight replay uses additional inference compute more effectively than other approaches. IR3 uses more tokens than Verify-Only, so the accuracy difference cannot be attributed solely to replay.
Comparisons with representative context-management and test-time scaling methods at matched end-to-end costs would clarify whether InsightReplay offers a practical advantage.
W2. The contributions of different operations to the reported gains are not sufficiently isolated.
InsightReplay combines several operations, including insight extraction, summarization, replay near the generation position, question restatement, and further verification. Although the attention analysis shows that attention scores decline with distance, it does not directly establish that bringing conclusions closer causes the reported accuracy improvements.
The paper lacks controlled comparisons that isolate these operations. For example, replaying the same insights at different positions, comparing verbatim conclusions with regenerated summaries, or using length-matched control content under identical prompts and continuation budgets would help identify the source of the gains.
Questions for authors
Q1. What is the full inference cost of InsightReplay, and do its gains persist under matched total costs?
Table 2 reports completion tokens, while Appendix I provides throughput-amortized batch time for IR1, including insight extraction and continuation. However, individual-request latency, complete token-processing costs, and IR3 end-to-end runtime remain unclear.
Could the authors report total input/output tokens, cached input tokens, and end-to-end latency or GPU compute cost across all model calls for IR1 and IR3? Could they also compare accuracy with Verify-Only and representative test-time scaling methods under matched total compute budgets?
Q2. How statistically robust are the positive IR3 gains across all 24 settings?
Table 2 reports positive IR3 gains across all settings, but some improvements are as small as 0.1–0.2 percentage points. Appendix J provides independent replication and paired-bootstrap confidence intervals for only six IR1 settings, leaving the statistical robustness of the main IR3 results unclear.
Could the authors report problem-level paired-bootstrap 95% confidence intervals and win/tie/loss counts for IR3 versus Base across all 24 settings, using the existing paired trajectories? This would help determine whether the consistently positive point estimates represent statistically robust improvements.
Q3. How do the different history-retention strategies in core InsightReplay and its agent variant fit the paper's motivation?
Section 2.1 motivates retaining the original reasoning trace alongside extracted insights, and the core method in Section 3.1 follows this design. However, the SWE-Bench variant in Appendix E replaces the interaction history with a summary of accumulated findings, introducing a different history-management mechanism.
Why does the agent variant replace the history rather than retain it? Is this primarily due to context-window constraints? Could the authors compare full-history retention and summary replacement on SWE-Bench while keeping the model, workflow, and prompts otherwise identical? Such a comparison would clarify the role of history compression in the agentic variant and its relationship to the core InsightReplay mechanism.