- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates into whether the conventional end-to-end accuracy can well represent the recoverability and compressibility of intermediate states in LLMs, by designing an audited midpoint evaluation protocol that interrupts reasoning trajectories prior to final answer generation and assesses midpoint recoverability (R) by forcing answer elicitation and measures conditional survival (S) with KV-cache compression.The central finding is that post-training can substantially alter midpoint recoverability without detectable changes in final-answer accuracy, with the direction and magnitude depending on the training recipe
Critical strengths
- Interesting motivation that connects reasoning models with inference infrastructure. Accuracy on completed chains says little about whether a model's partial reasoning survives KV compression, and the paper turns this into a measurable quantity.
- The experiments are carefully controlled. Besides comparing public checkpoints, the authors train GRPO arms from the same base with and without a length penalty over multiple seeds, and use paired tests on the same questions.
- The authors are candid about limitations and report results that cut against their own headline, e.g., the fixed-token-budget reversal in Appendix S.
Critical weaknesses
- Unclear contribution: the main text and the appendix are inconsistent.
- Cutting at a fraction of the trace requires knowing the full trace length, which, as Appendix S itself says, "an operator cannot know online". Yet the abstract only says that the length-shortened checkpoints fall below Base. It does not mention that under a token budget, which is what a serving system actually has, these models do better.
- What does "added post hoc on 2026-09-08" mean?
- Inappropriate framing. I do not quite understand how the compression assumptions relate to reasoning-model serving. It also seems that you do not compare with R-KV?
Questions for authors
- Please consider rewriting parts of the main text and adjusting the figures for better readability.
- Panels (a) and (b) of Figure 1 should be aligned; the layout of Figures 3, 4, and 5 looks odd and should be adjusted.
- Several parts are far too long, e.g., the abstract (L12–46), L298–323, and L330–349.
- Please clarify: the abstract lists many specific numbers, but what does "Base 1.5B model" refer to? Which model family is it, and where do the public RL checkpoints come from? Further details would help.
- Suggestion: following Strength 1, why not also consider agent trajectories?