- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents a series of analyses on the termination of DeepSeek-R1-Distill-Qwen-7B and 1.5B on three math benchmarks: GSM8K, MATH-500, and AIME 1983–2024. They showed that short thinking budgets can improve accuracy on GSM8K but reduce accuracy on MATH-500, and that termination on AIME is sensitive to decoding settings and hardware. They also showed that a 4,000-token cap reduces inference compute by approximately 39% without lowering accuracy compared with a 10,000-token ceiling. They trained probes across model layers and token positions to predict thinking termination on AIME, and showed that the probes were largely unable to do so. They concluded that setting token caps is enough to reduce inference costs, while probing for termination is of limited use in this setting.
Critical strengths
- The paper describes the experimental configurations in detail, supporting the reproduction of their results. This is a strength, but it does not outweigh my concerns about the paper’s contributions and whether its conclusions are well supported.
Critical weaknesses
- The paper’s contribution is not obvious. Sections 4 - 6 largely present trivial results or reinforce well-known findings, while Section 7, which most directly addresses the main claim, is short and not convincing. This leaves the paper without a clear, substantial contribution to justify acceptance.
- The probe training setup is problematic. With n << d, the training data may be insufficient to learn a meaningful probe, so the conclusions in Section 7 are not well supported. The reported failure could reflect limitations of the training setup rather than the absence of predictive information, undermining the paper’s main conclusion about probing.
- There is no direct comparison between token caps and probes, despite the main claim being that token caps are better than probes. Instead, the authors present a series of analyses that are not directly comparable. Without a comparison under matched compute budgets or accuracy targets, the paper does not establish the claimed advantage of token caps, which is central to my overall assessment.
Questions for authors
- With n << d, how do you distinguish insufficient probe training from a lack of predictive information in the hidden states?
- What evidence supports the claim that token caps are better than probes, given the lack of a direct comparison?
2 comments
Thank you for the response and revisions.
- The matched-compute comparison is a useful addition, but the reliance on simulated probe scores limits what it establishes about how a trained probe would perform against a fixed cap.
- The new analyses make the conclusion better supported, though the narrower claim is less significant.
- The revision makes the contribution clearer, but I still find it limited given the narrow experimental setting and the reliance on simulation for the central comparison.
The revised manuscript is better, but the reliance on simulation for the central comparison and the limited contribution still fall short of the standard for ICLR, so I retain my score of 1 (clear rejection).
Thank you. Your central point, that we claimed caps beat probes without comparing them, is correct. We added the comparison.
Direct comparison at matched compute (new Sec. 7.5). For each problem we have the outcome and token count under every cap and under an early forced exit at ~150 tokens. A probe flags likely non-terminating generations at token 150 (simulated at a given AUC, since our real probe was trained on a different sample); flagged runs exit, the rest run under cap C. Best accuracy at a compute saving at least that of each cap:
At ≤40% savings, even a near-perfect probe adds ≤2 points, because forced exits at the cap already rescue looping generations (16% at 4k) and an exit at token 150 rescues 3%. Probes matter only at aggressive budgets, and only at AUC ≥0.85. This is the evidence for the title claim, and it does not depend on our probe failing.
n≪d. Agreed: our original null alone cannot separate "insufficient training" from "no information", and we no longer claim it does. New analyses: (1) PCA-reduced probes (k=2–50, fit within folds) and strong L2 (C=10⁻⁴) at all nine positions: best 0.69–0.70 post hoc, 0.61 under nested CV with the configuration selected inside training folds (p=0.016, 500 permutations), i.e. a weak signal; (2) a problem-level ceiling of AUC 0.83–0.87, so information exists, and a better-trained probe may find it. Section 7 is now framed as a bounded result for early-activation linear probes at n=200.
Contribution. We restated it: on hard problems, compute waste in R1-distilled models is dominated by generations that never terminate (65% of tokens), and termination is unstable (23–24% item flips, replicated on a second sample). Forced exits rescue part of those generations, which bounds what any early predictor can add over a cap. We now present Sec. 4 as supporting context rather than as a contribution.