- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies budget internalization: fixed-budget post-training improves accuracy near the rollout cap but can cause response length and accuracy to saturate when more test-time compute becomes available. The authors demonstrate this across three model families and multiple reasoning tasks, then use GRPO ablations and SFT controls to argue that the effect is primarily driven by length-censored training signals and truncation handling, rather than a particular GRPO objective term.
Critical strengths
- The phenomenon is evaluated broadly across several model families, training datasets, and out-of-domain benchmarks, rather than relying on a single run or task.
- The paper contains useful diagnostic controls, especially answer forcing, truncation-handling ablations, and the comparison between naturally short versus artificially truncated SFT traces.
Critical weaknesses
- Limited novelty/significance. The central observation—that training under a limited reasoning horizon can impair extrapolation to larger test-time budgets—is closely related to prior work on training-horizon and test-time-compute extrapolation. The proposed explanation is also fairly intuitive: when long or truncated trajectories are systematically disfavored, the policy specializes toward shorter solutions. The paper characterizes this effect extensively, but does not establish a substantially new mechanism or provide a method that resolves the trade-off.
- The interpretation of the in-budget gains is unclear. Improved accuracy near the training cap may primarily reflect short-horizon specialization—learning to finish and emit an answer within the censoring window—rather than an increase in the model’s underlying reasoning capability. The paper’s own overlength-penalty and artificially truncated SFT experiments produce the same qualitative pattern, which is consistent with this interpretation.
- The empirical evidence does not adequately separate training effects from run/selection variance. Each training condition uses only one training seed, while checkpoints are selected by validation accuracy at the training budget. This selection criterion can itself favor checkpoints specialized to the in-budget regime. I would like to see multi-seed replications of at least the main fixed-budget GRPO conditions, reporting the full accuracy/length curves for each seed, together with results from a fixed or pre-specified checkpoint (e.g., the final checkpoint) rather than only the checkpoint selected for in-budget validation accuracy. This would clarify whether the reported crossover is a stable property of training rather than partly a consequence of seed and checkpoint selection.
Questions for authors
- What is the key scientific novelty beyond existing observations that reasoning models often fail to extrapolate beyond their training-time reasoning horizon?
- How do the authors distinguish genuine reasoning improvement from short-horizon specialization that simply makes the model more likely to finish and answer within the training budget?
- Could the authors show whether the same budget-internalization curves persist across multiple training seeds or across checkpoints not selected specifically for validation accuracy at the training budget?
TLDR: In our setting we start with models already operating at large reasoning budgets (128k) and further post-train them in a more affordable setting (8k). In that setup, we show that model stops using previous reasoning capabilities it had, this is also observed on OOD tasks. To our knowledge, this has not been previously shown in literature. At the same time, it has a far-reaching consequences for continual-learning research, where performing new task adaptation at an affordable research setting (8-16k) tokens of an already post-trainined model that reasons a lot (up to 128k) makes it harder to make any conclusions about positive/negative transfer, due to budget internalization phenomena we show. We’d like to highlight that developement of newer methods that allow for low-budget training while preserving test-time scaling, is an important new avenue for continual learning research as well.
Q1
We thank the reviewer for encouraging us to clarify our contribution and its relationship to prior work. Our work focuses on how further fixed-budget post-training can degrade the test-time scaling capabilities of an already capable reasoning checkpoint, even while improving its in-budget accuracy. Our motivation arose from studying catastrophic forgetting. When adapting an already capable reasoning model to an new task, we observed improvements on unrelated tasks when evaluated at a limited inference budget (e.g., 16k tokens). However, following evaluations at larger budgets revealed a large performance degradation relative to the starting checkpoint. Thus, a positive transfer at one evaluation budget can coexist with a large regression in previously demonstrated capabilities at another. While shorter generations following fixed-budget RLVR may be intuitive, the degradation of previously acquired reasoning capabilities does not necessarily follow. An alternative outcome would be reasoning compression, where the model achieves comparable or better accuracy using fewer tokens without sacrificing its high-budget performance. Distinguishing between these outcomes requires empirical investigation. Our contribution is to systematically characterize this phenomenon, which we term budget internalization, across three model families and multiple training tasks. In particular, we demonstrate that:
These findings identify an evaluation problem for adaptation and continual learning research: improvements at a limited inference budget do not establish preservation of the starting model’s reasoning capabilities. Evaluations must also cover budgets at which those capabilities were previously demonstrated; otherwise, apparent positive transfer can conceal large high-budget losses. Additionally, such evaluation procedure helps to distinguish genuine capability gain from just learning to conclude reasoning on time.
Q2 We verify this via budget forcing experiments. In Figure 4, we show the gap between the base model being forced to emit an answer at a given budget and a model trained with a small budget of 4k tokens. We observe that 1) forcing the base model to produce an answer does increase its scores across various evaluation budgets and various benchmarks, so some of the performance gain is likely due to the model just being able to complete its answers, 2) however, we also see that on smaller evaluation budgets (4k-8k), the trained model actually outperforms this intervention. Therefore, there is a performance gain that comes on top of simply being able to answer within a given budget.
Q3
We have updated Figure 1 to show additional runs on three different seeds and added Tables 13–14 in Appendix with per seed accuracy and mean response lengths at all budgets for validation selected and final (step-300) checkpoints. Across all training runs, switching checkpoint choice (from validation selected to final) changes average accuracy by less than 2.3% at the training budget and 1.9% at 64k. All training runs retain in-budget gains and high-budget deficits relative to the base model under both checkpoint choices, supporting the robustness of budget internalization to seed and checkpoint selection.
In the updated PDF we highlighted the updates with yellow.