acceptodds
Under review as a conference paper at ICLR 2027

Budget Internalization: Fixed-Budget RLVR Trades Test-Time Scaling for In-Budget Accuracy

19 views

Abstract

Reasoning models often become more accurate when they can generate longer answers at test time. This behavior is known as test-time scaling. However, RL with verifiable rewards (RLVR) usually trains them with a fixed rollout cap. We find that this cap changes model behavior, not only training cost. After training, models use few additional tokens beyond the training budget. They outperform the base models near that budget, but their accuracy stops improving and falls below the base model at larger budgets. The rollout cap thus becomes a learned constraint, that limits model's test-time scaling. We call this effect budget internalization. We observe it across Qwen3.5-4B, Gemma4-E2B, and Nemotron-3-Nano-4B and two reasoning training tasks: mathematical reasoning and diverse, procedurally generated problems. This effect also transfers from math training to science and coding tasks. Training-objective level ablations on Qwen3.5-4B show that most changes recover little test-time scaling. Stronger KL regularization and lower training-data truncation preserve more scaling, but stronger KL reduces in-budget accuracy. Forcing an answer on truncated training rollouts largely removes budget internalization, but also removes most of the in-budget gain. SFT on artificially truncated traces reproduces budget internalization, whereas SFT on naturally short traces does not, which shows that the effect is not specific to RL. This result links budget internalization to length-censored post-training rather than to an RL-specific objective term. Thus, a rollout cap does not only control training cost. It can trade low-budget gains for lower performance when additional inference compute is available.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aG3DOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies budget internalization: fixed-budget post-training improves accuracy near the rollout cap but can cause response length and accuracy to saturate when more test-time compute becomes available. The authors demonstrate this across three model families and multiple reasoning tasks, then use GRPO ablations and SFT controls to argue that the effect is primarily driven by length-censored training signals and truncation handling, rather than a particular GRPO objective term.

Critical strengths
  1. The phenomenon is evaluated broadly across several model families, training datasets, and out-of-domain benchmarks, rather than relying on a single run or task.
  2. The paper contains useful diagnostic controls, especially answer forcing, truncation-handling ablations, and the comparison between naturally short versus artificially truncated SFT traces.
Critical weaknesses
  1. Limited novelty/significance. The central observation—that training under a limited reasoning horizon can impair extrapolation to larger test-time budgets—is closely related to prior work on training-horizon and test-time-compute extrapolation. The proposed explanation is also fairly intuitive: when long or truncated trajectories are systematically disfavored, the policy specializes toward shorter solutions. The paper characterizes this effect extensively, but does not establish a substantially new mechanism or provide a method that resolves the trade-off.
  2. The interpretation of the in-budget gains is unclear. Improved accuracy near the training cap may primarily reflect short-horizon specialization—learning to finish and emit an answer within the censoring window—rather than an increase in the model’s underlying reasoning capability. The paper’s own overlength-penalty and artificially truncated SFT experiments produce the same qualitative pattern, which is consistent with this interpretation.
  3. The empirical evidence does not adequately separate training effects from run/selection variance. Each training condition uses only one training seed, while checkpoints are selected by validation accuracy at the training budget. This selection criterion can itself favor checkpoints specialized to the in-budget regime. I would like to see multi-seed replications of at least the main fixed-budget GRPO conditions, reporting the full accuracy/length curves for each seed, together with results from a fixed or pre-specified checkpoint (e.g., the final checkpoint) rather than only the checkpoint selected for in-budget validation accuracy. This would clarify whether the reported crossover is a stable property of training rather than partly a consequence of seed and checkpoint selection.
Questions for authors
  1. What is the key scientific novelty beyond existing observations that reasoning models often fail to extrapolate beyond their training-time reasoning horizon?
  2. How do the authors distinguish genuine reasoning improvement from short-horizon specialization that simply makes the model more likely to finish and answer within the training budget?
  3. Could the authors show whether the same budget-internalization curves persist across multiple training seeds or across checkpoints not selected specifically for validation accuracy at the training budget?
1 comment

TLDR: In our setting we start with models already operating at large reasoning budgets (128k) and further post-train them in a more affordable setting (8k). In that setup, we show that model stops using previous reasoning capabilities it had, this is also observed on OOD tasks. To our knowledge, this has not been previously shown in literature. At the same time, it has a far-reaching consequences for continual-learning research, where performing new task adaptation at an affordable research setting (8-16k) tokens of an already post-trainined model that reasons a lot (up to 128k) makes it harder to make any conclusions about positive/negative transfer, due to budget internalization phenomena we show. We’d like to highlight that developement of newer methods that allow for low-budget training while preserving test-time scaling, is an important new avenue for continual learning research as well.

Q1

We thank the reviewer for encouraging us to clarify our contribution and its relationship to prior work. Our work focuses on how further fixed-budget post-training can degrade the test-time scaling capabilities of an already capable reasoning checkpoint, even while improving its in-budget accuracy. Our motivation arose from studying catastrophic forgetting. When adapting an already capable reasoning model to an new task, we observed improvements on unrelated tasks when evaluated at a limited inference budget (e.g., 16k tokens). However, following evaluations at larger budgets revealed a large performance degradation relative to the starting checkpoint. Thus, a positive transfer at one evaluation budget can coexist with a large regression in previously demonstrated capabilities at another. While shorter generations following fixed-budget RLVR may be intuitive, the degradation of previously acquired reasoning capabilities does not necessarily follow. An alternative outcome would be reasoning compression, where the model achieves comparable or better accuracy using fewer tokens without sacrificing its high-budget performance. Distinguishing between these outcomes requires empirical investigation. Our contribution is to systematically characterize this phenomenon, which we term budget internalization, across three model families and multiple training tasks. In particular, we demonstrate that:

  • Further post-training can simultaneously improve in-budget accuracy and degrade previously acquired test-time scaling capabilities.
  • This degradation can extend beyond the training domain, making it relevant to capability retention during adaptation.
  • Budget internalization also occurs under SFT on artificially truncated traces, whereas naturally completed short traces preserve scaling.
  • Across the investigated training interventions, none fully preserves both the in-budget accuracy gains and the starting model’s high-budget performance.

These findings identify an evaluation problem for adaptation and continual learning research: improvements at a limited inference budget do not establish preservation of the starting model’s reasoning capabilities. Evaluations must also cover budgets at which those capabilities were previously demonstrated; otherwise, apparent positive transfer can conceal large high-budget losses. Additionally, such evaluation procedure helps to distinguish genuine capability gain from just learning to conclude reasoning on time.

Q2 We verify this via budget forcing experiments. In Figure 4, we show the gap between the base model being forced to emit an answer at a given budget and a model trained with a small budget of 4k tokens. We observe that 1) forcing the base model to produce an answer does increase its scores across various evaluation budgets and various benchmarks, so some of the performance gain is likely due to the model just being able to complete its answers, 2) however, we also see that on smaller evaluation budgets (4k-8k), the trained model actually outperforms this intervention. Therefore, there is a performance gain that comes on top of simply being able to answer within a given budget.

Q3

We have updated Figure 1 to show additional runs on three different seeds and added Tables 13–14 in Appendix with per seed accuracy and mean response lengths at all budgets for validation selected and final (step-300) checkpoints. Across all training runs, switching checkpoint choice (from validation selected to final) changes average accuracy by less than 2.3% at the training budget and 1.9% at 64k. All training runs retain in-budget gains and high-budget deficits relative to the base model under both checkpoint choices, supporting the robustness of budget internalization to seed and checkpoint selection.

In the updated PDF we highlighted the updates with yellow.

Reviewer_ZTvZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies performance scaling for models that undergo RLVR training with different token budgets. The paper argues that models after RLVR training tend to use few additional tokens beyond the training budget. Despite these models outperform the base models near this budget, their accuracy stops improving and falls below the base model at larger budgets. The authors argue this constraint limits the model’s test-time scaling, which they term "budget internalization." The paper witnesses this effect in experiments on Qwen3.5-4B, Gemma4-E2B, and Nemotron-3-Nano-4B and two reasoning training tasks. The paper further conducts ablation studies on factors such as stronger KL regularization or lower training-data truncation.

Critical strengths

I did not identify substantial strengths in this manuscript.

Critical weaknesses

This is most likely a 100% AI-generated manuscript. The style of writing shows the characteristic style of AI: numerics and performance deltas are extensive throughout the text in the paper (relative performance changes such as +37.9 pp; 64K: −22.5 pp, appear 100 times in the manuscript), monotonic presentation and figures, and the loose layout.

The main argument--RLVR optimizes performance for the token budget used in training while degrading performance under much larger token budgets-- appears trivial.

The models used in the experiments, Qwen3.5-4B, Gemma4-E2B, and Nemotron-3-Nano-4B, have been optimized for very long token budgets in math tasks (e.g., 128k tokens). When undergoing RLVR targeting a much smaller target budget (e.g., 1-8k as studied in the paper), longer responses will be penalized for not finishing within the token budget. Thus, the RL-trained model will no longer try to produce such responses, even when the token budgets increase. This appears to be a very natural result, and this manuscript seems not to contribute beyond that.

Questions for authors

How do the authors view this manuscript's contribution to the research community?

1 comment

We strongly disagree with the reviewer’s statement that our manuscript is “100% AI-generated” and that our work involved “little or no human effort.”

We consider this a serious and unsubstantiated allegation, particularly in the context of an ethics review flag. The frequency of numerical results, writing style, figure presentation, and manuscript layout do not, in our view, constitute evidence supporting such a serious accusation.

For transparency, we have compiled a selection of anonymized, timestamped artifacts documenting the development of this work:

  • Internal research discussions from July, documenting the initial observations, preliminary experimental results, and formulation of the research question.
  • The original research proposal, including competing hypotheses, a discussion of related literature, experimental design, and subsequent discussions among the authors.
  • Overleaf revision history from September, documenting the iterative development and revision of the manuscript.

The supporting materials are available at the following anonymous link https://docs.google.com/document/u/1/d/e/2PACX-1vSLmnpC8oEdS5JNEOjxOkFmOTyYcPPK8xMGP9cBlLlMnzpm48HF31baOYs5M3nlAi2bs2BBuAfsmguX/pub.

These artifacts document months of research, scientific discussion, experimentation, and manuscript preparation by the authors. Identifying information has been redacted to preserve double-blind review.

We very much welcome and appreciate constructive criticism regarding the novelty, significance, methodology, presentation, or any other aspect of our work. However, we reject the reviewer’s allegations and expect that allegations concerning our research integrity be supported by evidence rather than subjective stylistic impressions.

In light of the provided documentation, we kindly ask the reviewer to reconsider their allegation and the associated ethics review flag.

To the Area Chair: Given the seriousness of the allegation and its potential influence on the overall assessment of our submission, we respectfully ask the Area Chair to independently assess the grounds for the allegation in light of the provided evidence and to ensure that unsubstantiated claims concerning our research integrity do not prejudice the evaluation of our work on its scientific merits.

Reviewer_1mSvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how fixed-budget RLVR changes the test-time scaling behavior of reasoning models. Training improves accuracy within the rollout budget but often produces policies that stop early and cease benefiting from larger inference budgets, eventually falling behind their base models. The authors call this behavior “budget internalization.” Experiments examine its dependence on training data, truncation handling, loss normalization, KL regularization, and supervised fine-tuning. The paper argues that length-censored training signals, rather than RL alone, can induce this behavior.

Critical strengths
  1. The paper identifies a practically relevant evaluation issue: improvements measured at the training budget can conceal substantial degradation at larger inference budgets. Reporting both accuracy and actual generation length makes this trade-off much clearer than a single-budget comparison would.

  2. The results distinguish between tasks where shorter reasoning largely preserves accuracy and tasks where it sacrifices useful test-time scaling. In particular, the different outcomes on AIME versus GPQA and LiveCodeBench make the empirical story more informative than a blanket claim that shorter reasoning is harmful.

  3. The interventions address several plausible explanations, including length normalization, truncation penalties, KL regularization, and forced answering. The supervised fine-tuning comparison is also a useful direction, since it asks whether the phenomenon extends beyond policy-gradient training.

Critical weaknesses
  1. The RFT comparison does not isolate the effect of length censoring.

Section 3 compares successful trajectories that naturally finish within 4k tokens with trajectories that originally exceed 4k, whose unfinished reasoning is truncated and followed by a correct answer. These conditions differ in trajectory selection and in whether the retained reasoning supports the appended conclusion. They therefore do not “differ only” in whether reasoning ends naturally. The paper also does not establish that the prompts and training exposure are matched. This experiment shows that the particular SFT construction can reproduce budget internalization, but does not isolate censoring as its cause. A comparison using matched prompts and complete versus truncated versions of successful trajectories, with sample counts and training budgets reported, would strengthen this conclusion.

  1. The answer-forcing token counts need to be reconciled with the stated procedure.

Section 2.3 and Appendix C.1 describe retaining a reasoning prefix that has reached the budget and then generating an additional answer. However, at the 4k AIME budget, Table 7 reports 100% truncation and a mean length of 4.00k for ordinary base decoding, versus only 0.72k with answer forcing. The already-generated prefix should still count toward inference cost. Please clarify which tokens these measurements include, report cumulative generation across both stages, and specify whether the additional answer window is included in the stated budget. This discrepancy affects the efficiency interpretation of answer forcing, although it does not by itself invalidate the reported accuracy improvement.

Questions for authors

See Critical Weaknesses above.

1 comment

Q1

The budget-forced trajectories are collected by truncating the reasoning at 4096 tokens, asking the model for its final answer with the budget-forcing mechanism from Section 2.3, and keeping only the trajectories where that answer matches the ground truth. Our description of this was unclear, and we hope this clarification addresses the weakness, and we have updated the relevant wording in the paper. Regarding the dataset composition, both RFT setups use the same prompt template and problems from the same DeepScaleR pool. The 'Short traces' and 'Budget-forced traces' datasets have 1044 and 13012 datapoints respectively, and we use the same training hyperparameters. The budget-forced run did 386 optimization steps and the short traces run did 90, so we reran the budget-forced training and evaluation for 90 steps (batch size 32) to match. The results below extend Table 3. The budget internalization effect is smaller at the matched training budget (64k: −38.4 vs base, down from −52.2), but it still shows the strongest 64k losses. The paper's conclusions remain unchanged. We have updated the submission and thank the reviewer for their constructive criticism.

AblationVariant2k4k8k (train)16k32k64k
Base model—0.0±0.00.3±0.69.2±6.234.2±10.865.4±10.178.8±8.5
RFTShort traces (90 steps)0.0±0.00.3±0.67.8±4.830.3±10.563.5±10.583.0±7.9
RFTOLD: Budget-forced traces (386 steps)0.0±0.05.7±3.525.1±9.225.5±9.126.4±9.026.6±9.0
RFTNEW: Budget-forced traces (90 steps)0.0±0.07.1±3.118.5±7.326.1±9.237.7±8.940.4±8.5

Q2

Thank you for highlighting this inconsistency. The previous length calculation omitted the retained reasoning prefix. We have corrected this issue and updated Table 7 and Figure 4 to compare budget forcing with ordinary decoding at the same total generated-token allowance, including the answer window. There was also a bug, where we have allowed longer generation than we report in the paper. We have fixed this by rescoring the corresponding trajectories to the reported generation cap: 120 tokens for AIME, 16 for GPQA, and 1,024 for LiveCodeBench. Appendix C.1 has more details on this procedure. Scores for Qwen are revised: 4k AIME mean is 4.18k tokens, correcting the reported 0.72k. At matched total generation budgets, its accuracy gains remain +8.9 (AIME), +56.2 (GPQA), and +14.5 (LCB) percentage points. We have updated the appendix figures and tables as well.

FYI: In the updated PDF we highlighted the updates with yellow.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.