acceptodds
Withdrawn from ICLR 2027

Fixed-Budget Accounting Can Mislead Reinforcement Learning for Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has made reasoning performance increasingly depend on how compute is allocated. At a fixed budget, should training spend compute on more optimizer updates, more parallel rollouts, or deeper refinement passes? This question is easy to state but hard to answer because the planned token budget may not match the number of tokens actually generated. We study this problem for Low-Rank Adaptation (LoRA) with Group Relative Policy Optimization (GRPO) under a strict fixed-budget audit across several LLMs. Our audit compares the three allocation axes under scheduled-token accounting, instruments realized-token usage, and then tests any accounting-driven reversal with direct realized-budget reruns. This exposes an accounting inversion: refinement can look stronger after realized-token normalization because its rollouts often terminate early, while the controlled scheduled-budget comparison favors spending compute on optimizer updates. The direct reruns resolve the ambiguity: the optimizer-update allocation still achieves performance across the evaluated benchmarks while using fewer generated tokens. We also run a matched inference-time audit comparing refinement depth with parallel sampling on reasoning benchmarks. Depth helps in some settings, but loses or ties in others, and out-of-distribution (OOD) evaluations show no consistent depth advantage. Our results turn a potentially misleading fixed-budget comparison into a reporting rule: state the budget being fixed, treat realized-token normalizations as diagnostics, and verify apparent reversals with direct reruns before making training claims.

Primary area:
reinforcement learning
Keywords:
LLM, RL, RLVR, reasoning models, compute allocation, token accounting, fixed-budget evaluation, test-time compute, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_weafOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates compute allocation issue in RLVR. Three axes are considered: optimizer updates, parallel rollouts and deeper refinement passes. Three arms are compared under two metrics: accuracy given fixed scheduled token and accuracy rescaled by actual token usage. The ranking is inversed under the two metrics. A rerun experiment is done to match actual token usage and optimizer updates achieves the best performance.

Critical strengths

This is an investigation paper that studies a useful topic: how to allocate compute in RLVR? The three axes considered are practical hyperparameters people tune in practice.

Critical weaknesses
  1. The conclusion of the paper is not novel. The result of the rerun experiment supports using fixed token budget as the accounting, which is a common practice. As an investigation paper, it is supposed to propose a viewpoint that is unfamiliar within the community. Besides, the motivation of doing the rerun experiment is the inversed ranking under realized-token diagnostic, which reports the ratio of accuracy and actual token consumption. However, the ratio itself is a coined metric, so the ranking out of it is not convincing.
  2. The experiment design is not rigorous and not sound. The rerun experiment does not strictly match the actual token consumption. More importantly, the paper only runs the experiment under a single token budget (3,276,800) and three allocation arms. As an investigation paper, it should consider multiple token budgets and run multiple allocation arms, so that the conclusion of favoring optimizer updates is universal.
Questions for authors

I would like the authors to address the two weakness points above, though I believe it is hard to respond in rebuttal.

Reviewer_8Pu2OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The work investigate the training strategies under fixed budget measured with Tokens. It analyzed three axes and the order of training, then compared the performance on MATH500. The authors concluded with a rule: state the budget being fixed,treat realized-token normalizations as diagnostics, and verify apparent reversals with direct reruns before making training claims.

Critical strengths

Clearly presented the methodology and the design of the research. The comparisons are thorough and the results showing unambiguious trend.

Critical weaknesses

The comparison results do not provide guidance for practical applications directly, rather, the authors made suggestions on how assess the performance under fixed budget with realized tokens. It is helpful to some extend.

Questions for authors
  1. Whether the results are unchanged for different models?
  2. The same or similar trends hold for different testing sets?
Reviewer_roTiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how the accounting unit changes a comparison of RLVR training allocations. Using LoRA-GRPO on three DeepSeek-R1-Distill models, it allocates a fixed scheduled response-token budget to more updates, more rollouts, or more refinement passes. More updates win most of the MATH500 comparisons. Dividing accuracy by the realized token fraction reverses much of that ranking, but direct Ashort/Blong reruns reportedly favor the update allocation while using fewer response tokens. The paper also compares refinement and parallel sampling at inference time, and tests transfer to GPQA-Diamond and MMLU-Pro. Its main recommendation is to report both scheduled and realized usage and check apparent accounting reversals with actual training runs.

Critical strengths
  1. The direct reruns are the most useful part of the paper. An accuracy divided by a token fraction does not tell us what a different training schedule will achieve, and the authors follow up with an actual comparison. Table 1 reports MATH500 gains of 2.7–6.7 percentage points under greedy evaluation while Ashort uses only 0.83–0.85 times Blong's response tokens. If these runs differ in the intended way, that is meaningful evidence for the update allocation in this setting. Exact equality of token use is not necessary when one configuration is both cheaper and more accurate.

  2. The treatment of repeated evaluation samples is reasonably careful. The inference and OOD analyses resample problems while keeping their sampling seeds together, and the appendix shows cases where treating every problem-seed pair as independent would change the conclusion. The paper also distinguishes a bounded equivalence result from a nonsignificant difference. These choices make the reported uncertainty more useful than simply counting winning cells.

Critical weaknesses
  1. The central rerun experiment is not specified well enough to assess the intervention. Figure 1 gives the original A/B/C tuples, but Sections 3–4 and Appendix C.3 do not give the corresponding (M,Bp,n,d,maxnew)(M,B_p,n,d,\mathrm{max_new}) values for Ashort and Blong, the rule for choosing their budgets or stopping points, or their absolute token totals and accuracies. Table 1 supplies only differences and token ratios. Its nine and eight matched training seeds for 7B and L8B are also not accounted for by the training-seed lists in Appendices A and J. This may be an incomplete description rather than an experimental error, but these reruns are the paper's main evidence for resolving the accounting inversion. I cannot tell from the manuscript exactly what was changed and matched. A per-run configuration and results table would address much of this concern.

  2. The inference comparison does not define how parallel samples become a single reported answer. Section 4 compares (d,n)=(4,1)(d,n)=(4,1) with (1,4)(1,4), and similarly at 8×, using a “pass@1 difference.” Appendix A gives the decoding settings but does not explain how the four or eight independent samples are reduced to the scored outcome. If pass@1 is the average correctness of those samples, increasing their number does not improve expected single-answer accuracy; it only estimates it more precisely. If the experiment uses majority voting, a verifier, or another selector, that rule needs to be specified. These possibilities measure different uses of inference budget, so the reported depth-versus-sampling crossover is difficult to interpret without this detail. I am not assuming that the implementation uses the first interpretation.

Questions for authors
  1. Please provide the configurations and budget-selection or stopping rule for Ashort/Blong, along with each matched training seed's response-token total and absolute MATH500 results. How do the nine and eight matched seeds in Table 1 map to the seed lists in Appendices A and J? This would determine whether the main rerun result can be interpreted as the intended allocation comparison.

  2. For the inference experiment, what exact operation maps the nn parallel completions to the pass@1 outcome? Please specify answer selection, any tie handling, and whether selection uses gold correctness. Does “matched compute” here mean the scheduled response-token allowance or measured generated tokens, and what are the realized totals for the compared configurations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.