acceptodds
Under review as a conference paper at ICLR 2027

What Do Rollout Tokens Buy? Hidden Costs and Budgeted Accuracy in RLVR

Abstract

Reinforcement learning with verifiable rewards (RLVR) pays for every token it generates, yet training choices are rarely judged by what those tokens buy. We study this question on competition mathematics with two training choices and one evaluation choice. First, we replace one third of a SmolLM3-3B training set with harder problems and keep the other 2,048 problems at the same positions. Responses to these unchanged problems become twice as long (+101%) at a similar training reward, and late in training most of their tokens are in responses that reach the length limit. Second, we halve the training response limit of Qwen3.5-9B from 4,096 to 2,048 tokens. Training compute falls by 28%, and majority-vote accuracy under a 16,384-token budget rises from 36.1% to 43.5% as far more evaluation answers finish. The gain remains when truncated answers are continued to 8,192 tokens. Third, under the same budget, counting problems with any correct answer and counting the answer a model returns can rank a base model and its RL-trained version in opposite orders. The SmolLM3-3B base finds a correct answer on more of 216 problems (57 against 39) but returns a correct majority answer on fewer (5 against 18), and majority vote favors the trained model in all three model pairs. In all three cases, more tokens did not mean more correct answers. We therefore suggest judging an RLVR training choice by the tokens it adds on problems that stay the same and by the answers the trained model returns under a fixed token budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning with verifiable rewards, mathematical reasoning, token budget, training data difficulty

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_By3tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the relationship between rollout token consumption, training choices, and inference accuracy in reinforcement learning with verifiable rewards. The authors conduct three experiments: replacing part of a training dataset with harder problems while keeping other problems unchanged, reducing the maximum training response length, and comparing evaluation metrics under a fixed inference token budget. The results show that harder training data increases response lengths on unchanged problems, while reducing the training limit from 4,096 to 2,048 tokens lowers estimated training compute by 28% and improves budgeted majority-vote accuracy by 7.4 percentage points. The paper also shows that budgeted coverage and majority-vote accuracy can rank models differently.

Critical strengths
  1. The main training results rely on limited experimental configurations. The difficulty experiment uses SmolLM3-3B, while the response-length experiment uses Qwen3.5-9B with only two length limits. No independent training repetitions are reported, making it difficult to determine the stability of the observed improvements across training configurations.
  2. The additional insight beyond prior studies is relatively limited. Response-length biases, length-constrained training, and budget-aware evaluation have already been investigated. Moreover, the harder-data experiment shows a trade-off between majority-vote accuracy and budgeted coverage, rather than a consistent lack of benefit from additional tokens. The evidence therefore supports specific observations more strongly than a general conclusion about the value of rollout length.
Critical weaknesses
  1. The methodological contribution appears incremental relative to existing value-guided decoding approaches. PITA primarily changes the supervision used to learn the guidance signal while retaining a similar inference-time guidance framework. In addition, the theoretical guarantees concern the exact preference-guided policy, whereas the implemented algorithm uses an approximation whose error is not analyzed.
  2. The empirical results do not establish convincing generalization. Although PITA achieves strong gains on GSM8K and synthetic reasoning tasks, improvements on MATH-500 and AlpacaEval are much smaller. Its performance is also sensitive to the amount of preference data, leaving uncertainty about its effectiveness across different tasks and preference distributions.
Questions for authors
  1. Are the improvements from the shorter training response limit consistent across independent training runs, additional response limits, or longer training schedules?
  2. Given that the harder-data experiment improves majority-vote accuracy on the 500 held-out problems but reduces budgeted coverage, how should the cost-benefit trade-off be interpreted? Can the authors clarify when additional rollout tokens are beneficial rather than wasteful?
  3. The reference to Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning on page 10 still uses its 2025 preprint information and retains the arXiv comment about EACL 2026, although the paper has been published at EACL 2026 (pp. 700–719). The reference should be updated accordingly. In addition, Figure 3 on page 7 contains small labels, and page 9 has substantial unused space after the conclusion. These formatting issues could be improved.
Reviewer_cZbmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the relationship between rollout token consumption and performance in RLVR. Through three empirical studies, the authors show that harder training problems can increase response lengths on unchanged problems, shorter training limits can reduce compute while improving accuracy, and different evaluation metrics can lead to opposite model rankings under a fixed token budget.

Critical strengths
  1. The paper is clearly written and easy to follow.
  2. The controlled experiments provide useful empirical observations.
Critical weaknesses
  1. Limited novelty. Several key observations have already been explored in prior work. Bounhar et al. (2025)[1] studied how training data difficulty affects response length, while ThinkPrune (Hou et al., 2026) [2]investigated reducing reasoning length through tighter training limits. More importantly, Yue et al. (2025) [3]already demonstrated that base models can outperform RL-trained models at large sampling budgets, while RL improves sampling efficiency at smaller values of k. Although this paper provides additional measurements under fixed token budgets, the incremental insights appear limited.

  2. Lack of a coherent research question and deeper analysis. The three findings appear relatively disconnected, with different experiments conducted on different models.

[1] Shorter but Not Worse: Frugal Reasoning via Easy Samples as Length Regularizers in Math RLVR [2] ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning [3] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

Questions for authors
  1. Can the authors provide a more systematic investigation connecting the three findings, such as testing whether shorter training limits mitigate the response-length inflation caused by harder training data, and analyzing the underlying mechanism?
Reviewer_aQYQOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors present three experiments that outline how RLVR on more tokens can make models perform worse: First they show that training on harder problems will overall increase token usage. Second, they show that a lower rollout limit during training will cause the model to produce shorter answers. Third, they present results showing that RLd models score relatively higher on a majority vote setting compared to a best-of-n setting when comparing to the base model.

Critical strengths

The first and second experiment and the general theme are interesting. RLVR is being widely applied now so the question of whether we can reduce its cost and how to control the cost of the resulting model.

Critical weaknesses
  • Overall the evaluations are only on one dataset (AIME variants). The sections 4 and 5 and corresponding experiments are only on one model (Section 4 SmolLM3 3B, Section 5 Qwen3.5 9B). This leaves the question whether the observations generalize open.
  • For Section 6 I don't really understand what the key point of the authors is. "Budgeted coverage" appears to be a pass@n metric and it seems that the base model is better than the RL model in pass@n, but worse in a majority vote. It is not clear why this should be suprising, since it is well known that RL reduces model variance [1,2] (thus this result is expected).
  • A lot of wording and oddities are strong signs for heavy LLM assistance and make reading and understanding unnecessarily difficult.

[1] https://arxiv.org/abs/2504.13837
[2] https://arxiv.org/abs/2505.22617
[3] https://arxiv.org/abs/2310.06452

Questions for authors
  • Do the findings of Section 4 and 5 generalize to other datasets and models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.