- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the relationship between rollout token consumption, training choices, and inference accuracy in reinforcement learning with verifiable rewards. The authors conduct three experiments: replacing part of a training dataset with harder problems while keeping other problems unchanged, reducing the maximum training response length, and comparing evaluation metrics under a fixed inference token budget. The results show that harder training data increases response lengths on unchanged problems, while reducing the training limit from 4,096 to 2,048 tokens lowers estimated training compute by 28% and improves budgeted majority-vote accuracy by 7.4 percentage points. The paper also shows that budgeted coverage and majority-vote accuracy can rank models differently.
Critical strengths
- The main training results rely on limited experimental configurations. The difficulty experiment uses SmolLM3-3B, while the response-length experiment uses Qwen3.5-9B with only two length limits. No independent training repetitions are reported, making it difficult to determine the stability of the observed improvements across training configurations.
- The additional insight beyond prior studies is relatively limited. Response-length biases, length-constrained training, and budget-aware evaluation have already been investigated. Moreover, the harder-data experiment shows a trade-off between majority-vote accuracy and budgeted coverage, rather than a consistent lack of benefit from additional tokens. The evidence therefore supports specific observations more strongly than a general conclusion about the value of rollout length.
Critical weaknesses
- The methodological contribution appears incremental relative to existing value-guided decoding approaches. PITA primarily changes the supervision used to learn the guidance signal while retaining a similar inference-time guidance framework. In addition, the theoretical guarantees concern the exact preference-guided policy, whereas the implemented algorithm uses an approximation whose error is not analyzed.
- The empirical results do not establish convincing generalization. Although PITA achieves strong gains on GSM8K and synthetic reasoning tasks, improvements on MATH-500 and AlpacaEval are much smaller. Its performance is also sensitive to the amount of preference data, leaving uncertainty about its effectiveness across different tasks and preference distributions.
Questions for authors
- Are the improvements from the shorter training response limit consistent across independent training runs, additional response limits, or longer training schedules?
- Given that the harder-data experiment improves majority-vote accuracy on the 500 held-out problems but reduces budgeted coverage, how should the cost-benefit trade-off be interpreted? Can the authors clarify when additional rollout tokens are beneficial rather than wasteful?
- The reference to Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning on page 10 still uses its 2025 preprint information and retains the arXiv comment about EACL 2026, although the paper has been published at EACL 2026 (pp. 700–719). The reference should be updated accordingly. In addition, Figure 3 on page 7 contains small labels, and page 9 has substantial unused space after the conclusion. These formatting issues could be improved.