- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigates compute allocation issue in RLVR. Three axes are considered: optimizer updates, parallel rollouts and deeper refinement passes. Three arms are compared under two metrics: accuracy given fixed scheduled token and accuracy rescaled by actual token usage. The ranking is inversed under the two metrics. A rerun experiment is done to match actual token usage and optimizer updates achieves the best performance.
Critical strengths
This is an investigation paper that studies a useful topic: how to allocate compute in RLVR? The three axes considered are practical hyperparameters people tune in practice.
Critical weaknesses
- The conclusion of the paper is not novel. The result of the rerun experiment supports using fixed token budget as the accounting, which is a common practice. As an investigation paper, it is supposed to propose a viewpoint that is unfamiliar within the community. Besides, the motivation of doing the rerun experiment is the inversed ranking under realized-token diagnostic, which reports the ratio of accuracy and actual token consumption. However, the ratio itself is a coined metric, so the ranking out of it is not convincing.
- The experiment design is not rigorous and not sound. The rerun experiment does not strictly match the actual token consumption. More importantly, the paper only runs the experiment under a single token budget (3,276,800) and three allocation arms. As an investigation paper, it should consider multiple token budgets and run multiple allocation arms, so that the conclusion of favoring optimizer updates is universal.
Questions for authors
I would like the authors to address the two weakness points above, though I believe it is hard to respond in rebuttal.