- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper looks at RLVR rollouts cut off by the generation cap. For each group, the method builds a cheap update from the truncated prefixes with imputed rewards, then continues a random subset of groups to the full horizon and uses them to correct that update without bias. Which groups to continue is set by a square root rule on predicted residual risk per unit cost, and the risk model is fit with inverse probability weighting over all groups. The paper gives supporting theory and reports gains over several allocation baselines with two Qwen3.6 models on six benchmarks.
Critical strengths
I like the idea of treating cap censoring as two-phase sampling. Since the imputed update is only used as a control variate, the corrected update is unbiased no matter how poor the imputation head is, and a bad head simply wastes budget. The allocation then has a closed form, so the method should be easy to add to an existing training pipeline.
The paper also points out something adaptive allocators usually miss. If the risk model is fit only on the continued groups while continuation probabilities vary within a bin, the fit is biased, and weighting all arrivals fixes this at almost no cost.
Critical weaknesses
My main concern is that the gains do not seem to come from the mechanism the paper describes. The argument is that better allocation lowers the correction error and that this is what improves accuracy, but in the paper's own results the two do not move together. The marginal variant fit only on selected groups barely reduces the error compared with fixed allocation, yet it gains more accuracy than Uncertainty or PAIR, which reduce the error far more. So the accuracy ranking tells us little about whether joint residuals or IPW are doing any work.
I also find the size of the gain hard to believe. Fixed allocation improves on the released checkpoint by less than a point, while the joint method adds about seven, including large jumps on HLE and on long context tasks that are not in the training data. Both arms are unbiased and differ only in which groups receive a tenth of the budget, over roughly 45 small LoRA steps. The starting checkpoint is never evaluated under the paper's own protocol, so I cannot tell how much the model actually improves.
Some of the reported statistics do not fit the stated protocol. With 33 HMMT problems and one generation per problem, every HMMT seed standard deviation in the main table is smaller than the smallest nonzero value this setup can produce. In the error ablation, the joint method's numbers are exact whole percentage reductions of round baseline values. I do not know how these numbers were obtained, and until that is clear I am reluctant to rely on the tables.
The theory adds less than claimed. Correcting a cheap prediction on a random subset, with square root allocation by risk over cost, is close to the active statistical inference of Zrnic and Candès (2024), and PAIR (Nomand et al., 2026) does something similar for rollouts. The selective feedback result is the familiar bias of complete-case averaging. The coupling theorem needs equal marginal risks with different cross terms, but the implementation bins only by task type and censoring count, and the two groups in the paper's example share a bin. The joint learner would then allocate like the marginal rule.
References
Kimi Team. Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv preprint arXiv:2501.12599, 2025.
Nomand, P., Voss, E., Hale, M., and Reyes, S. PAIR: Pairwise-Aware Inclusion Reweighting for Adaptive Rollout Allocation in RLVR. arXiv preprint arXiv:2608.11368, 2026.
Zhou, Y., Li, J., Su, Y., et al. APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generation. arXiv preprint arXiv:2509.18521, 2025.
Zrnic, T. and Candès, E. J. Active Statistical Inference. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235 of PMLR, 2024.
Questions for authors
Clarifications:
- How is HMMT scored, and could you share per-seed results for the main table and the error ablation?
- How does the starting checkpoint score under your evaluation protocol?
- Why does the marginal variant fit on selected groups gain more accuracy than Uncertainty and PAIR when it reduces the correction error much less?
- What is the correction error with no correction at all (Native C) at the shared checkpoint? With continuation probabilities around one half, the added variance looks about as large as that error.
Suggestion: A simpler alternative is to resume unfinished rollouts in a later step, as Kimi k1.5 (Kimi Team, 2025) and APRIL (Zhou et al., 2025) do. I would like to see the paper discuss it, along with the connection to optimal Poisson sampling.