acceptodds
Under review as a conference paper at ICLR 2027

LEARNING WHERE TO CONTINUE: BUDGETED CORRECTION OF CENSORED POLICY GRADIENTS

Abstract

In language-model training, reinforcement learning with verifiable rewards improves reasoning through outcome-based feedback. Generation limits leave some trajectories unfinished, while completing every rollout is costly. Imputation provides low-cost updates but leaves open where further generation most reduces gradient estimation error. We propose a budgeted continuation framework that selects rollout groups using joint prefix–tail residual risk and corrects their imputed updates using selected completions. Inverse-probability weighting over all arriving groups accounts for selective feedback when learning this risk. We establish unbiasedness for the unclipped, finite-horizon gradient of a frozen policy, derive the optimal allocation under an expected budget constraint, and bound the excess risk of its learned counterpart. An exact autoregressive construction shows that prefix–tail coupling can make allocation based on segment-wise second moments strictly suboptimal, even with a perfectly calibrated imputation head. Across six benchmarks in mathematics, coding, knowledge and long-context reasoning, joint-residual allocation increases mean accuracy by 6.27 and 6.37 percentage points on Qwen3.6-27B and Qwen3.6-35B-A3B, respectively, and reduces gradient correction error by 24.67% and 28.02% relative to fixed continuation at the same training budget. In ablations, accounting for prefix–tail coupling and correcting selective feedback each yields lower correction error and higher mean accuracy on both dense and mixture-of-experts models.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning with Verifiable Rewards, Large Language Model Reasoning, Policy Gradient Estimation, Right Censoring, Adaptive Budget Allocation, Inverse Probability Weighting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_x4PTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper looks at RLVR rollouts cut off by the generation cap. For each group, the method builds a cheap update from the truncated prefixes with imputed rewards, then continues a random subset of groups to the full horizon and uses them to correct that update without bias. Which groups to continue is set by a square root rule on predicted residual risk per unit cost, and the risk model is fit with inverse probability weighting over all groups. The paper gives supporting theory and reports gains over several allocation baselines with two Qwen3.6 models on six benchmarks.

Critical strengths

I like the idea of treating cap censoring as two-phase sampling. Since the imputed update is only used as a control variate, the corrected update is unbiased no matter how poor the imputation head is, and a bad head simply wastes budget. The allocation then has a closed form, so the method should be easy to add to an existing training pipeline.

The paper also points out something adaptive allocators usually miss. If the risk model is fit only on the continued groups while continuation probabilities vary within a bin, the fit is biased, and weighting all arrivals fixes this at almost no cost.

Critical weaknesses

My main concern is that the gains do not seem to come from the mechanism the paper describes. The argument is that better allocation lowers the correction error and that this is what improves accuracy, but in the paper's own results the two do not move together. The marginal variant fit only on selected groups barely reduces the error compared with fixed allocation, yet it gains more accuracy than Uncertainty or PAIR, which reduce the error far more. So the accuracy ranking tells us little about whether joint residuals or IPW are doing any work.

I also find the size of the gain hard to believe. Fixed allocation improves on the released checkpoint by less than a point, while the joint method adds about seven, including large jumps on HLE and on long context tasks that are not in the training data. Both arms are unbiased and differ only in which groups receive a tenth of the budget, over roughly 45 small LoRA steps. The starting checkpoint is never evaluated under the paper's own protocol, so I cannot tell how much the model actually improves.

Some of the reported statistics do not fit the stated protocol. With 33 HMMT problems and one generation per problem, every HMMT seed standard deviation in the main table is smaller than the smallest nonzero value this setup can produce. In the error ablation, the joint method's numbers are exact whole percentage reductions of round baseline values. I do not know how these numbers were obtained, and until that is clear I am reluctant to rely on the tables.

The theory adds less than claimed. Correcting a cheap prediction on a random subset, with square root allocation by risk over cost, is close to the active statistical inference of Zrnic and Candès (2024), and PAIR (Nomand et al., 2026) does something similar for rollouts. The selective feedback result is the familiar bias of complete-case averaging. The coupling theorem needs equal marginal risks with different cross terms, but the implementation bins only by task type and censoring count, and the two groups in the paper's example share a bin. The joint learner would then allocate like the marginal rule.

References

Kimi Team. Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv preprint arXiv:2501.12599, 2025.

Nomand, P., Voss, E., Hale, M., and Reyes, S. PAIR: Pairwise-Aware Inclusion Reweighting for Adaptive Rollout Allocation in RLVR. arXiv preprint arXiv:2608.11368, 2026.

Zhou, Y., Li, J., Su, Y., et al. APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generation. arXiv preprint arXiv:2509.18521, 2025.

Zrnic, T. and Candès, E. J. Active Statistical Inference. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235 of PMLR, 2024.

Questions for authors

Clarifications:

  1. How is HMMT scored, and could you share per-seed results for the main table and the error ablation?
  2. How does the starting checkpoint score under your evaluation protocol?
  3. Why does the marginal variant fit on selected groups gain more accuracy than Uncertainty and PAIR when it reduces the correction error much less?
  4. What is the correction error with no correction at all (Native C) at the shared checkpoint? With continuation probabilities around one half, the added variance looks about as large as that error.

Suggestion: A simpler alternative is to resume unfinished rollouts in a later step, as Kimi k1.5 (Kimi Team, 2025) and APRIL (Zhou et al., 2025) do. I would like to see the paper discuss it, along with the connection to optimal Poisson sampling.

Reviewer_EjkuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the missing terminal rewards and biased gradient estimates caused by generation truncation (C < H) in reinforcement learning for large language models. The authors propose a framework based on selective continuation and gradient correction. Specifically, an imputation head is used to estimate the rewards of incomplete trajectories. The joint prefix–tail gradient residual risk is then used to determine which rollout groups should be continued to H, and inverse-probability weighting (IPW) is applied to correct the gradient estimates. The authors also consider the feedback bias introduced by selective continuation and design a corresponding risk estimation method. Theoretically, the paper proves that the corrected gradient estimator is unbiased under a frozen policy, an unclipped objective, and a finite-horizon setting. It also derives the optimal continuation allocation strategy under a budget constraint and analyzes the importance of coupling between prefix and tail residuals. Experiments are conducted on two models with different architectures and six reasoning benchmarks. The results demonstrate that the proposed method can improve task accuracy while reducing gradient correction error.

Critical strengths
  1. The proposed method is supported by a solid theoretical foundation. The authors cleverly introduce ideas from probability and statistics for handling censored data and missing feedback into reinforcement learning, providing a targeted solution to the missing rewards and biased gradient estimates caused by generation truncation. One particularly commendable aspect is that the authors not only prove the unbiasedness of the corrected gradient estimator under specific conditions, but also derive the optimal continuation allocation strategy under a budget constraint. Furthermore, their theoretical analysis reveals the relationship between prefix–tail residual coupling and its influence on the allocation strategy. Overall, the theoretical analysis is quite comprehensive, and the method is supported by a clear mathematical foundation rather than relying solely on heuristic sampling strategies. The experiments also demonstrate relatively substantial performance improvements. These aspects make the paper methodologically innovative to some extent and provide a theoretically grounded approach to addressing gradient estimation problems caused by generation truncation in LLM reinforcement learning.

  2. I think the experimental design is relatively rigorous, with a diverse and convincing set of baselines. The authors validate their method on both Dense and MoE architectures, covering a variety of benchmark tasks. The comparison methods are relatively comprehensive, including several conventional continuation strategies and related methods such as PAIR, VIP, and VIGOR. The experiments are conducted under a unified training budget. The results show that, compared with fixed-probability continuation, the proposed method improves the average accuracy of both models. In addition, the paper includes ablation studies, computational cost evaluations, and other analyses. These experiments not only demonstrate the effectiveness of the method across different models and tasks, but also help clarify the sources of the performance improvements, thereby providing relatively strong support for the paper's main conclusions.

Critical weaknesses
  1. The risk estimation approach used in practice may be overly coarse-grained, making it difficult to adequately distinguish differences between rollout groups. The theoretical analysis shows that the optimal continuation probability depends on the conditional joint residual risk of each rollout group. However, in the actual implementation, the authors partition rollout groups into bins according to task type and the number of censored rollouts, and estimate the risk using the historical average residual risk within each bin. Even for rollout groups belonging to the same task and containing the same number of censored trajectories, their actual residual risks may differ substantially. Consequently, the current method may struggle to identify which rollout groups would benefit most from further continuation. Moreover, one of the paper's core theoretical contributions is its emphasis on the coupling between prefix and tail residuals, but such coarse-grained grouping may not fully exploit the information contained in different prefixes. Although the existing experiments demonstrate that joint residual risk estimation outperforms separate segment-wise residual estimation, it remains unclear how much valuable information is lost through this bin-level estimation approach and to what extent this information loss affects the optimal allocation of the continuation budget.

  2. The practical advantages of the two-stage truncation and selective continuation strategy over directly generating complete trajectories remain insufficiently clear. The paper proposes a comprehensive solution for the setting in which the generation length is initially limited to C < H. However, an important question worth further discussion is whether, under the same computational budget, first truncating generation at C and then performing reward imputation, selective continuation, and gradient correction necessarily provides greater overall benefits than directly generating a smaller number of complete trajectories. Although the paper compares its method with existing allocation approaches such as VIP and VIGOR, these methods also differ in their sampling strategies and optimization objectives. Therefore, the current results do not fully isolate the benefits introduced by the two-stage truncation and correction mechanism itself. A more direct comparison with a simple strategy that generates trajectories directly to H, under the same training objective and computational budget, would help further clarify the practical necessity and computational efficiency advantages of the proposed framework.

Questions for authors
  1. I have some concerns regarding the imputation head. Although the theoretical analysis shows that the corrected gradient estimator remains unbiased even when the predictions of the imputation head are inaccurate, I am still interested in understanding how important the prediction quality of the imputation head is in practical training. After all, the theoretical guarantee relies on certain assumptions, and even if the gradient estimator is unbiased, does the accuracy of the imputation head really have no effect on training under a finite budget? In particular, how does its prediction quality affect the variance of the corrected gradients and the allocation of the continuation budget? I would appreciate it if the authors could further discuss the calibration of the imputation head and the sensitivity of the final performance to its prediction quality. For example, if a weaker or insufficiently trained imputation head were used, would the proposed method still achieve similar performance improvements?

  2. Could the authors further explain why the performance improvement over VIGOR on HMMT Feb 26 in Table 1 is relatively small? As shown in Table 1, the proposed method clearly outperforms VIGOR on most benchmark tasks. However, on HMMT Feb 26, the improvements for the two models are only 1.4 and 1.0 percentage points, respectively, which are much smaller than those observed on several other benchmarks. In addition, the 95% confidence intervals reported in the paper are calculated from the aggregated results across all six benchmarks rather than separately for each task. Therefore, it is currently difficult to determine whether the improvements on HMMT are consistent, especially considering the relatively small size of the HMMT Feb 26 test set. I would appreciate it if the authors could further explain why the observed improvements on HMMT are noticeably smaller than those on some of the other benchmark tasks.

  3. As a possible direction for future work, I would like to ask whether the authors have considered extending this framework to Agentic RL scenarios. In such tasks, agents typically need to perform long-horizon reasoning, tool calls, and interactions with the environment. Moreover, feedback from the environment may be fed back into the LLM during the trajectory. Would the proposed framework and its theoretical guarantees still hold in Agentic RL settings? Unlike standard language model generation, agent trajectories often involve heterogeneous tool invocation costs, stochastic environment state transitions, and more complex reward signals.

Reviewer_5HS1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a method for budgeting rollouts during online reinforcement learning. Rollouts can be expensive and rolling out till the maximum evaluation context can be expensive during training, so the authors propose an algorithm for deciding which rollouts are worth rolling out before updating the model through a policy gradient. The algorithm is not clearly stated and the paper is not legible. The results all demonstrate that the algorithm has better performance compared to all the baselines they tried such as VIGOR, VIP, PAIR etc on the Qwen3.6-27B and Qwen3.6-35B-A3B models.

Critical strengths

None

Critical weaknesses
  1. Illegible writing: The paper is impossible to read with many undefined terms such as 'Censored Policy Gradients', 'joint prefix-tail residual risk', 'prefix-tail coupling', 'selected-only risk learning', 'segment wise second moments'.
  2. The method is not described in clear description anywhere: What are frozen joint-residual estimates as mentioned in Figure 1? What is vi^\hat{v_i} is Section 3? What is 'form hbh_b' in Algorithm 1? It is unclear what each of these terms mean and how they form the final algorithm, therefore it is unclear what the empirical results are supposed to show and what they mean.
  3. Baseline not described: There is a baseline called 'Length/cost'. What does this baseline denote? Similarly what does the 'Uncertainty' baseline mean?

I would recommend that the authors rewrite the paper to clearly explain what the method is with as little jargon as possible to make the paper readable and the insights/results clearer.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.