acceptodds
Under review as a conference paper at ICLR 2027

From Rollout Scores to Policy Updates: Estimator Completion for Agentic RL

Abstract

Agentic reinforcement learning often scores groups of policy-generated rollouts before selecting which ones enter an update. Selection compresses graded priorities into a mask, leaving open how the discarded score mass should affect the estimator. We formulate this missing step as Estimator Completion (EC). Given a scorer and an optimizer's native aggregation on selected groups, conserving the selected conditional, the scorer's tail conditional, and their partition mass uniquely determines a completed group measure. It is a KL projection of the score distribution, differs from hard filtering by exactly the omitted mass, and yields an explicit head-recalibration and tail-restoration correction. Reward-dispersion filtering provides a concrete test: logged batches retain ordered scores and reward-active rejected groups, and we carry EC's score-conditioned coordinates into PPO, GRPO, DAPO, and DrGRPO policy reductions. Across four tasks, all four Qwen2.5-3B EC variants improve the four-task average of peak-validation means over RAGEN-2; EC-PPO does so across all six tested models. This gives graded rollout selection a principled estimator interface beyond its keep/drop rule.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, agentic reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_PnftOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how soft group priorities from rollout filtering—specifically RAGEN-2's top-pp reward-variance filter where sg=std(Rg)s_g = \text{std}(R_g) and πg∝exp⁡(sg/T)\pi_g \propto \exp(s_g/T)—should be used when forming the policy gradient in agentic RL. Because hard top-pp truncation throws away both the unselected tail mass τ=∑g∉Sπg\tau = \sum_{g \notin S} \pi_g and the relative weights rT(g)=πg/τr_T(g) = \pi_g/\tau inside the tail, the authors propose Estimator Completion (EC). Imposing three consistency conditions (q(⋅∣S)=hSq(\cdot\mid S)=h_S, q(⋅∣Sc)=rTq(\cdot\mid S^c)=r_T, and q(Sc)=τq(S^c)=\tau) uniquely yields the completed weighting q∗=(1−τ)hS+τrTq_* = (1-\tau)h_S + \tau r_T (Theorem 4.1) and gradient correction G∗−GH=−τGH+τGTG_* - G_H = -\tau G_H + \tau G_T (Corollary 4.2). They then build four optimizer variants (EC-PPO, EC-GRPO, EC-DAPO, EC-DrGRPO) under a two-weight template αaHa+βaTa\alpha_a H_a + \beta_a T_a (Eq. 25) and test them on Sokoban, FrozenLake, MetaMathQA, and Countdown.

Critical strengths
  • The paper points out a real quirk in current RLVR pipelines: rollout filters compute soft scores πg\pi_g to pick a subset SS, but the loss reduction then treats all kept groups equally and ignores the dropped mass τ\tau. Splitting the fix into head rescaling (−τGH-\tau G_H) and tail restoration (+τGT+\tau G_T) is intuitive.
  • The evaluation covers four optimizers (PPO, GRPO, DAPO, DrGRPO), six backbones, and four tasks without extra rollouts.
Critical weaknesses
  1. The algorithms in App. B.2 (pp. 14-15) do not follow the main theorem. Sec. 4 derives q∗=(1−τ)hS+τrTq_*=(1-\tau)h_S+\tau r_T (Eq. 14), i.e., (αa,βa)=(1−τ,τ)(\alpha_a,\beta_a)=(1-\tau,\tau) in Eq. (25), as the unique completion. Yet none of the four variants in App. B.2 uses (1−τ,τ)(1-\tau,\tau): (i) EC-GRPO (ll. 745-753) sets (α,β)=(1−τ,0)(\alpha,\beta)=(1-\tau,0), dropping the tail ScS^c completely and only scaling kept advantages by 1−τ1-\tau; (ii) EC-DAPO (Eq. 31) uses (1,τ)(1,\tau) with sum 1+τ>11+\tau>1; (iii) EC-PPO (Eq. 30) uses λB=DT/(DH+DT)≈τ/(∣S∣+τ)\lambda_B=D_T/(D_H+D_T)\approx \tau/(|S|+\tau); and (iv) EC-DrGRPO (ll. 766-777) uses (1,cτ)(1,c\tau) via RMS matching.
  2. Theorem 4.1 (Eq. 14) is just total probability under Eqs. (11)-(13). Also, Figs. 2D/3D define tail activity as ag=M−1∑m∣Rg,m∣a_g=M^{-1}\sum_m |R_{g,m}| (l. 803), but all-correct groups (Rg,m≡1R_{g,m}\equiv 1) have sg=0,ag=1s_g=0, a_g=1 while contributing zero centered advantage (μg=0\mu_g=0) in GRPO/DAPO/DrGRPO.
  3. In Tables 1 and 4, average Qwen2.5-3B gains for PPO/GRPO/DAPO (+0.5 to +1.2 pp) are smaller than the 3-seed SDs (1.5-5.9 pp).
Questions for authors
  1. Since EC-GRPO sets (α,β)=(1−τ,0)(\alpha,\beta)=(1-\tau,0) (dropping the tail, l. 749), EC-DAPO sets (1,τ)(1,\tau), and EC-DrGRPO sets (1,cτ)(1,c\tau), how do these runs validate Theorem 4.1's (1−τ,τ)(1-\tau,\tau) completion?
  2. What fraction of dropped tail groups are all-correct (sg=0,ag=1s_g=0, a_g=1) with μg=0\mu_g=0?

Above all, my major concern is the theorem-algorithm weight discrepancy, I'm open to discuss.

Reviewer_fqo4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

EC reintroduces into the policy loss the softmax score mass of rollout groups that RAGEN-2's filter had discarded, applying four weighting rules tailored to different optimizers. It reports modest average improvements over four tasks and six models.

Critical strengths

This paper conducted multiple sets of experiments, but there was a disconnect between theory and experiment.

Critical weaknesses

1.The theory is cosmetic Theorem 4.1 is nothing more than the law of total probability, while the KL projection, the "exact correction," and the various bounds amount to one-line algebra or triangle inequalities. Nothing accounts for why retaining τ\tau — a quantity tied to an arbitrary softmax temperature and to the reward scale — ought to benefit learning.

  1. The implementation departs from the theory Not one of the four variants employs the derived (1−τ,τ)(1-\tau, \tau) weights; EC-GRPO merely rescales the loss by (1−τ)≥0.9(1-\tau) \ge 0.9, a factor AdamW largely disregards. The reinstated tail groups are predominantly zero-variance groups whose advantages vanish, so EC-DAPO is effectively a no-op on those batches.

  2. The gains lie within noise. The reported figures are peak validation scores over 3 seeds, selected on the very same 512 prompts used for reporting, with improvements of 0.5–2 points against seed standard deviations of 2–5. EC's runs are also roughly 1.7× noisier, and under a best-of-run protocol this alone inflates the reported peak.

4.Novelty is limited The paper adopts RAGEN-2's experimental grid and hyperparameters wholesale and reuses a softmax vector already present in RAGEN's codebase. It fails to cite earlier work on soft reweighting of low-variance groups.

Questions for authors
  1. Please report per-seed, final-step results on a held-out test set.

  2. Please add an ablation of GRPO with the loss simply scaled by (1−τ)(1-\tau), and another that reinstates tail groups with random weights.

  3. What proportion of the reinstated groups exhibit zero reward variance?

Reviewer_t4VgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Estimator Completion (EC), a framework that addresses information loss caused by hard rollout filtering in RAGEN-2 by incorporating discarded score information into policy updates. The authors adapt EC to PPO, GRPO, DAPO, and DrGRPO through optimizer-specific modifications to policy-loss aggregation. Experiments compare EC with RAGEN-2 and unfiltered baselines across four tasks: Sokoban, FrozenLake, MetaMathQA, and Countdown.

Critical strengths
  • The paper identifies the information loss caused by hard rollout filtering and provides a clear motivation for incorporating discarded score information into policy updates. The theoretical analysis provides a principled formulation for addressing this issue.
  • The experimental results show that EC achieves better average performance than RAGEN-2 across four tasks with four policy optimization algorithms.
Critical weaknesses
  • Although the theoretical analysis provides a reasonable motivation for reusing discarded score information, it mainly establishes the validity of the completed probability distribution under the proposed constraints, without providing theoretical guarantees that the resulting policy updates improve learning performance.
  • The authors use different coefficients for different policy optimization algorithms, but it remains unclear how these choices affect performance. For example, EC-GRPO does not incorporate the tail term into the policy update and only rescales the head term. An ablation study using the mass-consistent coefficients derived in Section 4.2 across all four algorithms would make the results more convincing.
Questions for authors
  • Have the authors considered removing the top-p hard filtering step and directly using the softmax-normalized scores to weight all rollout groups in policy updates? How would this approach compare with the proposed EC method?
  • As mentioned in the weaknesses, how would directly applying the mass-consistent coefficients from Section 4.2 affect performance compared to the optimizer-specific coefficients used in Section 5?
Reviewer_DDvLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies an information loss problem in agentic reinforcement learning, where binary rollout filtering discards information about the relative priorities and probability mass of rejected groups.

To address this issue, the authors propose Estimator Completion (EC), which combines the optimizer's native aggregation over selected groups with the scorer's conditional distribution over rejected groups. The authors provide a theoretical characterization of this completion and implement four variants compatible with PPO, GRPO, DAPO, and DrGRPO. Experiments across four agentic RL environments show improvements over RAGEN-2 filtering in average performance.

Critical strengths
  1. The paper identifies the information loss caused by binary rollout filtering and provides a useful head–tail decomposition to characterize the discarded information.
  2. The authors establish the uniqueness and KL projection properties of the proposed mass-consistent completion, providing a mathematically interpretable formulation.
  3. Experiments cover four environments and multiple model configurations, with improvements in average performance over RAGEN-2 across all four Qwen2.5-3B optimizer variants.
Critical weaknesses
  1. The experimental evaluation primarily compares EC with RAGEN-2 hard filtering, leaving unclear whether the proposed completion offers advantages over simpler ways of incorporating scorer information. For example, comparing the full EC method against variants that only recalibrate selected groups or only restore rejected groups, under the same optimizer and experimental setting, would help establish the benefit of combining the two components.
  2. The comparison across four optimizers is conducted primarily with Qwen2.5-3B, while experiments across different model configurations focus on PPO. This leaves the generality of EC across different model–optimizer combinations insufficiently established. Evaluating more combinations would help determine whether the reported gains transfer beyond the configurations currently studied.
  3. The paper reports the score mass and reward activity of rejected groups, but does not empirically examine how much these groups contribute to the policy gradient or how their updates interact with those from selected groups. Since the effect of tail restoration depends on the difference between the head and tail updates, such an analysis would help explain when restoring rejected groups is beneficial.
Questions for authors

Please refer to the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.