- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how soft group priorities from rollout filtering—specifically RAGEN-2's top- reward-variance filter where and —should be used when forming the policy gradient in agentic RL. Because hard top- truncation throws away both the unselected tail mass and the relative weights inside the tail, the authors propose Estimator Completion (EC). Imposing three consistency conditions (, , and ) uniquely yields the completed weighting (Theorem 4.1) and gradient correction (Corollary 4.2). They then build four optimizer variants (EC-PPO, EC-GRPO, EC-DAPO, EC-DrGRPO) under a two-weight template (Eq. 25) and test them on Sokoban, FrozenLake, MetaMathQA, and Countdown.
Critical strengths
- The paper points out a real quirk in current RLVR pipelines: rollout filters compute soft scores to pick a subset , but the loss reduction then treats all kept groups equally and ignores the dropped mass . Splitting the fix into head rescaling () and tail restoration () is intuitive.
- The evaluation covers four optimizers (PPO, GRPO, DAPO, DrGRPO), six backbones, and four tasks without extra rollouts.
Critical weaknesses
- The algorithms in App. B.2 (pp. 14-15) do not follow the main theorem. Sec. 4 derives (Eq. 14), i.e., in Eq. (25), as the unique completion. Yet none of the four variants in App. B.2 uses : (i) EC-GRPO (ll. 745-753) sets , dropping the tail completely and only scaling kept advantages by ; (ii) EC-DAPO (Eq. 31) uses with sum ; (iii) EC-PPO (Eq. 30) uses ; and (iv) EC-DrGRPO (ll. 766-777) uses via RMS matching.
- Theorem 4.1 (Eq. 14) is just total probability under Eqs. (11)-(13). Also, Figs. 2D/3D define tail activity as (l. 803), but all-correct groups () have while contributing zero centered advantage () in GRPO/DAPO/DrGRPO.
- In Tables 1 and 4, average Qwen2.5-3B gains for PPO/GRPO/DAPO (+0.5 to +1.2 pp) are smaller than the 3-seed SDs (1.5-5.9 pp).
Questions for authors
- Since EC-GRPO sets (dropping the tail, l. 749), EC-DAPO sets , and EC-DrGRPO sets , how do these runs validate Theorem 4.1's completion?
- What fraction of dropped tail groups are all-correct () with ?
Above all, my major concern is the theorem-algorithm weight discrepancy, I'm open to discuss.