- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Existing LLM-agent RL methods claim to improve credit assignment through different state or history grouping strategies. This paper tries to propose an auditing framework to decouple how samples are grouped and how values/advantages are computed after grouping. After disentangling them, the authors find that GiGPO and G2PO use essentially the same base grouping, while HGPO has different grouping but still produces advantage estimates that are very close to those of GiGPO.
Critical strengths
-
The motivation is clear and I agree with auditing the step-level credit assignment through a clean decomposition can be beneficial
-
The controlled comparison is very clean. It makes the observed differences in the advantage vectors much easier to attribute to that component itself.
Critical weaknesses
-
My main concern is the significance of the contribution. The paper carefully decomposes and attributes step-level advantage estimators, but the analysis does not lead to meaningful guidance for training and new credit-assignment methods. So far in my opinion, the practical and scientific implications of the decomposition remain limited.
-
The technical novelty of the auditing framework seems to also be limited. The ICC analysis seems to be standard, and does not have a lot of practical significance given sparse rewards.
-
The paper writing seems very weird to me. The introduction is unusually brief and does not sufficiently motivate the scientific stakes of the proposed decomposition. And the writing gives me the impression not of a paper, but a blog documenting some thoughts...
Questions for authors
My main question is on the practical value of the paper and it will significantly influence my assessment on the paper:
Is the framework proposed primarily a tool for attributing differences in step-level advantage estimates to different estimator components? If so, what actionable guidance does this attribution provide for agent training or credit-assignment design? For example, does the framework help determine when a particular estimator should be preferred, which components are beneficial under different training regimes, or how to design a better estimator? Currently the paper seems to mainly focus on identifying where estimator differences come from.
5 comments
Official Response to Reviewer 2gsD (Part 2/4)
About the Technical Contribution and Presentation
The ICC analysis seems to be standard, and does not have a lot of practical significance given sparse rewards.
The introduction is unusually brief and does not sufficiently motivate the scientific stakes of the proposed decomposition.
We use standard ICC (§2). Proposition 1 specifies the unbalanced moment identity and its assumptions; its role is to measure within-anchor history association with explicit support and inferential units:
The moment identity applies to the untruncated between component under fixed non-informative label sizes and common independent effects (Proposition 1; Appendix C). Whole-task resampling provides uncertainty intervals for the ICC ratio while preserving within-task trajectory dependence.
Sparse reward makes support accounting consequential: only 90/1,964 preferred anchors carry return variation. The nominal-to-informative count ratio is about 21.82. This ratio quantifies the difference between nominally eligible and return-varying anchor counts.
The contribution is the controlled attribution and support protocol. Its empirical content is the crossed comparison: hierarchy can change while the signal stays close, whereas the same flat partition can accompany a large change when value construction changes (Table 1). The independent producer and Yang fallback checks (§4.6; Appendix B) test this attribution beyond a single emitter. E4–E6 connect the diagnostics to prediction, regularization, and complete-operator training.
The scientific question can be stated directly: when a structured estimator differs from a flat one, is the cause the partition, the value construction, or the available return support? Matched controls address the first two; support analysis addresses the third. The design recommendations and E4–E6 results connect this decomposition to predictive accuracy and policy learning.
Official Response to Reviewer 2gsD (Part 3/4)
About the History Support and Predictive Usefulness (E6)
For example, does the framework help determine when a particular estimator should be preferred, which components are beneficial under different training regimes, or how to design a better estimator?
E6 tests the design choice suggested by finite support: shrink history predictions toward the flat anchor estimate, and select the strength using other tasks.
Experiment settings.
Using the same collections and whole-trajectory holdouts as E4, the raw-history predictor requires at least two training rows from two independent trajectories; otherwise it uses exactly the flat anchor mean. Shrinkage combines history and flat predictions with distinct-trajectory weight n(h): p(kappa)=[n(h)p(history)+kappa p(flat)]/[n(h)+kappa]. The grid is {0,1,2,4,8,16,32,infinity}. For each target task, kappa is selected from pooled loss on all other tasks, with ties favoring larger kappa. Anchor means may still use other trajectories of the target task. We report row-weighted MSE with 2,000 paired whole-task bootstrap draws (seed 25576), repeating task-excluded selection in every draw.
Results.
| Collection | Tasks/rows | Supported rows | Flat MSE | Raw-history MSE | Tuned MSE | Tuned-flat [95% task CI] |
|---|---|---|---|---|---|---|
| ALFWorld 1.5B | 100/33508 | 20206 | 0.413 | 0.302 | 0.262 | -0.152 [-0.293, -0.045] |
| Supplementary 7B | 239/51768 | 25805 | 1.342 | 1.101 | 1.031 | -0.311 [-0.419, -0.217] |
| ScienceWorld/no-op removed | 80/5797 | 2593 | 0.10 | 0.078 | 0.075 | -0.026 [-0.034, -0.018] |
| Collection | Supported flat MSE | Supported tuned MSE | Fallback MSE (all methods) |
|---|---|---|---|
| ALFWorld 1.5B | 0.446 | 0.195 | 0.363 |
| Supplementary 7B | 1.13 | 0.505 | 1.554 |
| ScienceWorld/no-op removed | 0.097 | 0.04 | 0.103 |
Shrinkage reduces MSE relative to both the flat and raw-history predictors. Selected kappa values are 2 for all 100 ALFWorld 1.5B tasks, 1 for all 239 supplementary 7B tasks, and 1 for all 80 ScienceWorld tasks; no task selects another grid value. Selection sets overlap because each excludes only its target task. Fallback predictions are identical across methods.
This shows how the support diagnosis guides a specific, testable regularization change. It evaluates auxiliary RTG prediction; the separate E5 comparison evaluates native-operator policy learning.
Differences, intervals and derived metrics are computed before rounding.
Official Response to Reviewer 2gsD (Part 1/4)
Thank you for recognizing the motivation and controlled comparison. We address your questions below.
About the Practical Value of the Framework
The framework identifies which intervention can test a proposed explanation. History refinement can have a small effect because the flat level survives; additional G2PO value machinery can change signals substantially without changing the flat partition; and subdividing a constant-return anchor cannot create return-relative contrast. These findings guide component selection and experimental design.
The history footprint is quantified beyond cosine: on the verified 1.5B snapshot, normalized mean absolute GiGPO/HGPO difference is 0.072 and strict sign reversals are 0.09% (§4.4; Appendix J). This measures the intervention's size; E5 tests its complete-operator learning outcome.
The support constraint is specific. If every row in an anchor has Ri=c, then zG(Ri)=(c−c)/(sG(R)+ϵ)=0, and each history refinement retains the same zero centered signal. Other components, including an episode-level term, may still contribute to the update.
This leads to a practical sequence: count return-varying anchors, check surviving refinements and value constructions on matched rows, then test learning with independent runs. E4/E6 implement the prediction and regularization checks described in Parts 3–4/4. The eight-pair training extension (E5) gives HGPO-minus-GiGPO final success of +15.039 percentage points at update 20 (95% paired-t interval [5.216, 24.863]; two-sided exact sign-flip p=0.016) on 128 held-out ALFWorld tasks. This comparison evaluates the complete operators, including their normalization and episode-term differences, under the stated 20-update budget. The full protocol and seed table appear in the response to ovnA, Part 3/5.
Thus the added training result addresses downstream usefulness under a fixed budget, while the audit explains which component comparisons are needed to interpret it. ICC remains a standard supporting tool.