- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Granularity-Adaptive Credit Assignment (GACA), a critic-free reinforcement learning method that adaptively combines episode-level and step-level advantages. The mixing weight is an increasing function of the sampled response's mean per-token negative log-likelihood (NLL), normalized by the trajectory-average NLL. The method reuses rollout log-probabilities and introduces little additional computation. The theoretical analysis characterizes optimal conditional mixing under squared credit-estimation error and develops a conditional connection between expected NLL, action diversity, and action-value variation. Experiments consider ALFWorld and WebShop with 1.5B and 7B models.
The method is simple and potentially useful, and the matched fixed-weight and score-permutation ablations are encouraging. However, the principal benchmark comparisons do not isolate the algorithmic improvement, the reported uncertainty does not capture training variability, and the evidence for the proposed mechanism remains incomplete.
Critical strengths
-
Simple and inexpensive method: The adaptive rule requires only existing rollout log-probabilities and scalar operations. The timing breakdown reports approximately 0.09 seconds of additional computation per iteration, making the method attractive as a practical modification to group-based policy optimization.
-
Internal controls: The ALFWorld 1.5B experiments compare against five fixed mixtures, including a weight close to the adaptive method's empirical mean. The within-trajectory permutation control preserves the collection of weights while changing their assignment to decisions. These are informative controls for distinguishing adaptive allocation from a change in the average mixing coefficient.
-
Promising empirical results: GACA reports higher mean success rates than the listed references across both benchmarks and model sizes. Within the matched ALFWorld ablation, it improves over the best tested fixed mixture by 3.83% points.
Critical weaknesses
- The principal comparisons are not controlled for training budget and implementation differences: Table 1 quotes external results obtained under different framework versions, history windows, training budgets, and evaluation protocols. Appendix D.1 states that GiGPO uses 150 updates and HGPO uses 160 iterations, whereas the main ALFWorld 1.5B GACA result is selected at update 275 from a 300-update run. At update 150, GACA achieves 92.19%, compared with its selected result of 96.88%. The increase from continued training is therefore material relative to the reported margins over the strongest references.
The paper mentioned these differences, but this does not make the comparison causally informative. Likewise, the 15-step and 30-step WebShop configurations cannot be treated as directly comparable: a longer horizon changes both the available opportunities and the difficulty of learning.
I suggest the author(s) conduct matched reruns of GiGPO and HGPO under the same framework, rollout budget, history window, reward configuration, and evaluation protocol to establish the contribution of adaptive mixing. The current results only support the narrower statement that GACA achieves strong attained performance under its chosen configuration.
-
Training variability and checkpoint-selection uncertainty are insufficiently assessed: For ALFWorld, each backbone uses one trained model, and the reported standard deviations are across three stochastic evaluations of a selected checkpoint. These quantify evaluation randomness conditional on training, not the reproducibility of the training algorithm. The ablation table likewise provides no estimate of variation across independent training runs.
-
The theoretical results do not establish that the implemented increasing NLL gate improves credit estimation: Proposition 1 correctly characterizes the oracle conditional mixture through the conditional error moments of the two estimators. However, GACA neither estimates these moments nor learns the corresponding oracle gate. Consequently, the proposition establishes the potential value of score-dependent mixing, rather than a guarantee for the particular NLL-based rule.
The subsequent argument is conditional on a coarse-to-fine error model, fixed residual moments, controlled within-action textual entropy, and positive action-value separation. These assumptions are not verified for the implemented group-relative advantages. In particular, the reliability of step-level credit may change with anchor-group size, history inconsistency, and rollout variability, none of which enters the proposed gate.
There is also an important distinction between an increasing lower bound and an increasing target. Corollary 5 provides a nondecreasing lower envelope on the preferred weight as expected NLL increases. It does not imply that the actual optimal weights are ordered by NLL across states. States with low NLL may still have large action-value variation or strongly prefer step-level credit.
Finally, the implementation uses a single sampled NLL normalized by the completed trajectory, rather than expected NLL at a state. Preserving the ordering of sampled NLL within a trajectory does not establish preservation of the ordering of expected NLL or optimal mixture weights. These results are useful motivation, but their connection to the implemented gate should be stated more narrowly.
- Important reproducibility details remain incomplete: Table 4 marks the ALFWorld training success reward as unspecified, while the frozen-policy diagnostics use reward 10. This matters because the relative scale of success rewards and invalid-action penalties affects the advantages, even when standard-deviation normalization is applied.
The detailed archived configuration primarily covers ALFWorld 1.5B. Equivalent information is needed for ALFWorld 7B and both WebShop models, including actual update counts, loss reduction, entropy regularization, selection rules, and the source of the reported standard deviations. The distinction between configured epochs and actual parameter updates should also be made explicit.
Questions for authors
-
Does GACA retain its advantage over GiGPO and HGPO when training budgets, framework versions, reward settings, and checkpoint-selection protocols are matched?
-
How stable are the adaptive-versus-fixed and adaptive-versus-permuted improvements across independent training seeds? Can the selected checkpoints be evaluated on a separate held-out set?
-
Do the matched fixed-weight and permutation ablations replicate on WebShop or the 7B backbone?
Thank you for focusing on training budgets and variation across runs. GACA has higher mean success than GiGPO at both 150 and 300 updates in the reported three-training-seed comparison. Its mixing rule also accommodates HGPO’s history-based step estimate.
W1/Q1. GACA has higher mean success at both fixed update counts.
At update 150, GACA achieves 92.19 ± 4.75%, compared with 86.46 ± 0.45% for GiGPO, across training seeds 0/1/2. At update 300, the results are 95.31 ± 2.34% and 93.49 ± 1.97%, respectively (Appendix I.1, Table 11). In one fully logged training-seed pair, GACA also uses 8.93% fewer active response steps and 5.99% fewer generated response tokens through update 150 (Appendix I.3, Table 14).
W2/Q2. The reported SD measures variation across trained models.
Each trained model contributes one evaluation score, and the sample SD is computed across training seeds. The permutation control reaches 87.76 ± 2.26% at update 150 across three training seeds. Appendix I.1, Table 11 gives individual scores and evaluation settings; the original fixed-weight sweep is in Appendix G, Table 9.
W1/Q1. HGPO's history-aware grouping can supply GACA's step credit.
HGPO constructs step estimates by grouping steps with consistent histories. GACA controls how much that local estimate contributes relative to episode credit. Using HGPO's output as S in G=(1−λ)E+λS combines these two mechanisms (Appendix A).
W3. The theory motivates the gate through diversity and conditional mixing.
Proposition 1 characterizes the MSE-optimal mixture through conditional error moments. Under the stated assumptions, Corollary 5 gives a nondecreasing lower bound on the optimal step weight at fixed model and regularity parameters; it does not order the actual optimal weights across states. GACA follows the increasing direction using sampled, trajectory-normalized NLL. Normalization preserves the NLL ranking of the sampled responses within each trajectory; the NLL–diversity analysis concerns expected NLL at each state (Section 3.3; Appendices B–C, including C.6).
W4. The execution records specify rewards, training length, and checkpoint selection.
The archived ALFWorld and WebShop 1.5B runs use success reward 10 and invalid-action penalty −0.1. Appendix D records configured epochs, archived updates, and checkpoint selection; Appendix E gives loss reduction and regularization. For example, the WebShop 1.5B result comes from update 225 of a run configured for 300 epochs. Appendix D.1 separately identifies the published 15-step and 30-step WebShop settings and their source protocols.
Q3. The repeated permutation experiment tests the allocation mechanism on ALFWorld 1.5B.
Appendix I.1 adds three training seeds for the permutation control to the fixed-weight and permutation ablations in Appendix G. These ALFWorld 1.5B experiments compare adaptive, fixed, and permuted mixing; the WebShop and 7B experiments evaluate task performance in the other settings.