acceptodds
Desk rejected at ICLR 2027

Granularity-Adaptive Credit Assignment for Long-Horizon LLM Agent Reinforcement Learning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon language-model agents trained with reinforcement learning oftenreceive sparse outcome rewards that do not reveal which decisions along a tra-jectory deserve credit. Episode-level advantages provide coarse trajectory-widecredit, while step-level comparisons offer finer resolution with context-dependentestimation noise. We propose Granularity-Adaptive Credit Assignment (GACA),a critic-free method that adaptively mixes episode- and step-level credit for eachdecision during policy optimization. GACA normalizes the sampled response'smean per-token negative log-likelihood (NLL) within each trajectory and uses theresulting criticality score to determine the step-specific mixture. The computationreuses rollout log-probabilities without additional training rollouts or model eval-uations. Our analysis characterizes optimal score-dependent mixing and derivesconditions linking expected NLL to a lower bound on the preferred step-levelweight. Across ALFWorld and WebShop with 1.5B and 7B backbones, GACAachieves the highest reported mean success rates among the compared methods,while introducing negligible additional computation.

Primary area:
reinforcement learning
Keywords:
large language model agents, reinforcement learning, credit assignment, advantage estimation, uncertainty-aware learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WFTxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Granularity-Adaptive Credit Assignment (GACA), a critic-free reinforcement learning method that adaptively combines episode-level and step-level advantages. The mixing weight is an increasing function of the sampled response's mean per-token negative log-likelihood (NLL), normalized by the trajectory-average NLL. The method reuses rollout log-probabilities and introduces little additional computation. The theoretical analysis characterizes optimal conditional mixing under squared credit-estimation error and develops a conditional connection between expected NLL, action diversity, and action-value variation. Experiments consider ALFWorld and WebShop with 1.5B and 7B models.

The method is simple and potentially useful, and the matched fixed-weight and score-permutation ablations are encouraging. However, the principal benchmark comparisons do not isolate the algorithmic improvement, the reported uncertainty does not capture training variability, and the evidence for the proposed mechanism remains incomplete.

Critical strengths
  1. Simple and inexpensive method: The adaptive rule requires only existing rollout log-probabilities and scalar operations. The timing breakdown reports approximately 0.09 seconds of additional computation per iteration, making the method attractive as a practical modification to group-based policy optimization.

  2. Internal controls: The ALFWorld 1.5B experiments compare against five fixed mixtures, including a weight close to the adaptive method's empirical mean. The within-trajectory permutation control preserves the collection of weights while changing their assignment to decisions. These are informative controls for distinguishing adaptive allocation from a change in the average mixing coefficient.

  3. Promising empirical results: GACA reports higher mean success rates than the listed references across both benchmarks and model sizes. Within the matched ALFWorld ablation, it improves over the best tested fixed mixture by 3.83% points.

Critical weaknesses
  1. The principal comparisons are not controlled for training budget and implementation differences: Table 1 quotes external results obtained under different framework versions, history windows, training budgets, and evaluation protocols. Appendix D.1 states that GiGPO uses 150 updates and HGPO uses 160 iterations, whereas the main ALFWorld 1.5B GACA result is selected at update 275 from a 300-update run. At update 150, GACA achieves 92.19%, compared with its selected result of 96.88%. The increase from continued training is therefore material relative to the reported margins over the strongest references.

The paper mentioned these differences, but this does not make the comparison causally informative. Likewise, the 15-step and 30-step WebShop configurations cannot be treated as directly comparable: a longer horizon changes both the available opportunities and the difficulty of learning.

I suggest the author(s) conduct matched reruns of GiGPO and HGPO under the same framework, rollout budget, history window, reward configuration, and evaluation protocol to establish the contribution of adaptive mixing. The current results only support the narrower statement that GACA achieves strong attained performance under its chosen configuration.

  1. Training variability and checkpoint-selection uncertainty are insufficiently assessed: For ALFWorld, each backbone uses one trained model, and the reported standard deviations are across three stochastic evaluations of a selected checkpoint. These quantify evaluation randomness conditional on training, not the reproducibility of the training algorithm. The ablation table likewise provides no estimate of variation across independent training runs.

  2. The theoretical results do not establish that the implemented increasing NLL gate improves credit estimation: Proposition 1 correctly characterizes the oracle conditional mixture through the conditional error moments of the two estimators. However, GACA neither estimates these moments nor learns the corresponding oracle gate. Consequently, the proposition establishes the potential value of score-dependent mixing, rather than a guarantee for the particular NLL-based rule.

The subsequent argument is conditional on a coarse-to-fine error model, fixed residual moments, controlled within-action textual entropy, and positive action-value separation. These assumptions are not verified for the implemented group-relative advantages. In particular, the reliability of step-level credit may change with anchor-group size, history inconsistency, and rollout variability, none of which enters the proposed gate.

There is also an important distinction between an increasing lower bound and an increasing target. Corollary 5 provides a nondecreasing lower envelope on the preferred weight as expected NLL increases. It does not imply that the actual optimal weights are ordered by NLL across states. States with low NLL may still have large action-value variation or strongly prefer step-level credit.

Finally, the implementation uses a single sampled NLL normalized by the completed trajectory, rather than expected NLL at a state. Preserving the ordering of sampled NLL within a trajectory does not establish preservation of the ordering of expected NLL or optimal mixture weights. These results are useful motivation, but their connection to the implemented gate should be stated more narrowly.

  1. Important reproducibility details remain incomplete: Table 4 marks the ALFWorld training success reward as unspecified, while the frozen-policy diagnostics use reward 10. This matters because the relative scale of success rewards and invalid-action penalties affects the advantages, even when standard-deviation normalization is applied.

The detailed archived configuration primarily covers ALFWorld 1.5B. Equivalent information is needed for ALFWorld 7B and both WebShop models, including actual update counts, loss reduction, entropy regularization, selection rules, and the source of the reported standard deviations. The distinction between configured epochs and actual parameter updates should also be made explicit.

Questions for authors
  1. Does GACA retain its advantage over GiGPO and HGPO when training budgets, framework versions, reward settings, and checkpoint-selection protocols are matched?

  2. How stable are the adaptive-versus-fixed and adaptive-versus-permuted improvements across independent training seeds? Can the selected checkpoints be evaluated on a separate held-out set?

  3. Do the matched fixed-weight and permutation ablations replicate on WebShop or the 7B backbone?

1 comment

Thank you for focusing on training budgets and variation across runs. GACA has higher mean success than GiGPO at both 150 and 300 updates in the reported three-training-seed comparison. Its mixing rule also accommodates HGPO’s history-based step estimate.

W1/Q1. GACA has higher mean success at both fixed update counts.

At update 150, GACA achieves 92.19 ± 4.75%, compared with 86.46 ± 0.45% for GiGPO, across training seeds 0/1/2. At update 300, the results are 95.31 ± 2.34% and 93.49 ± 1.97%, respectively (Appendix I.1, Table 11). In one fully logged training-seed pair, GACA also uses 8.93% fewer active response steps and 5.99% fewer generated response tokens through update 150 (Appendix I.3, Table 14).

W2/Q2. The reported SD measures variation across trained models.

Each trained model contributes one evaluation score, and the sample SD is computed across training seeds. The permutation control reaches 87.76 ± 2.26% at update 150 across three training seeds. Appendix I.1, Table 11 gives individual scores and evaluation settings; the original fixed-weight sweep is in Appendix G, Table 9.

W1/Q1. HGPO's history-aware grouping can supply GACA's step credit.

HGPO constructs step estimates by grouping steps with consistent histories. GACA controls how much that local estimate contributes relative to episode credit. Using HGPO's output as S in G=(1−λ)E+λS combines these two mechanisms (Appendix A).

W3. The theory motivates the gate through diversity and conditional mixing.

Proposition 1 characterizes the MSE-optimal mixture through conditional error moments. Under the stated assumptions, Corollary 5 gives a nondecreasing lower bound on the optimal step weight at fixed model and regularity parameters; it does not order the actual optimal weights across states. GACA follows the increasing direction using sampled, trajectory-normalized NLL. Normalization preserves the NLL ranking of the sampled responses within each trajectory; the NLL–diversity analysis concerns expected NLL at each state (Section 3.3; Appendices B–C, including C.6).

W4. The execution records specify rewards, training length, and checkpoint selection.

The archived ALFWorld and WebShop 1.5B runs use success reward 10 and invalid-action penalty −0.1. Appendix D records configured epochs, archived updates, and checkpoint selection; Appendix E gives loss reduction and regularization. For example, the WebShop 1.5B result comes from update 225 of a run configured for 300 epochs. Appendix D.1 separately identifies the published 15-step and 30-step WebShop settings and their source protocols.

Q3. The repeated permutation experiment tests the allocation mechanism on ALFWorld 1.5B.

Appendix I.1 adds three training seeds for the permutation control to the fixed-weight and permutation ablations in Appendix G. These ALFWorld 1.5B experiments compare adaptive, fixed, and permuted mixing; the WebShop and 7B experiments evaluate task performance in the other settings.

Reviewer_mnVGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Granularity-Adaptive Credit Assignment (GACA) for training long-horizon LLM agents with sparse outcome rewards. GACA dynamically combines episode-level and step-level advantages using a mixing weight derived from each response’s negative log-likelihood (NLL), normalized within its trajectory. Responses with higher relative NLL receive greater weight on step-level credit. The method reuses existing rollout log-probabilities without additional models or training rollouts. The paper provides theoretical analysis of adaptive mixing and evaluates GACA on ALFWorld and WebShop using 1.5B and 7B models. It reports improved success rates over the listed reference methods, while internal ALFWorld ablations compare adaptive mixing with fixed weights and randomly permuted weight assignments

Critical strengths
  1. Simple design with negligible additional cost. GACA adapts the balance between episode-level and step-level credit using existing rollout log-probabilities. It requires no additional models or training rollouts, making it a practical modification to the existing training framework.

  2. Promising gains supported by targeted ablations. On ALFWorld 1.5B, adaptive mixing outperforms the best tested fixed mixture by 3.83 percentage points. Randomly permuting the weights across steps reduces performance, and all tested nonzero modulation strengths outperform the fixed-weight control. These results support the usefulness of assigning mixing weights to individual decisions.

Critical weaknesses

The rationale for the NLL-based trade-off is not sufficiently established. The ablations support the usefulness of adaptive mixing, but it remains unclear why higher response NLL should imply that step-level credit is more reliable than episode-level credit. A model can make a consequential mistake with high confidence, while a low-probability response may simply reflect unusual wording. The action-diversity analysis provides partial support, but the action-value measurements in Appendix H.3 do not clearly establish greater value variation at higher-NLL states. Concrete trajectory examples, particularly where episode-level and step-level advantages disagree, would help explain how the weighting improves credit assignment and when it may fail. This gap limits my confidence in the proposed mechanism despite the promising empirical results.

Questions for authors
  1. What evidence supports using higher NLL to place greater weight on step-level credit? In particular, how should this interpretation be reconciled with Appendix H.3, which does not show clear action-value separation between higher- and lower-NLL states?

  2. How does the weighting rule handle consequential mistakes made with high confidence? Could the authors discuss when the NLL-based weighting may be inappropriate and whether the retained episode-level signal helps in such cases?

  3. Could the authors provide concrete trajectory examples where episode-level and step-level advantages disagree? Showing the NLL score, mixing weight, and resulting advantage alongside a fixed-mixture alternative would help explain how adaptive mixing produces more appropriate credit assignment

1 comment

Thank you for asking why NLL should affect credit and how confident mistakes are handled. GACA uses relative NLL to adjust the episode–step balance, motivated by action diversity and the conditional mixing analysis. Low-NLL decisions retain both signals, so episode feedback can still contribute to the credit assigned to confident actions.

Q1. Action diversity connects expected NLL to a conditional bound on the step weight.

The paired diagnostic in Appendix H.2 supports the association between NLL and executed-action diversity: 0.179 versus 0.104 at higher- and lower-score states (Table 10). The diagnostic in the original Appendix H.3 did not establish greater action-value variation at higher-NLL states. The theoretical connection from the NLL-based diversity bound to the preferred step weight remains conditional on the stated value-separation and error assumptions (Section 3.3; Appendices B–C).

Q2. At low NLL, both credit terms determine the mixed sign.

For E<0<SE < 0 < S, mixed credit is negative exactly when (1−λ)∣E∣>λS(1-\lambda)|E| > \lambda S. As relative NLL approaches zero, λ\lambda approaches 0.50.5, and the condition becomes ∣E∣>S|E| > S in the equal-weight limit. Thus a sufficiently negative episode estimate can offset a positive step estimate; the default gate also keeps at least half the weight on step credit. If both estimates assign positive credit to a mistaken action, their mixture also remains positive (Section 3.2; Appendix E).

Q3. The same successful trajectory can reward progress and penalize a wasteful action.

The task is to cool a pan and put it in a cabinet. While carrying the cooled pan, the agent closes the cabinet, reopens it, then places the pan inside. Another sample in the same observation group places it immediately, so the unnecessary closing action receives negative step credit (S=−0.707) despite positive episode credit (E=+3.222). Fixed-0.75 mixing stays positive at +0.275; relative NLL 2.288 gives λ=1 and GACA credit −0.707. Earlier, returning to the cabinet has lower relative NLL and retains positive GACA credit (+0.579). Appendix I.2 describes the 21-step trajectory; Table 13 compares the credit formulas on this same recorded rollout.

Reviewer_st46OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Granularity-Adaptive Credit Assignment (GACA), a framework designed to improve credit assignment in long-horizon LLM agent reinforcement learning. GACA adaptively combines episode-level and step-level advantages based on the uncertainty of individual decisions. Specifically, it uses response-level negative log-likelihood (NLL) as a proxy for decision criticality, assigning greater weight to step-level advantages for decisions with higher uncertainty. The proposed method incurs minimal computational overhead and requires neither additional rollouts nor critic models. Experiments on ALFWorld and WebShop using Qwen2.5-based models show consistent improvements over existing baselines, while ablation studies further validate the effectiveness of the adaptive credit assignment mechanism.

Critical strengths
  1. Important and relevant research problem. Credit assignment is a fundamental challenge in long-horizon LLM agent reinforcement learning, where sparse outcome rewards make it difficult to identify the contributions of individual decisions. Exploring adaptive credit granularity is a meaningful direction that could improve training effectiveness.

  2. Simple and efficient method with promising results. The proposed method is straightforward to implement and introduces negligible computational overhead without requiring additional rollouts or critic models. Experiments on ALFWorld and WebShop demonstrate consistent improvements over existing baselines, suggesting the practical potential of the approach.

Critical weaknesses
  1. Limited novelty and insufficient comparisons with related work. The idea of leveraging model uncertainty (e.g., NLL or surprisal) to adaptively modulate advantages has been explored in prior studies, such as AEM (arXiv:2605.00425) and STARE (arXiv:2606.19236). Although GACA differs by interpolating between episode-level and step-level advantages, the underlying idea appears incremental. The lack of direct comparisons with these methods makes it difficult to isolate the contribution of adaptive credit granularity from general uncertainty-based advantage modulation.

  2. Unreliable NLL-based uncertainty estimation. NLL values can vary substantially across different token types, such as reasoning, natural language, and action tokens. Averaging token-level NLL over an entire response may therefore reflect differences in token composition rather than uncertainty about the actual decision. The paper lacks sufficient analysis or ablations to justify response-level NLL as a reliable measure of decision criticality.

  3. Insufficient justification for uncertainty-based credit allocation. High NLL does not necessarily indicate that a decision is more important or requires finer-grained credit assignment. For example, a model may be highly confident in an incorrect yet critical action, while being uncertain about an inconsequential decision. The paper does not sufficiently establish why policy uncertainty should determine the relative contributions of episode-level and step-level advantages.

  4. Ignoring the reliability of step-level advantage estimates. GACA dynamically combines existing episode-level and step-level advantages without explicitly accounting for the estimation quality of the latter. Step-level advantages can be noisy or biased due to limited state matching and sparse rewards, particularly in long-horizon tasks. Assigning greater weight to such estimates based solely on NLL may amplify unreliable credit signals, and the paper lacks sufficient analysis of this potential issue.

Questions for authors

Please see weaknesses.

1 comment

Thank you for asking what GACA adds and why it uses NLL. GACA adjusts the episode–step balance for each decision, with higher relative NLL shifting weight toward step credit. This design is motivated by the NLL–action-diversity link and our conditional analysis of when greater step weighting is preferable.

W1. Novelty: GACA adapts the balance between two sources of credit.

AEM and STARE apply positive weights to existing advantages; GACA changes the relative contributions of episode and step estimates (Appendix A). This can change which actions receive positive credit. On the recorded cabinet-closing action, GiGPO's sum gives +2.515 and fixed-0.75 mixing gives +0.275, whereas GACA gives −0.707. Across 1.62 million recorded steps, the sign differs from fixed-0.75 on about 12% of steps with opposing estimates. These comparisons apply the credit formulas to the same GACA rollouts (Appendix I.2, Tables 12–13).

W2. Token composition: higher scores are associated with more diverse executed actions.

In the paired diagnostic, fresh responses execute different actions more often at higher-score states: 0.179 versus 0.104. Responses that execute the same environment action count as the same action, even when their reasoning text differs. This measurement supports the action-diversity motivation (Appendix H.2, Table 10).

W3. NLL guides credit granularity through action diversity.

Episode credit shares the trajectory outcome across decisions, while step credit compares discounted returns within observation groups. Our model captures how the episode signal can attenuate differences between actions. Under the stated error conditions, greater action-value variation favors more step credit when other model parameters are fixed. Expected NLL connects to this result through action diversity: the assumptions on text variation and average value separation yield a nondecreasing lower bound on the optimal step weight, with regularity and error parameters fixed (Section 3.3; Appendices B–C).

Low-NLL decisions still receive step credit with weight at least 0.5, while sufficiently negative episode credit can offset a positive step estimate (Section 3.2; Appendix E).

W4. Step-estimation noise enters the mixing analysis explicitly.

Proposition 1 expresses the mixture's error using both estimates' conditional mean squared errors and their error cross-moment. The direction in Proposition 2 holds under its stated error conditions. The implementation uses relative NLL to guide the mixture and falls back to prompt-group credit for singleton observation groups (Section 3.3; Appendices C.6 and E).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.