- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Learned Hindsight Credit Assignment (LHCA) for training long-horizon LLM agents with sparse outcome rewards. Building on hindsight credit assignment, it trains an online LoRA adapter to estimate the likelihood of an observed action conditioned on its preceding context and future information. The ratio between this hindsight likelihood and the rollout-policy likelihood supplies a turn-level credit signal. In practice, the method applies trajectory-wise normalization, clipping, and a protective mask for successful trajectories, then combines the resulting hindsight advantage with the standard GRPO advantage. Experiments on ALFWorld, Sokoban, and BabyAI report improvements over GRPO and selected HCAPO configurations. Ablations examine future conditioning, the GRPO component, and scoring temperature, showing that performance depends substantially on the choice of future information.
Critical strengths
- The paper provides a reasonably extensive experimental evaluation.
- The writing makes the paper’s core idea relatively easy to understand.
- I examined the paper’s reproducibility and did not identify any major concerns.
Critical weaknesses
I strongly recommend rejecting this paper. The contribution is highly incremental relative to HCAPO, with essentially no novelty in its core components. I rarely make such a strong statement, but the two methods are exceptionally similar. The paper does not demonstrate a satisfactory advance over HCAPO beyond adding a learnable component.
Moreover, I could not find a quantification of the additional training time introduced by this component. Training it necessarily incurs some computational cost, and the modest performance gains may not justify that overhead. Finally, the reported improvement on ALFWorld does not appear to be statistically significant.
Questions for authors
Please address the following questions, although I will most likely maintain my rejection recommendation:
- The performance difference on ALFWorld is very small. Please provide results from additional evaluation runs and a statistical significance test.
- Please report the training time.
- Please clarify what distinctive, non-incremental contributions your method offers over HCAPO beyond introducing a learnable component.
Additionally, the related-work section is relegated to the appendix. I do not consider this an appropriate presentation choice; in other review processes, such as ARR, this would lead to desk rejection.
6 comments
Thank you for your rebuttal. Could you please report the training time of GiGPO and GRPO for a fair comparison?
Regarding Q1: What I would like to see is statistical significance, rather than significance based on aggregated results. In fact, based on the table you provided, your method does not appear to consistently outperform HCAPO (as the confidence intervals cross 0.5), and some results even show performance degradation. Given that your method also incurs a 6%–13% increase in training time compared to HCAPO, could you reassess the actual benefits brought by your method?
Thank you for your quick response. LHCA costs 6-8% more per step than HCAPO on ALFWorld (13-19% on Sokoban) and about 11% more than GRPO and GiGPO. In return, it significantly outperforms GRPO, outperforms HCAPO in aggregate (P = 0.68 and 0.70, both CIs above 0.5), and learns complementary signals on top of GiGPO to give best results at 3B and 7B. Unlike LHCA, HCAPO does not estimate the HCA ratio as its score credits actions the policy already prefers, regardless of their effect on the outcome.
Please see our reply for detailed explanation.
Training time. From step times, we see GRPO GiGPO < HCAPO < LHCA. GRPO and GiGPO use the same training pipeline without any additional forward (GiGPO's anchor-state grouping adds negligible time; Feng et al., 2025, Fig. 6), so their step time is HCAPO's minus its credit pass. With an efficient credit pass, LHCA () takes 1.06–1.08× HCAPO's step time on ALFWorld and 1.13–1.19× on Sokoban, and about 1.11–1.12× GRPO's on ALFWorld (1.23–1.32× on Sokoban). On ALFWorld, generation dominates the step (77–88% of HCAPO's), so the credit pass is a small part of the cost.
Statistical significance against HCAPO. The probability of improvement with a stratified bootstrap is itself a statistical significance test recommended by Agarwal et al. (2021) for RL with few seeds per setting. For HCAPO, both intervals are above 0.5: 0.68 [0.56, 0.79] with and 0.70 [0.58, 0.82] with . The intervals that include 0.5 are against GiGPO, where we claim parity, not improvement. We agree that per-setting tests against HCAPO are not significant. On ALFWorld all hindsight methods reach 85–88%, and re-evaluating one checkpoint already varies by 2.4 points, so differences between them are not resolvable for any method pair. The cells where LHCA is slightly below HCAPO are within this noise. The gains concentrate where credit assignment is harder: LHCA is ahead of HCAPO in 5 of 6 Sokoban settings (per-setting P = 0.76–0.84) and in both BabyAI settings.
What the additional cost provides.
Over GRPO. +20.5, +11.2 and +7.5 points on ALFWorld at 1.5B, 3B and 7B, with every LHCA seed above every GRPO seed at 1.5B and 3B.
Over HCAPO.
- LHCA follows HCA theory HCA credits an action by the ratio , and this ratio is what makes the hindsight identity (Prop 1) recover . LHCA estimates this ratio. HCAPO's normalized score is proportional to alone, because its denominator is the same for every turn of the trajectory, so the identity does not hold for it.
- How it matters When the future does not carry any information about an action (), HCA and our proposed approach result in and the update reduces to GRPO. HCAPO's score instead increases with policy's own probability of the action, so the actions the policy already prefers receive more credit regardless of their effect on the outcome.
- Empirical Differences LHCA outperforms HCAPO in aggregate under the same future statistic, and its learned estimator improves with training instead of relying on the policy's zero-shot judgment.
Relative to GiGPO.
- On their own, LHCA and GiGPO perform on par (P = 0.55 and 0.61). They obtain credit from different sources: GiGPO compares actions taken from identical anchor states, which requires exact state matches across trajectories (a limitation its authors note), while LHCA learns credit from the observed future and needs no repeated states.
- The two methods are therefore complementary, and LHCA adds directly on top of GiGPO at essentially the cost of LHCA alone: GiGPO+LHCA achieves the best mean result in every ALFWorld and Sokoban setting at 3B and 7B, and outperforms GiGPO in aggregate (P = 0.65 [0.51, 0.79] with ).
- It is also more stable: on Sokoban 7B, GiGPO's 95% CI half-width is ±26.2 points, versus ±8.8 for GiGPO+LHCA (, 63.5 vs 57.2 mean).
Rebuttal Response
We thank the reviewer for the review and for noting that the core idea is easy to follow, and the method reproducible. We address the significance (Q1), training time (Q2), and the difference from HCAPO (Q3), where we show that the two methods estimate different quantities. We will upload the revised PDF, with all changes highlighted, shortly and will notify you when it is available.
Q1. Statistical significance and additional evaluation runs. We reran every method with 5 seeds at three model sizes (324 training runs), with 95% confidence intervals and the probability of improvement (common reply).
Q2. Training time. We report step times in the common reply (Timing Analysis). The two methods differ only in the credit pass, and generation dominates the step. With an efficient credit pass, LHCA with oT is within 6–8% of HCAPO's step time on ALFWorld at every size (6–19% across benchmarks), and this overhead provides a measurable gain over HCAPO under the same future statistic.
Q3. Contributions over HCAPO. We respectfully disagree that the contribution of LHCA over HCAPO is "adding a learnable component".
In short, LHCA estimates the hindsight ratio that HCA prescribes, with a learned estimator that costs a LoRA adapter rather than a second full-size model, placing it between training-free prompted methods (HCAPO) and full-model learned estimators (iStar, PRIME).
Related work. We agree, and will move related work into the main text, with an expanded comparison to HCAPO, iStar, GiGPO and PRIME.
References: