acceptodds
Under review as a conference paper at ICLR 2027

Learning to Assign Credit in Hindsight for Long-Horizon LLM Agents

Abstract

Sparse outcome rewards make it hard to tell which actions actually mattered, and training on them directly risks treating pivotal actions and irrelevant ones alike. Existing work has approached this issue through both prospective credit estimation and hindsight prompting. We take a complementary approach: rather than prompt for hindsight, we explicitly learn it. We ask: Given the outcome and the state, how likely was the action that was taken? Comparing this to current policy likelihood provides a fine-grained attribution of each action to the realized future, letting us amplify or suppress its share of reward accordingly. We train a lightweight parameter-efficient estimator of this hindsight conditional, avoiding repeated prompting and reliance on instruction following. We integrate the resulting action-level attribution into the GRPO policy-gradient objective to construct our method, Learned Hindsight Credit Assignment (LHCA). Through extensive experiments and ablations on ALFWorld, Sokoban and BabyAI, we show LHCA outperforms prior credit-assignment methods, including prompted hindsight and preference-trained process-reward estimators, highlighting the effectiveness of explicitly learning hindsight attribution.

Primary area:
reinforcement learning
Keywords:
Intermediate rewards, long horizon RL, Hindsight credit assignment, Agentic RL

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_tkKbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Learned Hindsight Credit Assignment (LHCA) for training long-horizon LLM agents with sparse outcome rewards. Building on hindsight credit assignment, it trains an online LoRA adapter to estimate the likelihood of an observed action conditioned on its preceding context and future information. The ratio between this hindsight likelihood and the rollout-policy likelihood supplies a turn-level credit signal. In practice, the method applies trajectory-wise normalization, clipping, and a protective mask for successful trajectories, then combines the resulting hindsight advantage with the standard GRPO advantage. Experiments on ALFWorld, Sokoban, and BabyAI report improvements over GRPO and selected HCAPO configurations. Ablations examine future conditioning, the GRPO component, and scoring temperature, showing that performance depends substantially on the choice of future information.

Critical strengths
  1. The paper provides a reasonably extensive experimental evaluation.
  2. The writing makes the paper’s core idea relatively easy to understand.
  3. I examined the paper’s reproducibility and did not identify any major concerns.
Critical weaknesses

I strongly recommend rejecting this paper. The contribution is highly incremental relative to HCAPO, with essentially no novelty in its core components. I rarely make such a strong statement, but the two methods are exceptionally similar. The paper does not demonstrate a satisfactory advance over HCAPO beyond adding a learnable component.

Moreover, I could not find a quantification of the additional training time introduced by this component. Training it necessarily incurs some computational cost, and the modest performance gains may not justify that overhead. Finally, the reported improvement on ALFWorld does not appear to be statistically significant.

Questions for authors

Please address the following questions, although I will most likely maintain my rejection recommendation:

  1. The performance difference on ALFWorld is very small. Please provide results from additional evaluation runs and a statistical significance test.
  2. Please report the training time.
  3. Please clarify what distinctive, non-incremental contributions your method offers over HCAPO beyond introducing a learnable component.

Additionally, the related-work section is relegated to the appendix. I do not consider this an appropriate presentation choice; in other review processes, such as ARR, this would lead to desk rejection.

6 comments
AuthorsOpenReview ↗

Rebuttal Response

We thank the reviewer for the review and for noting that the core idea is easy to follow, and the method reproducible. We address the significance (Q1), training time (Q2), and the difference from HCAPO (Q3), where we show that the two methods estimate different quantities. We will upload the revised PDF, with all changes highlighted, shortly and will notify you when it is available.

Q1. Statistical significance and additional evaluation runs. We reran every method with 5 seeds at three model sizes (324 training runs), with 95% confidence intervals and the probability of improvement (common reply).

  • ALFWorld: LHCA's improvement over GRPO is significant at every size: +20.5, +11.2 and +7.5 points (oTo_T, out-of-distribution) at 1.5B, 3B and 7B. A randomly chosen LHCA seed beats a randomly chosen GRPO seed with probability 1.00, 1.00 and 0.96 (exact Mann–Whitney p = 0.008, 0.008 and 0.016). Over all benchmarks and sizes, P(LHCA > GRPO) = 0.81 [0.71, 0.90], and LHCA also significantly outperforms iStar and PRIME (P = 0.88–0.99).
  • Against HCAPO, with the same future statistic, LHCA outperforms HCAPO in aggregate over benchmarks and model sizes: P(LHCA > HCAPO) = 0.68 [0.56, 0.79] with oTo_T and 0.70 [0.58, 0.82] with τ>t\tau_{>t}, both CIs above 0.5.
  • Re-evaluating a fixed ALFWorld checkpoint with 5 sampling seeds gives an evaluation std of 2.4 points, so single-setting differences of a few points are within evaluation noise for any method. This is why we report the aggregate statistic.

Q2. Training time. We report step times in the common reply (Timing Analysis). The two methods differ only in the credit pass, and generation dominates the step. With an efficient credit pass, LHCA with oTo_T is within 6–8% of HCAPO's step time on ALFWorld at every size (6–19% across benchmarks), and this overhead provides a measurable gain over HCAPO under the same future statistic.

Q3. Contributions over HCAPO. We respectfully disagree that the contribution of LHCA over HCAPO is "adding a learnable component".

  • HCAPO, as described, estimates the hindsight likelihood. HCAPO defines ρt=h(at∣st,F)/π(at∣st)\rho_t = h(a_t \mid s_t, F)/\pi(a_t \mid s_t) (Eq. 5) and motivates the denominator by π(at∣st)=EF[h(at∣st,F)]\pi(a_t \mid s_t) = \mathbb{E}_{F}[h(a_t \mid s_t, F)], an average over futures at a fixed state and action. The denominator used in Eq. 7 is the mean hindsight score over the turns of the trajectory, 1T∑kh(ak∣sk,F)\frac{1}{T}\sum_{k} h(a_k \mid s_k, F), which is the same for every turn. The resulting ρt\rho_t is therefore proportional to h(at∣st,F)h(a_t \mid s_t, F), and the policy likelihood does not enter.
  • LHCA estimates the hindsight ratio. LHCA divides the hindsight likelihood by the policy's likelihood of the same action, token by token, so each ρt\rho_t estimates h/πh/\pi as in HCA (Harutyunyan et al., 2019). Trajectory normalization is applied on top of this ratio, removing an offset shared by all turns.
  • Learned but efficient. HCAPO obtains hh by prompting the policy being trained with the outcome, so its credit depends on the base model's zero-shot judgement of its own actions. LHCA trains hh directly, by reconstructing the policy's own actions conditioned on their futures, so the estimate improves with experience. Unlike iStar and PRIME, which train a second full-size model as an implicit process reward model to learn step-level advantages, LHCA trains only a LoRA adapter, trained online alongside the policy.
  • Results. Compared like-for-like, LHCA outperforms HCAPO in aggregate (Q1), and it is complementary to GiGPO: GiGPO+LHCA has the highest mean in every ALFWorld and Sokoban setting at 3B and 7B.

In short, LHCA estimates the hindsight ratio that HCA prescribes, with a learned estimator that costs a LoRA adapter rather than a second full-size model, placing it between training-free prompted methods (HCAPO) and full-model learned estimators (iStar, PRIME).

Related work. We agree, and will move related work into the main text, with an expanded comparison to HCAPO, iStar, GiGPO and PRIME.

References:

  • Harutyunyan, A., Dabney, W., Mesnard, T., Gheshlaghi Azar, M., Piot, B., Heess, N., van Hasselt, H., Wayne, G., Singh, S., Precup, D., and Munos, R. Hindsight credit assignment. In Advances in Neural Information Processing Systems 32, 2019.

Thank you for your rebuttal. Could you please report the training time of GiGPO and GRPO for a fair comparison?

Regarding Q1: What I would like to see is statistical significance, rather than significance based on aggregated results. In fact, based on the table you provided, your method does not appear to consistently outperform HCAPO (as the confidence intervals cross 0.5), and some results even show performance degradation. Given that your method also incurs a 6%–13% increase in training time compared to HCAPO, could you reassess the actual benefits brought by your method?

Thank you for your quick response. LHCA costs 6-8% more per step than HCAPO on ALFWorld (13-19% on Sokoban) and about 11% more than GRPO and GiGPO. In return, it significantly outperforms GRPO, outperforms HCAPO in aggregate (P = 0.68 and 0.70, both CIs above 0.5), and learns complementary signals on top of GiGPO to give best results at 3B and 7B. Unlike LHCA, HCAPO does not estimate the HCA ratio as its score credits actions the policy already prefers, regardless of their effect on the outcome.

Please see our reply for detailed explanation.

Training time. From step times, we see GRPO ≈\approx GiGPO < HCAPO < LHCA. GRPO and GiGPO use the same training pipeline without any additional forward (GiGPO's anchor-state grouping adds negligible time; Feng et al., 2025, Fig. 6), so their step time is HCAPO's minus its credit pass. With an efficient credit pass, LHCA (oTo_T) takes 1.06–1.08× HCAPO's step time on ALFWorld and 1.13–1.19× on Sokoban, and about 1.11–1.12× GRPO's on ALFWorld (1.23–1.32× on Sokoban). On ALFWorld, generation dominates the step (77–88% of HCAPO's), so the credit pass is a small part of the cost.

Statistical significance against HCAPO. The probability of improvement with a stratified bootstrap is itself a statistical significance test recommended by Agarwal et al. (2021) for RL with few seeds per setting. For HCAPO, both intervals are above 0.5: 0.68 [0.56, 0.79] with oTo_T and 0.70 [0.58, 0.82] with τ>t\tau_{>t}. The intervals that include 0.5 are against GiGPO, where we claim parity, not improvement. We agree that per-setting tests against HCAPO are not significant. On ALFWorld all hindsight methods reach 85–88%, and re-evaluating one checkpoint already varies by 2.4 points, so differences between them are not resolvable for any method pair. The cells where LHCA is slightly below HCAPO are within this noise. The gains concentrate where credit assignment is harder: LHCA is ahead of HCAPO in 5 of 6 Sokoban settings (per-setting P = 0.76–0.84) and in both BabyAI settings.

What the additional cost provides.

Over GRPO. +20.5, +11.2 and +7.5 points on ALFWorld at 1.5B, 3B and 7B, with every LHCA seed above every GRPO seed at 1.5B and 3B.

Over HCAPO.

  • LHCA follows HCA theory HCA credits an action by the ratio ρ=h/π\rho = h/\pi, and this ratio is what makes the hindsight identity (Prop 1) recover Q(s,a)Q(s, a). LHCA estimates this ratio. HCAPO's normalized score is proportional to hh alone, because its denominator is the same for every turn of the trajectory, so the identity does not hold for it.
  • How it matters When the future does not carry any information about an action (h=πh = \pi), HCA and our proposed approach result in ρ=1\rho = 1 and the update reduces to GRPO. HCAPO's score instead increases with policy's own probability of the action, so the actions the policy already prefers receive more credit regardless of their effect on the outcome.
  • Empirical Differences LHCA outperforms HCAPO in aggregate under the same future statistic, and its learned estimator improves with training instead of relying on the policy's zero-shot judgment.

Relative to GiGPO.

  • On their own, LHCA and GiGPO perform on par (P = 0.55 and 0.61). They obtain credit from different sources: GiGPO compares actions taken from identical anchor states, which requires exact state matches across trajectories (a limitation its authors note), while LHCA learns credit from the observed future and needs no repeated states.
  • The two methods are therefore complementary, and LHCA adds directly on top of GiGPO at essentially the cost of LHCA alone: GiGPO+LHCA achieves the best mean result in every ALFWorld and Sokoban setting at 3B and 7B, and outperforms GiGPO in aggregate (P = 0.65 [0.51, 0.79] with τ>t\tau_{>t}).
  • It is also more stable: on Sokoban 7B, GiGPO's 95% CI half-width is ±26.2 points, versus ±8.8 for GiGPO+LHCA (oTo_T, 63.5 vs 57.2 mean).
Reviewer_5kvoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Learned Hindsight Credit Assignment (LHCA), which transfers Hindsight Credit Assignment (HCA) (Harutyunyan et al., 2019) to GRPO-trained LLM agents. The paper uses a LoRA adapter to model the hindsight distribution of HCA, which returns the action distribution probs, conditioned both on the current state and statistics of the future. It then weights the GRPO by the hindsight likelihood of the action taken at each turn. The paper evaluates LHCA on ALFWorld, Sokoban, BabyAI against GRPO, iStar and HCAPO over 2 seeds.

Critical strengths

The paper is sound and the idea of transferring HCA to LLM post-training is very promising. HCA's hindsight ratio recovers the state-action value under its assumptions (Prop. 1), and learning it directly is a natural way to bring hindsight to LLM agents without prompting..

Critical weaknesses

Despite a sound idea, the actual algorithm implementation drifts quite strongly from the theoretical foundations of HCA, the experiments are too limited to support the paper's claims at the moment, and the Related Work section is missing in the main text.

Major

1. Methods

  • The algorithm implemented drifts from the theoretical estimator in Prop. 1, to the point of raising questions on how much the algorithm actually estimates the hindsight advantage anymore:
    • The hindsight likelihood is length-normalised and temperature-sharpened (Eq. 8), so it is not a probability of the action anymore;
    • The hindsight ratio is normalised by its trajectory mean and clipped to [0.6, 1.4] (Eq. 11), masked on successful trajectories, and added to the GRPO advantage (App. A.4). Prop. 1 also needs Gt=γT−tR(τ)G_t = \gamma^{T-t}R(\tau) to be a function of f(τ>t)f(\tau_{>t}), which does not hold for f=oTf = o_T (the remaining number of turns is not determined by the final observation) nor on Sokoban, whose reward sums step rewards from before tt (lines 912–915).
    • For these reasons, how far does the practical signal stay from QπQ^\pi is not clear, and the paper proposes no ablation to understand it.
    • For the same reasons, Algorithm 1. does not correspond anymore to what's actually computed, making it harder to evaluate the method properly.
    • It follows that the proposed method only vaguely resembles HCA and its strong theoretical foundations.
    • What are the results without these stabilisation devices? While there are some ablations, it is not clear how the method does without them, and the specific reasons they are necessary beyond improving performance.
  • LHCA uses per-turn discount to calculate the return (Eq. 10). GRPO uses undiscounted returns (Eq. 2), so, from these results, it is impossible to disentangle the effects of hindsight from discounting. The GRPO baseline should be run on discounted rewards (or added to the canonical GRPO one).

2. Claims

  • Outperforming prior methods. "LHCA outperforms prior credit-assignment methods, including prompted hindsight and preference-trained process-reward estimators" (lines 22–24). Against HCAPO the differences are 1.0, 2.9 and 3.4 points (0.4, 1.6 and 1.7 standard errors over 2 seeds), and with the same future statistic (oTo_T) LHCA is worse on Sokoban (45.0 vs 48.2, Table 4). iStar collapses on ALFWorld for reasons the paper cannot explain (App. A.5). "Matches" is what the results show.
  • Calibration. "Hindsight signal Calibration matters" (line 363), but calibration is never measured, only success is (Figure 1.). Instead, if the paper wants to measure how well the hindsight distrubution is learned, calibration should measure the accuracy of the learned distribution on an held-out validation set.
  • Model scale. "on all three benchmarks and both model scales" (lines 360–361): each benchmark uses one model size only?
  • Additional costs. "without requiring any additional computation" (lines 855): LHCA trains an adapter every step and adds a forward pass per action; its cost is not reported.

3. Evidence

  • The paper targets long-horizon decision problems for LLM agents. This problem has known benchmarks (Paglieri et al., 2025), but doesn't evaluate on them. The paper should investigate the performance against them, to contextialise the method's performance and evaluate the method solidly.
  • Results for GiGPO (Feng et al., 2025) should be reported as baselines as well.
  • 2 seeds are too few to separate LHCA from HCAPO.
  • When the future statistic is fixed, LHCA does not beat HCAPO on both benchmarks. LHCA comes out ahead only with the statistic chosen per benchmark on reported results ("best future conditioning", l. 327–328), which makes the results look cherry-picked. Why LHCA's best statistic changes is not explained: the text accounts for HCAPO's behaviour (l. 403–404) but leaves LHCA's open ("necessitating more careful design", l. 406–407).

4. Related work

  • The main text has no related work section: it is in the appendix (A.1), as a list of methods with one line each. Without it, the paper never positions LHCA against the closest work in the main text. I cannot support acceptance if the related work is not in the main text.

Minor

5. Writing

  • The motivation that hindsight "typically requires an LLM to reason about the completed trajectory" (lines 47–48) is contradicted by the paper's own related work, which lists counterfactual rollouts, causal models and distance-based methods.
  • The AI use statement still contains a template placeholder ("[If applicable: …]", lines 489).
  • The reproducibility statement cites the wrong equations (Eqs. 9–10 and 12, lines 507–508).
  • Hindsight belongs to preliminary, not to methods.

References

  • Feng, L., Xue, Z., Liu, T., and An, B. Group-in-group policy optimization for LLM agent training. In Advances in Neural Information Processing Systems 38, 2025.
  • Harutyunyan, A., et al. Hindsight credit assignment. In Advances in Neural Information Processing Systems 32, 2019.
  • Liu, X., Wang, K., Wu, Y., Huang, F., Li, Y., Jiao, J., and Zhang, J. Agentic reinforcement learning with implicit step rewards. In The Fourteenth International Conference on Learning Representations, 2026.
  • Paglieri, D., Cupiał, B., Coward, S., Piterbarg, U., Wołczyk, M., Khan, A., Pignatelli, E., Kuciński, Ł., Pinto, L., Fergus, R., Foerster, J. N., Parker-Holder, J., and Rocktäschel, T. BALROG: Benchmarking agentic LLM and VLM reasoning on games. In The Thirteenth International Conference on Learning Representations, 2025.
  • Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024.
  • Tan, H.-Z., Yang, X.-W., Chen, H., Shao, J.-J., Wen, Y., Shen, Y., Luo, W., Du, X., Guo, L.-Z., and Li, Y.-F. Hindsight credit assignment for long-horizon LLM agents. CoRR, abs/2603.08754, 2026.
Questions for authors

Additional questions to those in the weaknesses:

  • The whole purpose of the original HCA formulation is to reduce the variance of a return estimator by conditioning the action evaluation on samples from the policy. This helps as the variance of sampled future trajectories is lower than the variance of the true marginal. I wonder whether the lack of strong results depends the fact that GRPO uses MC returns, rather than a learned baseline (the state-conditional HCA, which works better, does).
  • Do you need a turn-level MDP? Can't the current MDP formalism already formalise the problem?
  • The paper never says what metrics the numbers in the Table 1 and 2 report. What is it? Success rate?
13 comments
AuthorsOpenReview ↗

Rebuttal Response (Part 1)

We thank the reviewer for a thorough and constructive review and for finding the idea sound and promising. We address the relation between the implemented estimator and Prop. 1 (P1-P3), the claims (P4–P7), and the evidence (P8–P11: 5 seeds, three model sizes, GiGPO, like-for-like future statistics). We will also move related work into the main text (P12). We will upload the revised PDF, with all changes highlighted, shortly and will notify you when it is available.

P1. The algorithm implemented drifts from the theoretical estimator in Prop. 1

LHCA adds practical components beyond the theoretical estimator. Each component addresses a specific failure mode of the raw hindsight ratio at LLM scale. For each, we give the reason it can be helpful and its ablation (1.5B; full table in the common reply):

  • Length normalization and temperature (Eq. 8). LLMs produce probabilities at the token level, so h/πh/\pi can only be computed per token. The sequence level ratio is a product over the action's tokens, so its scale grows exponentially with action length (e.g., a per-token ratio of 1.05 over 40 tokens gives ≈7\approx 7), and its variance would be governed by length rather than how informative an action is. We therefore use geometric mean of the token level ratios, which puts all actions in the same scale. This is standard for sequence-level importance ratios (GSPO; Zheng et al., 2025). TtempT_{\text{temp}} controls how far ρ\rho can move from 1 and is selected by a sweep. We will state this explicitly.
  • Trajectory normalization. The adapter can assign uniformly higher likelihood to all actions under some futures. Dividing by the trajectory mean removes this offset, so ρ\rho only redistributes credit among the turns of a trajectory (−3.0 on ALFWorld and −5.8 on Sokoban without it).
  • Do-no-harm. Prevents the hindsight term from penalizing actions in successful trajectories. It is a small safeguard, not integral to the gains (within 1.5 points)..
  • GRPO macro term. By Proposition 2 (GRPO+LHCA decomposition), the hindsight advantage is a multiple of the GRPO advantage plus a hindsight correction; where ρ\rho is noisy, the correction can crowd out the outcome signal. The macro term keeps the GRPO outcome signal, which is always a valid policy-gradient direction, in every update, so noisy hindsight cannot override it (−12.2 on Sokoban without it). GiGPO likewise combines episode- and step-level advantages and finds that removing either degrades performance (Feng et al., 2025, Sec. 5.4).
  • How far the practical signal stays from QHQ^H. The shuffled-ρ\rho control keeps every component and the distribution of ρ\rho, and only breaks which action receives which credit. It costs 6.8 points on ALFWorld and 5.2 on Sokoban, so the practical ρ\rho still assigns credit to the right actions.
  • Without the components, the estimator closest to Prop. 1 (no macro term, no do-no-harm, γ=1\gamma = 1, no trajectory normalization, and no tempering, Ttemp=1T_{\text{temp}} = 1) loses 17.2 points on ALFWorld and 11.4 on Sokoban, as the raw ratio is too noisy to train with at this scale (on Sokoban, the per-step spread of ρ\rho is 3$\times$ larger than at our default Ttemp=3T_{\text{temp}} = 3: std 0.033 vs 0.012, with values up to 3.9). We will explicitly state the qualitative and quantitative importance of each component.

We will rewrite Algorithm 1 to match the actual computation (normalization, clipping, do-no-harm, macro term).

P2. Prop 1 needs Gt=γT−tR(τ)G_t=\gamma^{T-t}R(\tau) to be a deterministic function of f(oT)f(o_T).

We thank the reviewer for pointing this out. Our submission did not state the hindsight input precisely. The hindsight model is conditioned on [ hindsight_prompt | future | outcome_summary | promptt\text{prompt}_t ] where outcome_summary is "Success = True/False, Reward = R(τ)R(\tau), Total Turns = T" and promptt\text{prompt}_t contains the tt turns so far. T−tT - t and R(τ)R(\tau) are therefore both determined by the conditioning, and Gt=γT−tR(τ)G_t = \gamma^{T−t}R(\tau) is a deterministic function of it, as Prop. 1 requires. We will state the exact input format in the paper. On Sokoban, step rewards are summed into R(τ)R(\tau) and assigned at the end of the episode, so every turn's return is γT−tR(τ)\gamma^{T-t}R(\tau), which is determined by the conditioning in the same way.

AuthorsOpenReview ↗

Rebuttal Response (Part 2)

P3. GRPO with discounting

Our ρ=1\rho=1 is exactly this baseline as it keeps LHCA's discounted turn-level return γT−tR\gamma^{T-t}R and all other components, and removes only the hindsight ratio, so it is GRPO with discounted turn-level advantages. On Sokoban, LHCA outperforms the discounted baseline by 6.7 points (49.4 vs 42.7), which falls to the level of plain GRPO (41.4). On ALFWorld, the discounted baseline is more competitive (87.3 vs 88.1). Discounting alone recovers a lot of gain over plain GRPO (64.9), but hindsight credit still matters here as assigning the same ρ\rho values to the wrong turns (shuffled ρ\rho) costs 6.8 points.

Both components are part of LHCA as GRPO natively does not apply turn-level advantages, and the full method is best or tied for best on both benchmarks.

P4. Outperforming prior methods.

With 5 seeds and three model sizes, the revised results support the claim (see common reply):

  • LHCA significantly outperforms GRPO (P(improvement) 0.81 and 0.82, CIs above 0.5).
  • With the same future statistic, LHCA outperforms HCAPO: P(LHCA > HCAPO) = 0.68 [0.56, 0.79] with oTo_T and 0.70 [0.58, 0.82] with τ>t\tau_{>t}, both CIs above 0.5.
  • LHCA is competitive with GiGPO and complementary to it: GiGPO+LHCA has the highest mean in every ALFWorld and Sokoban cell at 3B and 7B.

We will state precisely that the comparison with HCAPO is an aggregate one over benchmarks and model sizes.

We tried several ways to address the iStar collapse:

  • The released iStar implementation does not update its implicit PRM. The DPO loss computation is detached from the PRM parameters, code, so the PRM gradient is exactly zero throughout training (an open issue on the repository reports the same).
  • We therefore re-implemented iStar following the paper's equations and hyperparameters, and verified that it trains stably on WebShop, the paper's own benchmark.
  • On ALFWorld at 1.5B, training collapses through the valid-action ratio (from ≈0.99\approx 0.99 to ≈0.1\approx 0.1), and success follows it. None of the variants we tried (summed vs averaged trajectory reward, with or without advantage normalization) prevented this. At 3B and 7B iStar trains, but stays well below LHCA.

To check that the gap to iStar is not specific to one implementation, we add PRIME (Cui et al., 2025), the implicit process-reward method that iStar builds on. We follow PRIME's algorithm, adapted to multi-turn as in the iStar paper, with two stabilizing modifications (a length-normalized step reward and a class-balanced PRM loss). LHCA outperforms both: P(LHCA > PRIME) = 0.90 and P(LHCA > iStar) = 0.99 (oTo_T), CIs above 0.5.

P5. Calibration. "Hindsight signal Calibration matters" (line 363), but calibration is never measured, only success is (Figure 1.). Instead, if the paper wants to measure how well the hindsight

We agree that "calibration" is not the right term for what Figure 1 measures, which is its downstream success as function of the hindsight source. In the revision, this comparison will be made directly in the main multi-seed results, where LHCA outperforms HCAPO in aggregate under the same future statistic and we will remove the term 'calibration'.

P6. Model scale. "on all three benchmarks and both model scales" (lines 360–361): each benchmark uses one model size only?

The revised results cover all three benchmarks across model sizes: Qwen2.5-1.5B, 3B and 7B-Instruct on ALFWorld and Sokoban, and 7B on BabyAI.

P7. Additional costs. "without requiring any additional computation" (lines 855): LHCA trains an adapter every step and adds a forward pass per action; its cost is not reported.

We will correct this sentence. LHCA trains a small LoRA adapter at every step and adds one hindsight forward pass per turn. We now report wall-clock overhead over HCAPO in the common reply (Timing analysis). Unlike iStar and PRIME, which train a second full-size model as an implicit process reward model, LHCA trains only a LoRA adapter.

AuthorsOpenReview ↗

Rebuttal Response (Part 3)

P8. The paper targets long-horizon decision problems for LLM agents. This problem has known benchmarks (Paglieri et al., 2025), but doesn't evaluate on them. The paper should investigate the performance against them, to contextialise the method's performance and evaluate the method solidly.

ALFWorld and Sokoban are the established testbeds for RL training of multi-turn, long horizon LLM agents and in particular credit assignment across turns (GiGPO, Feng et al., 2025; RAGEN, Wang et al., 2025; Turn-PPO, Li et al., 2025; Zhang, 2026). BabyAI is a BALROG environment (Paglieri et al., 2025), which we run through its AgentGym implementation (Xi et al., 2025).

All three give a single sparse reward at episode end, with episodes of up to 50, 15 and 20 turns (medians 17, 11 and 16), consistent with the 5–20 turns Zhang (2026) reports for ALFWorld. Each turn is a full response, so an episode spans $\approx$1,500–1,800 generated tokens, and the return must be attributed across tens of turns, the regime LHCA targets.

We will add more details on choices of our benchmarks.

P9. Results for GiGPO (Feng et al., 2025) should be reported as baselines as well.

We add GiGPO as a baseline at all three sizes (common reply). LHCA is competitive with it (P(LHCA > GiGPO) = 0.55 and 0.61). Since the two assign credit from different sources (GiGPO groups identical anchor states, LHCA learns from the future), we test whether they capture the same signal: GiGPO+LHCA outperforms GiGPO, P = 0.65 [0.51, 0.79], and has the highest mean in every ALFWorld and Sokoban setting at 3B and 7B. The hindsight signal is therefore complementary to GiGPO's.

P10. 2 seeds are too few to separate LHCA from HCAPO.

We now use 5 seeds at three model sizes. Please see common reply.

P11. When the future statistic is fixed, LHCA does not beat HCAPO on both benchmarks. LHCA comes out ahead only with the statistic chosen per benchmark on reported results ("best future conditioning", l. 327–328), which makes the results look cherry-picked. Why LHCA's best statistic changes is not explained: the text accounts for HCAPO's behaviour (l. 403–404) but leaves LHCA's open ("necessitating more careful design", l. 406–407).

We now report both future statistics for both methods and all comparisons are like-for-like. With 5 seeds, the choice of statistic matters little for LHCA: on ALFWorld its two statistics are within 1.5 points of each other at every size (87.0/86.9, 84.3/85.8, 87.6/87.9), and both outperform HCAPO in aggregate. We therefore report both and drop the per-benchmark choice.

P12. The main text has no related work section: it is in the appendix (A.1), as a list of methods with one line each. Without it, the paper never positions LHCA against the closest work in the main text. I cannot support acceptance if the related work is not in the main text.

We agree and will move related work into the main text with an expanded discussion positioning LHCA against HCAPO, iStar, GiGPO, PRIME and the HCA line of work.

Writing fixes: Thank you, we will fix all four. We will reword the motivation (prompted LLM hindsight is one family among counterfactual-rollout, causal and distance-based methods), remove the placeholder, correct the equation references, and move hindsight to the preliminaries.

Q1. GRPO uses MC returns rather than a learned baseline. GiGPO uses anchor-state grouping to give a state conditioned baseline instead of GRPO's state independent group mean. LHCA adds the most on top of GiGPO at 3B and 7B: GiGPO+LHCA has the highest mean in every ALFWorld and Sokoban cell at those sizes, and P(GiGPO+LHCA (τ>t\tau_{>t}) > GiGPO) = 0.65 [0.51, 0.79]. This supports the reviewer's intuition that hindsight helps more with a better baseline. A learned critic is the natural next step.

Reviewer_ibiCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates hindsight credit assignment (Harutyunyan et al., 2019) in long-horizon RL training of LLM agents. Specifically, it parameterizes the hindsight function with a Low-rank adaptation (LoRA) of the base policy: the LoRA adapter is trained to predict the token sequence generated by the policy, conditioned on outcome events at the beginning of the sequence. With the learned hindsight function, the paper modifies the GRPO update and proposes an LHCA update that uses hindsight weights to reweigh the return of each token. Practically, the hindsight weights are normalized and clipped. Empirically, combining LHCA and the GRPO losses is competitive against HCAPO, a prompting-based hindsight approach, in AFLWorld, BabyAI, and Sokoban, outperforming GRPO and a DPO baseline. A case study shows the learned hindsight function downweighs an inconsequential action against other actions.

Critical strengths

The strength of the paper is that it proposes a novel way to parameterize the hindsight function. Specifically, the paper parameterizes the hindsight function using a LoRA adapter on the base policy, and future/outcome conditioning is implemented by prepending the query with the outcome of the trajectory. This implementation has two potential advantages. First, it does not require training the hindsight function from scratch. Second, if such a parameterization leads to an initial hindsight function that is the same as the current policy, then the proposed update reduces to GRPO (although the unbiased HCA update would reduce to no update in expectation; see also Weakness 1).

Critical weaknesses

Although the proposed approach to model hindsight function is promising, the paper has significant weaknesses in the below aspects:

  1. Unaddressed bias. Different from standard HCA that updates all actions under the policy distribution (see Eq. 8 in Harutyunyan et al., 2019), the paper only updates the sampled action (Eqs. 3 and 10). The correlation between the sampled action and the hindsight-weighted sampled advantage induces a bias, which is not theoretically analyzed nor empirically investigated.

  2. Weak statistical significance. The empirical investigation uses only two seeds, and yet, it draws many conclusions from these results (throughout Section 3.2). In many cases, the results have overlapping standard deviations (e.g., Lines 358-359, 370-371). In some cases, results appear to have only one seed (e.g., Lines 385-386). Either more seeds should be added to establish statistical significance or a justification given why the current experiments and claims are appropriate with just this number of seeds. Note also that standard deviation is not a measure of confidence, and a proper confidence interval should be used.

Questions for authors

The below are the main questions.

  1. Can you derive the update (Eqs. 3 and 10) from Proposition 1? Such a derivation would help identify and discuss the bias introduced (Weakness 1).
  2. What optimizer is used in the experiments? It’s important to disclose this important experimental detail as the optimizer can impact the loss scale/step size when multiple losses are combined.
  3. What’s the impact of trajectory ratio normalization and the do-no-harm mask? For the latter, would GRPO also perform better with the mask?
  4. Can you elaborate on the three step progression of hindsight calculation (Line 367)? Specifically, how is hindsight calculated in (a) and (b)? Is it the same as HCAPO?
  5. In Table 1, is it fair to compare LHCA with the best future conditioning against HCAPO with each different conditioning? If not, then the table should be corrected.

Other questions or issues that have little impact on the rating:

  1. The temperatures used in Tables 1-4 should be stated clearly in the main text. Is the temperature reported in the Appendix A.4 also used for HCAPO?
  2. Line 107, yt,iy_t,i is not defined. It’s also unclear if ∣at∣|a_t| fixed or variable across tt.
  3. Line 141, it’s unclear what “both” is pointing to.
  4. The base models for each experiment should be made clearer (e.g., Tables 3-4, Figure 1).
  5. Line 169, Proposition 1 is the degenerate case of Theorem 1 instead of Theorem 2 from Harutyunyan et al. (2019).
6 comments
AuthorsOpenReview ↗

Rebuttal Response (Part 1)

We thank the reviewer for a detailed and technically careful review and for recognizing the LoRA parameterization of the hindsight function as novel. We address both main concerns: we characterize the bias of the sampled-action estimator precisely and show that it preserves the gradient direction up to a bounded second-order term (P1, Q1), and we expand the evaluation to 5 seeds, three model sizes and new baselines, reported with confidence intervals and aggregate significance tests (P2 and the common reply). We will upload the revised PDF, with all changes highlighted, shortly and will notify you when it is available.

P1. Unaddressed Bias

We thank the reviewer for this observation, which led us to a precise characterization of the estimator.

Proposition 1 restates the hindsight identity of Harutyunyan et al. (2019), Q(st,a)=EF∼P(F∣st)[ρ(a,F)G(F)]Q(s_t,a) = \mathbb{E}_{F\sim \mathbb{P}(F\mid s_t)}[\rho(a,F)G(F)] with ρ(a,F)=h(a∣st,F)/πθ(a∣st)\rho(a,F) = h(a\mid s_t,F)/\pi_\theta(a\mid s_t). An unbiased estimate of this term needs either several rollouts from each turn state or scoring every candidate action against each sampled future. Both multiply the cost per rollout, and the second option queries the hindsight model on (action, future) pairs it never sees in training. LHCA instead scores the sampled action against its own future, which needs one hindsight evaluation per turn.

Proposition (Sampled-action hindsight estimate, proof and gradient derivation in App. A.3). The sampled-action estimator targets Q~(a)=EF∼p[ρ2(a,F)G]\widetilde{Q}(a) = \mathbb{E}_{F\sim p}[\rho^2(a,F)G] instead of Q(a)=EF∼p[ρ(a,F)G]Q(a) = \mathbb{E}_{F\sim p}[\rho(a,F)G]. Writing ρ2=1+2(ρ−1)+(ρ−1)2\rho^2 = 1 + 2(\rho-1) + (\rho-1)^2 gives the exact decomposition

Q~(a)−V=2(Q(a)−V)+EF∼p[(ρ(a,F)−1)2 G],\widetilde{Q}(a) - V = 2\big(Q(a) - V\big) + \mathbb{E}_{F\sim p}\big[(\rho(a,F)-1)^2\,G\big],

where the remainder is bounded by Gmax⁡ χ2(P(F∣a) ∥ P(F))G_{\max}\,\chi^2\big(\mathbb{P}(F\mid a)\,\|\,\mathbb{P}(F)\big). Relative to the state value, the estimator's advantage is the true advantage scaled by 2, plus a term that is second order in ρ−1\rho-1; group standardization removes the factor 2. The expected update is

∑a∇πθ(a∣s)Q~(a)=2∇J+∑a∇πθ(a∣s)EF∼p[(ρ−1)2G],\sum_a \nabla\pi_\theta(a\mid s)\widetilde{Q}(a) = 2\nabla J + \sum_a \nabla\pi_\theta(a\mid s)\mathbb{E}_{F\sim p}\big[(\rho-1)^2G\big],

so LHCA follows the true policy gradient up to scale and a bounded second-order term. With ρ≡1\rho \equiv 1 the remainder vanishes and the update reduces exactly to REINFORCE/GRPO, as the reviewer notes.

Empirically we find that ρ\rho stays close to 1, so the remainder is small and the clip to [0.6, 1.4] is active on at most 0.02% of turns per step.

The learned ρ\rho nonetheless carries useful credit: shuffling it across the turns of each trajectory costs 6.8 points on ALFWorld and 5.2 on Sokoban, and removing it costs 6.7 points on Sokoban (common reply).

The sampled-action estimator is a deliberate design choice: it keeps the cost to one hindsight evaluation per turn, with a bias that is now characterized and bounded. We will add the proposition and derivation to the paper (App. A.3).

P2. Weak statistical significance.

All main results now use 5 seeds per method at three model sizes (1.5B, 3B, 7B) on ALFWorld and Sokoban, and 3 seeds at 7B on BabyAI. We report 95% confidence intervals instead of standard deviations, together with the aggregate probability of improvement (Agarwal et al., 2021).

  • LHCA significantly outperforms GRPO, PRIME and iStar (P(improvement) 0.81–0.99).
  • With the same future statistic, LHCA outperforms HCAPO: 0.68 [0.56, 0.79] with oTo_T and 0.70 [0.58, 0.82] with τ>t\tau_{>t}, both CIs above 0.5.
  • LHCA is complementary to GiGPO; GiGPO+LHCA (τ>t\tau_{>t}) improves over both GiGPO and LHCA (P = 0.65 and 0.67, CIs above 0.5).

We will revise all our results and claims accordingly.

Q1. Can you derive the update (Eqs. 3 and 10) from Proposition 1? Such a derivation would help identify and discuss the bias introduced (Weakness 1).

Yes, App. A.3 derives the expected LHCA update from the Proposition above for the sampled action estimator. The result, summarized in P1, is twice the actual gradient plus a remainder that is second order in ρ−1\rho-1 and bounded by Gmax⁡ χ2(P(F∣a) ∥ P(F))G_{\max}\,\chi^2\big(\mathbb{P}(F\mid a)\,\|\,\mathbb{P}(F)\big).

Q2. What optimizer is used in the experiments? It's important to disclose this important experimental detail as the optimizer can impact the loss scale/step size when multiple losses are combined.

We use AdamW throughout. The policy is trained with lr 1e-6, betas (0.9, 0.999), weight decay 0.01 and a constant schedule, as in prior work. The hindsight LoRA adapter (rank 32, α\alpha = 64) has its own AdamW optimizer (lr 2e-5, 1e-5 at 7B, weight decay 0.01) and is trained only on its reconstruction loss. The two losses never interact. We will add these details to the paper.

AuthorsOpenReview ↗

Rebuttal Response (Part 2)

Q3. What's the impact of trajectory ratio normalization and the do-no-harm mask? For the latter, would GRPO also perform better with the mask?

We ablate both these settings at 1.5B (full discussion in the common reply):

  • Trajectory normalization. Removing it costs 3.0 points on ALFWorld and 5.8 on Sokoban. Without it ρ\rho can carry a trajectory-level offset (for example, the adapter assigning uniformly higher likelihood to every action of some trajectories), which acts as noise on the return rather than as credit across turns. Dividing by the trajectory mean removes this offset, so ρ\rho only redistributes credit within a trajectory.
  • Do-no-harm prevents the hindsight term from penalizing actions in successful trajectories. It is a small safeguard, not integral to the gains (removing it changes results by at most 1.5 points).
  • GRPO with the mask. For GRPO with a binary outcome reward, a successful trajectory will always have its reward higher than the mean, therefore never a negative group-normalized advantage, so the mask is a no-op by construction. It only acts when the turn-level (discounted, ρ\rho-weighted) advantage pushes some turns of a successful trajectory below the group mean.

Q4. Can you elaborate on the three step progression of hindsight calculation (Line 367)? Specifically, how is hindsight calculated in (a) and (b)? Is it the same as HCAPO?

In our submission, Figure 1 compared three sources of the hindsight score hh: (a) prompting the frozen base policy (b) prompting the current actor, which is exactly HCAPO (c) a LoRA adapter trained online to reconstruct the action given the future (LHCA)

(a) and (b) use the same prompt and scoring as HCAPO and differ only in which model scores. The motivation for (c) is that (b) assumes an actor that learns to act also learns to judge its own actions in hindsight, which need not hold. In the revision, this comparison is made directly in the main multi-seed results: using the same future statistic, LHCA outperforms HCAPO in aggregate (P(improvement) 0.68 and 0.70, CIs above 0.5), and these results replace Figure 1.

Q5. In Table 1, is it fair to compare LHCA with the best future conditioning against HCAPO with each different conditioning? If not, then the table should be corrected.

We agree that the like-for-like comparison is the right one. The revised tables report HCAPO and LHCA with both future statistics (oTo_T and τ>t\tau_{>t}) and compare like with like. LHCA outperforms HCAPO under either statistic: P(LHCA > HCAPO) = 0.68 [0.56, 0.79] with oTo_T and 0.70 [0.58, 0.82] with τ>t\tau_{>t}.

Q6. The temperatures used in Tables 1-4 should be stated clearly in the main text. Is the temperature reported in the Appendix A.4 also used for HCAPO?

No. Each method has its own temperature TtempT_{\text{temp}}, selected by the same sweep. For each method, environment and future statistic, TtempT_{\text{temp}} was selected by a sweep over Ttemp∈{1,3,5}T_{\text{temp}} \in \{1,3,5\} at 1.5B and 3B and the 7B value was taken as the best TtempT_{\text{temp}} averaged over the 1.5B and 3B sweeps.

MethodFutureALFWorld 1.5B / 3B / 7BSokoban 1.5B / 3B / 7BBabyAI 7B
HCAPOoTo_T5 / 3 / 31 / 5 / 53
HCAPOτ>t\tau_{>t}5 / 3 / 31 / 5 / 53
LHCAoTo_T5 / 5 / 55 / 3 / 33
LHCAτ>t\tau_{>t}3 / 5 / 53 / 1 / 33
AuthorsOpenReview ↗

Rebuttal Response (Part 3)

Q7. Line 107, yi,ty_{i,t} is not defined. It's also unclear if ∣at∣|a_t| fixed or variable across tt.

We will define yi,ty_{i,t} as the tokens of action ata_t, and state that ∣at∣|a_t| is the action length in tokens, which varies with tt.

Q8. Line 141, it's unclear what "both" is pointing to.

'Both' refers to the two uses of PRMs: (a) guiding long generations and (b) assigning credit to intermediate actions during RL training. We will rewrite the sentence.

Q9. The base models for each experiment should be made clearer (e.g., Tables 3-4, Figure 1). We will state the base model in every table and figure caption. The main results now cover Qwen2.5-{1.5B, 3B, 7B}-Instruct, and the ablations use 1.5B.

Q10. Line 169, Proposition 1 is the degenerate case of Theorem 1 instead of Theorem 2 from Harutyunyan et al. (2019).

Thank you. Proposition 1 conditions on a future statistic that determines the return. It can be read as the return-conditional form (Theorem 2) with the return replaced by such a statistic, or, as the reviewer notes, as a degenerate case of the state-conditional Theorem 1. We will cite Theorem 1 and explain the reduction.

References:

  • Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems 34, 2021.
  • Harutyunyan, A., Dabney, W., Mesnard, T., Gheshlaghi Azar, M., Piot, B., Heess, N., van Hasselt, H., Wayne, G., Singh, S., Precup, D., and Munos, R. Hindsight credit assignment. In Advances in Neural Information Processing Systems 32, 2019.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Common Response (Part 1)

We thank the reviewers for their careful reading and useful feedback. We are glad that reviewers found the idea of learning hindsight distribution directly "very promising" and "a natural way to bring hindsight to LLM agents without prompting" (5kvo) and the LoRA parameterization of the hindsight function novel (ibiC).

The shared concerns were statistical significance of results [ibiC, 5kvo, tkKb], baselines and same conditioning comparisons [ibiC Q5, 5kvo], the empirical considerations taken to implement the estimator drifting from HCA theory [ibiC, 5kvo], and training cost [5kvo, tkKb]. We re-ran the same experiments with the same method, hyperparameters and protocol using 5 seeds instead of 2 and add the baselines, ablations and analysis the reviewers requested. We will upload the revised PDF, with all changes highlighted, shortly and will notify you when it is available.

Summary of Changes

  • We re-train our baselines and methods using 5 seeds instead of 2 (3 on BabyAI), on three model sizes (Qwen2.5-1.5B/3B/7B) [ibiC, 5kvo, tkKb]
  • HCAPO and LHCA reported with both future statistics, compared like-for-like [ibiC Q5, 5kvo]
  • GiGPO as a baseline [5kvo P9]
  • 95% confidence intervals and the probability of improvement (Agarwal et al., 2021) [ibiC, tkKb]
  • Leave-one-out ablations of every component [ibiC Q3, 5kvo]
  • Training-time measurements [5kvo, tkKb]
  • A characterization of the bias of the sampled-action estimator (App. A.3) [ibiC]
  • Related work moved to the main text [5kvo, tkKb]

Main Findings. With 5 training seeds, three model sizes and the baselines reviewers request, the conclusions of our submission hold, now with statistical significance. LHCA significantly outperforms GRPO and the implicit-PRM baselines (iStar and PRIME), and outperforms HCAPO in aggregate under the same future statistic. On the newly requested GiGPO baseline [5kvo P9], LHCA is competitive. To test whether the two capture the same credit, we combine them: GiGPO+LHCA achieves the best mean result in every ALFWorld and Sokoban setting at 3B and 7B, so the two credit signals are complementary.

Additional Seeds and Baselines

Experiment groupMethodsVariantsALFWorldSokobanBabyAIRuns
Main comparisonGRPO, iStar, PRIME, GiGPO, HCAPO (oTo_T, τ>t\tau_{>t}), LHCA (oTo_T, τ>t\tau_{>t})83 sizes ×\times 5 seeds3 sizes ×\times 5 seeds7B ×\times 3 seeds264
CombinationGiGPO+LHCA (oTo_T, τ>t\tau_{>t})23 sizes ×\times 5 seeds3 sizes ×\times 5 seedsnot run60
Total324

To address statistical significance [ibiC, 5kvo, tkKb], we re-trained every method with more training seeds (table above). Every method is trained on Qwen2.5-{1.5B, 3B, 7B}-Instruct with 5 seeds on ALFWorld, Sokoban and at 7B with 3 seeds on BabyAI. We add GiGPO (Feng et al., 2025) [5kvo P9] and PRIME (Cui et al., 2025) [5kvo P4], the implicit PRM method that iStar builds on, as baselines, and the combination GiGPO+LHCA on ALFWorld and Sokoban (60 runs). HCAPO and LHCA are both reported with both future statistics, oTo_T and τ>t\tau_{>t} [ibiC Q5, 5kvo]. In total this is 324 runs on 8-GPU A100-40GB or 4-GPU H100-80GB nodes. All numbers are from the step-150 checkpoint; ALFWorld numbers are on the out-of-distribution (unseen tasks) split.

We report results in two complementary ways: per cell means with 95% confidence intervals and the aggregate probability of improvement (Agarwal et al., 2021), which pools evidence across settings and is the recommended analysis for RL with few seeds per task.

Differences between methods are small relative to seed variance, because:

  • All methods optimize the same objective from the same outcome reward and differ in how a trajectory's credit is distributed across its turns, so differences between methods show up as consistent shifts across seeds and settings rather than as large single-run gaps.
  • Multi-turn rollouts compound environment, sampling and task-order stochasticity, and small differences early in training propagate to the final policy.
  • Evaluation itself is noisy: re-evaluating one fixed ALFWorld checkpoint with 5 sampling seeds gives a std of 2.4 points [tkKb Q1]

For these reasons, per-setting tests with 5 seeds have limited power for any method pair, and the aggregate statistic below is the more informative comparison.

AuthorsOpenReview ↗

Common Response (Part 2)

Mean with 95% CI

Mean over nn seeds, with the 95% t-interval xˉ±t0.975, n−1 s/n\bar{x} \pm t_{0.975,\,n-1}\, s/\sqrt{n} in brackets.

ALFWorld, Sokoban: n=5, t = 2.776, so the half-width is 1.24 ×\times the seed std. BabyAI: n=3, t = 4.303, so the half-width is 2.48 ×\times the seed std.

Intervals are clipped to [0, 100] and we remove one collapsed iStar 3B seed from CI computation.

ALFWorld out-of-dist (success rate, %)

Method1.5B: mean [95% CI]3B: mean [95% CI]7B: mean [95% CI]
GRPO66.5 [58.6, 74.4]73.1 [69.1, 77.2]80.1 [75.4, 84.9]
GiGPO84.0 [74.6, 93.5]86.9 [83.1, 90.6]83.9 [72.6, 95.1]
PRIME26.1 [13.9, 38.4]46.3 [44.8, 47.8]65.1 [53.9, 76.2]
iStar1.8 [0.0, 5.1]26.3 [18.0, 34.6]38.1 [26.8, 49.3]
HCAPO (oTo_T)88.0 [84.0, 92.1]84.6 [82.4, 86.9]84.6 [80.2, 89.1]
HCAPO (τ>t\tau_{>t})86.6 [81.9, 91.3]87.2 [83.2, 91.1]87.2 [82.7, 91.7]
LHCA (oTo_T)87.0 [79.7, 94.3]84.3 [82.5, 86.2]87.6 [84.0, 91.3]
LHCA (τ>t\tau_{>t})86.9 [82.2, 91.5]85.8 [81.8, 89.9]87.9 [85.5, 90.3]
GiGPO+LHCA (oTo_T)84.3 [80.3, 88.3]91.8 [90.0, 93.5]88.8 [84.3, 93.3]
GiGPO+LHCA (τ>t\tau_{>t})87.6 [84.2, 91.0]88.8 [85.7, 91.9]90.6 [88.4, 92.8]

Sokoban and BabyAI (success rate, %)

MethodSokoban 1.5B: mean [95% CI]Sokoban 3B: mean [95% CI]Sokoban 7B: mean [95% CI]BabyAI 7B: mean [95% CI]
GRPO41.2 [37.5, 44.9]33.8 [15.5, 52.0]55.9 [34.4, 77.5]34.3 [20.7, 48.0]
GiGPO43.0 [38.6, 47.5]44.0 [38.6, 49.4]57.2 [31.1, 83.4]34.7 [17.1, 52.2]
PRIME26.0 [11.6, 40.4]38.9 [31.0, 46.8]44.6 [20.7, 68.5]28.5 [27.3, 29.7]
iStar29.1 [25.7, 32.5]28.8 [13.7, 43.9]33.6 [22.2, 45.0]3.7 [0.0, 19.4]
HCAPO (oTo_T)45.1 [38.2, 52.1]38.9 [32.3, 45.4]32.3 [10.5, 54.1]35.5 [30.5, 40.5]
HCAPO (τ>t\tau_{>t})42.1 [36.6, 47.6]41.8 [37.8, 45.9]38.5 [25.0, 52.1]33.2 [23.5, 42.8]
LHCA (oTo_T)42.7 [33.9, 51.5]46.1 [40.5, 51.8]52.7 [38.3, 67.0]40.3 [35.6, 45.0]
LHCA (τ>t\tau_{>t})47.8 [38.8, 56.8]45.0 [31.7, 58.2]50.0 [41.6, 58.3]39.5 [34.5, 44.5]
GiGPO+LHCA (oTo_T)44.5 [39.2, 49.8]46.7 [37.3, 56.1]63.5 [54.7, 72.2]–
GiGPO+LHCA (τ>t\tau_{>t})42.7 [36.6, 48.8]53.5 [43.1, 63.9]62.5 [52.3, 72.7]–

We find that on ALFWorld, LHCA's 95% CI lies entirely above GRPO's at 1.5B and 3B (and at 7B for τ>t\tau_{>t}), and above PRIME's and iStar's at every size. Moreover, LHCA has the highest mean among the standalone methods on Sokoban 1.5B (τ>t\tau_{>t}), on Sokoban 3B (oTo_T), and on BabyAI 7B with both future statistics. Finally, GiGPO+LHCA has the highest mean in every ALFWorld and Sokoban cell at 3B and 7B.

Probability of improvement (Agarwal et al., 2021)

P(A > B) is the probability that a randomly chosen seed of A outperforms a randomly chosen seed of B (ties taken as 0.5). We compute it in each cell and average over all the cells presented above. The 95% CI is a stratified bootstrap (seeds resampled within each cell, 20,000 resamples). A CI entirely above 0.5 means A is better than B in aggregate.

A vs BP(A > B) [95% CI]
LHCA (oTo_T) vs GRPO0.81 [0.71, 0.90]
LHCA (oTo_T) vs GiGPO0.55 [0.42, 0.68]
LHCA (oTo_T) vs PRIME0.90 [0.82, 0.97]
LHCA (oTo_T) vs iStar0.99 [0.95, 1.00]
LHCA (oTo_T) vs HCAPO (oTo_T)0.68 [0.56, 0.79]
LHCA (τ>t\tau_{>t}) vs GRPO0.82 [0.72, 0.91]
LHCA (τ>t\tau_{>t}) vs GiGPO0.61 [0.46, 0.75]
LHCA (τ>t\tau_{>t}) vs PRIME0.88 [0.79, 0.95]
LHCA (τ>t\tau_{>t}) vs iStar0.97 [0.92, 1.00]
LHCA (τ>t\tau_{>t}) vs HCAPO (τ>t\tau_{>t})0.70 [0.58, 0.82]
GiGPO+LHCA (oTo_T) vs GiGPO0.60 [0.44, 0.75]
GiGPO+LHCA (oTo_T) vs LHCA (oTo_T)0.64 [0.50, 0.78]
GiGPO+LHCA (τ>t\tau_{>t}) vs GiGPO0.65 [0.51, 0.79]
GiGPO+LHCA (τ>t\tau_{>t}) vs LHCA (τ>t\tau_{>t})0.67 [0.53, 0.79]
  • We find that LHCA outperforms GRPO, PRIME and iStar with both future statistics (P = 0.81–0.99, all CIs well above 0.5).
  • With the same future statistic, LHCA outperforms HCAPO: P = 0.68 [0.56, 0.79] with oTo_T and 0.70 [0.58, 0.82] with τ>t\tau_{>t}, both CIs above 0.5 [ibiC Q5, 5kvo, tkKb].
  • LHCA is competitive with GiGPO (P = 0.55 and 0.61) and complementary to it: GiGPO+LHCA (τ>t\tau_{>t}) outperforms both GiGPO (0.65 [0.51, 0.79]) and LHCA (0.67 [0.53, 0.79]).
AuthorsOpenReview ↗

Common Response (Part 3)

Ablations on LHCA

Table: Component ablations with Qwen2.5-1.5B-Instruct.

Requested by ibiC (Q3) and 5kvo (P1, P3), we ablate each component of LHCA in turn.

AblationALFWorld OOD SRΔ\DeltaSokoban SRΔ\Delta
LHCA88.1 ±\pm 2.949.4 ±\pm 3.5
 h_ratio = 187.3 ±\pm 0.0↓0.842.7 ±\pm 5.1↓6.7
 Shuffled h_ratio81.3 ±\pm 2.9↓6.844.2 ±\pm 1.9↓5.2
 No macro (M)88.8 ±\pm 3.0↑0.737.2 ±\pm 1.9↓12.2
 No do-no-harm (H)86.6 ±\pm 1.5↓1.548.0 ±\pm 0.4↓1.4
 γ=1\gamma = 1 (G)72.4 ±\pm 0.0↓15.741.9 ±\pm 0.6↓7.5
 No traj-norm (N)85.1 ±\pm 3.7↓3.043.6 ±\pm 1.9↓5.8
 M+H+G65.7 ±\pm 2.2↓22.430.9 ±\pm 0.1↓18.5
 M+H+G+N67.9 ±\pm 5.2↓20.244.1 ±\pm 2.0↓5.3
 Theory-closest70.9 ±\pm 1.5↓17.238.0 ±\pm 0.3↓11.4
Vanilla GRPO64.9 ±\pm 5.2↓23.241.4 ±\pm 2.3↓8.0

We run each ablation for 3 seeds and present median ±\pm MAD for all methods as building confidence intervals with 3 seeds is very noisy. Δ\Delta is the change from full LHCA. LHCA and GRPO results use 5-seeds, as in the main tables. Qwen2.5-1.5B-Instruct; ALFWorld out-of-distribution with oTo_T at Ttemp=5T_{\text{temp}} = 5, Sokoban with τ>t\tau_{>t} at Ttemp=3T_{\text{temp}} = 3. M = no macro, H = no do-no-harm, G = γ=1\gamma = 1, N = no traj-norm; combined ablations are written as letter sums. Theory-closest is M+H+G+N at Ttemp=1T_{\text{temp}} = 1, while the M+H+G+N row uses the base config's temperature.

  • The learned hindsight ratio carries real credit. [5kvo P1, ibiC P1] Shuffling ρ\rho across the turns of each trajectory (same values, assigned to the wrong actions) costs 6.8 points on ALFWorld and 5.2 on Sokoban, and removing it (ρ=1\rho = 1) costs 6.7 points on Sokoban, where performance falls to the GRPO level (42.7 vs 41.4).
  • Turn-level discounting [5kvo P3] is essential: γ=1\gamma = 1 costs 15.7 points on ALFWorld and 7.5 on Sokoban. This agrees with GiGPO (Feng et al., 2025) which uses discounted step-level returns (γ=0.95\gamma = 0.95) so that earlier, less useful actions receive less credit than later decisive ones.
  • Trajectory normalization removes trajectory-level offsets in ρ\rho (−3.0 on ALFWorld and −5.8 on Sokoban without it) [ibiC Q3, 5kvo P1]. Do-no-harm is a safeguard with a small effect (within 1.5 points). [ibiC Q3, 5kvo P1]
  • The variant closest to the theoretical estimator [5kvo P1] (Theory-closest: no macro, no do-no-harm, γ=1\gamma = 1, no trajectory normalization, Ttemp=1T_{\text{temp}} = 1) loses 17.2 points on ALFWorld and 11.4 on Sokoban. The components are what make hindsight credit assignment trainable at LLM scale, as the raw hindsight values can have low signal to noise ratio.
  • The GRPO macro term [5kvo P1] keeps the outcome signal in every update and stabilizes training on Sokoban (−12.2 without it).
AuthorsOpenReview ↗

Common Response (Part 4)

Timing Analysis

We compare the training step time of LHCA and HCAPO, the closest baseline in cost [5kvo P7, tkKb Q2] (1.5B on 4$\times$H100, 3B and 7B on 8$\times$A100). The two methods differ only in the credit pass: generation, the actor update and the remaining costs are the same, and generation dominates the step (77–88% of HCAPO's step on ALFWorld). HCAPO scores with one no-grad forward pass of the policy, while LHCA updates its LoRA adapter (forward + backward) and then scores. With an efficient credit pass (adapter on the policy's own backbone, packed sequences, scoring after the adapter update), LHCA with oTo_T is within 6–19% of HCAPO's step time. Our reported step times use, for each method, the faster of its training and efficient credit pass, measured in the same setup and substituted into our training runs; all other components are as measured.

SettingHCAPO step (s) (oTo_T / τ>t\tau_{>t})LHCA step (s) (oTo_T / τ>t\tau_{>t})LHCA ÷\div HCAPO (oTo_T / τ>t\tau_{>t})
ALFWorld 1.5B236 / 242251 / 3331.06 / 1.37
ALFWorld 3B426 / 473458 / 6161.07 / 1.30
ALFWorld 7B682 / 694737 / 9141.08 / 1.32
Sokoban 3B388 / 394462 / 7741.19 / 1.97
Sokoban 7B779 / 795878 / 13151.13 / 1.65

The step times above use the efficient credit pass as our training runs used a slower research implementation of the same computation. Integrating FSDP sharded actor with switchable LoRA with own optimizer along with torch.compile, sequence packing, logit chunking into existing RL codebases involves non-trivial engineering, which we consider out of scope. We will describe the path to actual implementation in our final version.

LHCA's additional overhead over HCAPO buys a measurable gain: under the same future statistic, LHCA outperforms HCAPO in aggregate (P = 0.68 with oTo_T and 0.70 with τ>t\tau_{>t}). Overall, HCAPO relies on the base model judging zero-shot how well an action explains the observed future, which cannot improve with experience. LHCA instead learns the hindsight distribution from the agent's own trajectories, so its credit signal improves with training data rather than being fixed by a prompt. This follows the broader lesson that learned methods which scale with data and compute eventually outperform handcrafted ones (Sutton, 2019).

References

  • Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems 34, 2021.
  • Cui, G., Yuan, L., Wang, Z., et al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456, 2025.
  • Feng, L., Xue, Z., Liu, T., and An, B. Group-in-group policy optimization for LLM agent training. In Advances in Neural Information Processing Systems 38, 2025.
  • Sutton, R. S. The bitter lesson. Incomplete Ideas (blog), 2019.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.