acceptodds
Under review as a conference paper at ICLR 2027

Hindsight Is 20/20: Adaptive Credit Assignment via Realized Futures for Long-Horizon LLM Agents

Abstract

Group-based reinforcement learning (RL) has emerged as an effective critic-free paradigm for training large language model (LLMs) and has recently been extended to long-horizon agentic tasks. However, sparse outcome rewards provide only weak supervision for intermediate decisions, making it difficult to determine which actions are beneficial or detrimental along successful and failed trajectories. Existing step-level group-based methods alleviate this issue through finer-grained intra-group comparisons, but their reliability depends critically on whether grouped actions are genuinely comparable and whether each group provides sufficient support for estimating expected returns. To address these limitations, we propose Hindsight-Enabled Retrospective Policy Optimization (HeRPO), a critic-free framework that elevates realized futures from return signals to posterior context for credit assignment. HeRPO extracts hindsight information from realized futures to construct hindsight-pruned grouping strategy and adaptively adjusts the balancing coefficient in generalized advantage estimation according to the effective sample support of each group. Moreover, for each executed action, HeRPO evaluates the policy both with and without privileged hindsight and converts the resulting shift in action preference into a bounded weight applied to its advantage. By jointly improving the quality of step-level credit signals and controlling the magnitude of policy updates, HeRPO enables more precise optimization for long-horizon agentic tasks. Extensive experiments on the agentic benchmarks demonstrate that HeRPO consistently outperforms other competing agentic reinforcement learning methods. Codes will be available on the GitHub.

Primary area:
reinforcement learning
Keywords:
Large Language Model, Agentic Reinforcement Learning, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mHQbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HeRPO, a hindsight-based framework for step-level credit assignment in long-horizon agentic RL. The method introduces three components: hindsight-pruned grouping to refine step-level comparisons, adaptive GAE based on effective group support, and hindsight self-reweighting to adjust policy updates. Experiments on several agentic benchmarks demonstrate improvements over existing RL methods.

Critical strengths
  1. Strong empirical performance. The proposed method achieves consistent improvements over competitive baselines across multiple agentic benchmarks.

  2. Reasonable motivation. The paper identifies meaningful limitations in existing step-level credit assignment methods and proposes a systematic framework to address them.

Critical weaknesses
  1. Concerns about the reliability of hindsight-based credit assignment. The proposed group filtering removes comparisons between identical actions with different realized futures. However, differences in future observations may arise from environmental stochasticity rather than the actions themselves. For example, in GUI environments, opening the same webpage may lead to different observations due to dynamic advertisements or other external factors. The proposed filtering strategy does not appear to distinguish such environmental variations from action-induced differences, potentially limiting its reliability and generalizability.

  2. Limited methodological novelty. Compared with existing step-level credit assignment methods, the proposed framework primarily refines grouping, advantage estimation, and the weighting of policy updates. In particular, HSR appears conceptually related to hindsight-conditioned policy evaluation approaches such as OPSD. The paper does not sufficiently establish the fundamental differences or advantages of its proposed mechanism over related approaches, making the overall methodological contribution somewhat incremental.

Questions for authors
  1. Environmental stochasticity in hindsight-pruned grouping. How does the proposed grouping strategy distinguish action-induced differences from environmental stochasticity? For example, in GUI environments, identical actions may produce different observations due to dynamic webpage content. Would such variations cause potentially useful comparisons to be incorrectly removed?

  2. Relationship between HSR and OPSD. HSR evaluates executed actions with and without hindsight information and uses the resulting likelihood difference to adjust policy updates. This appears conceptually related to OPSD-style policy distillation. Could the authors clarify the key differences and provide evidence that the proposed preference shift is a reliable indicator of action quality?

  3. Methodological novelty. Compared with existing methods such as GiGPO, GraphGPO, and HCAPO, the main modifications appear to involve more refined grouping and advantage weighting. What is the fundamental methodological contribution beyond these refinements?

  4. Role and definition of hindsight. The paper emphasizes hindsight as a central contribution. However, existing RL methods with verifiable rewards also retrospectively assign credit based on realized outcomes. What fundamentally distinguishes the proposed hindsight mechanism from conventional outcome-based credit assignment? A clearer conceptual distinction would strengthen the motivation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer mHQb (Part I)

Thank you for the detailed review. We appreciate your recognition of the empirical results and agree that the paper needs a clearer description of hindsight reliability and its relationship to prior hindsight-conditioned learning. Below, we provide a detailed, point-by-point response to each listed weakness and question. For clarity, "W" denotes a numbered Critical Weakness and "Q" denotes a numbered Question for Authors from your review. When a weakness and a question address the same issue, we combine their labels (e.g., "W1&Q1").

W1&Q1: HPG reliability.

Thank you for this valuable concern. We agree that a change in observation does not necessarily reflect a different effect of the current action. HPG focuses step-level credit assignment on comparisons between different actions taken at the same state. For samples with the same state and action, it removes their comparison link when their realized hindsight differs, since a difference in return may arise from later decisions or environmental variation and should not be attributed to a difference in the current action.

When the outcome or subsequent actions genuinely differ, the sample links should be pruned regardless of observation noise. The potential for false pruning arises when two such samples have the same hindsight but superficial noise exists. In our experimental analysis, samples sharing the same state, action, and hindsight account for only approximately 0.1% of all samples due to randomness. Considering the diversity of real environments and potential noise issues, we constructed a noise-tolerant HPG variant that compares key hindsight fields, including the outcome reward and subsequent actions, separately, while applying a similarity threshold of 0.95 to environmental observations. In the evaluated mild-noise cases, the variant reduced the false-pruning rate to 0% and preserved the original pruning decisions for other pairs.

Our HPG estimates a group value from the retained samples' reward returns, and AGAE uses that value to compute step-level advantages. In the ablation study, adding HPG improves ALFWorld overall success from 86.7% to 90.6% and WebShop success from 65.0% to 70.6% (Table 3). We will elaborate the design rationale of HPG in the revised manuscript.

Q2: Relationship to OPSD.

Thank you for highlighting this connection. While both OPSD and HSR leverage privileged hindsight information to reassess a policy's own sampled outputs, they differ fundamentally in how hindsight guides policy optimization. OPSD directly distills the privileged teacher's distribution into the student through token-level distillation gradients. In contrast, HSR quantifies the impact of hindsight by aggregating the old policy's log-likelihood differences with and without hindsight over an executed action (Eq. 9). This signal is then transformed into a detached, bounded weight that adaptively scales the action's original advantage (Eqs. 10–11). Crucially, HSR introduces no additional distillation objective and preserves the sign of the original advantage, modulating the strength of advantage-based policy updates rather than modifying the optimization direction.

To further substantiate this distinction, we evaluated GiGPO+OPSD under the same GiGPO training setting. GiGPO+OPSD achieved 87.5±1.3% on ALFWorld and 63.4±2.9% on WebShop, compared with 89.1±1.1% and 68.0±2.1% for GiGPO+HSR, respectively. These results highlight two key methodological differences. (1) HSR operates at the executed-action level, aligning hindsight feedback with the granularity of agent decisions, whereas OPSD's token-level distillation can produce heterogeneous guidance across tokens within the same action. (2) Optimization alignment: OPSD's distillation gradients may conflict with advantage-based policy gradients, while HSR applies bounded, positive reweighting directly to the advantages, preserving their optimization directions. Together, these distinctions explain how HSR integrates hindsight information more effectively into advantage-based policy optimization for agentic tasks. The consistent improvements across both benchmarks further demonstrate the effectiveness of action-level granularity and direction-preserving reweighting. We will incorporate the GiGPO+OPSD results and clarify these distinctions in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer mHQb (Part II)

W2&Q3: Methodological novelty.

Thank you for the opportunity to clarify the novelty. While GiGPO uses state-aligned groups, GraphGPO estimates credit through state-transition graphs, and HCAPO uses hindsight self-scoring to construct step advantages, HeRPO integrates adaptive GAE into critic-free, group-based agentic RL, using group-derived values to propagate credit across action steps. It also uses realized futures as privileged information to refine action comparisons, adapt temporal credit propagation, and calibrate advantage magnitude.

Specifically, HPG removes comparison links between samples that share the same state and action but have inconsistent hindsight. Differences in their subsequent returns may reflect later decisions or environmental variation rather than a difference in the current action. Eq. (4) estimates values from the retained reward returns. AGAE uses these group-derived values to estimate advantage and adjusts its coefficient adaptively according to the effective support of each retained group. Limited support favors longer-horizon Monta Carlo returns, while broader support favors local TD estimates. HSR measures how the realized future changes the old policy's preference for the executed action and converts that shift into a bounded weight on the advantage. This calibrates update magnitude without changing the update direction.

The ablations support the contribution of each component. AGAE yields the largest gains on both overall metrics (ALFWorld: 86.7% → 93.0%; WebShop success: 65.0% → 74.2%), and combining all three components reaches 95.8% and 80.7%, respectively (Table 3). HSR also improves the overall metrics of GRPO, GiGPO, and GraphGPO on both benchmarks (Table 4), supporting its extensibility beyond HeRPO. We will revise the introduction to state these contributions and distinctions more clearly.

Q4: Role and definition of hindsight.

We appreciate the request to define hindsight and explain its role in HeRPO. In HeRPO, hindsight refers to privileged future information available after an action is executed, including the terminal outcome, the immediate next observation, and up to K=2K=2 subsequent action–observation pairs (see Eq. 2 and Appendix B.2). Unlike conventional outcome-based RL, where the terminal reward provides a scalar trajectory-level signal for advantage estimation and determines the policy update direction, HeRPO leverages hindsight for two complementary purposes: (1) refining group-based advantage estimation through HPG, and (2) adaptively modulating update magnitudes for distinct actions through HSR. Hindsight does not introduce an additional reward signal or independently determine the policy update direction, it insteadly enables more precise step-level credit assignment.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns, particularly the clarification of the HPG mechanism and the additional comparison with OPSD.

However, I believe some of my main concerns remain only partially addressed. First, while the noise-tolerant HPG variant helps mitigate observation noise, the reliability of using realized futures to determine valid action comparisons remains insufficiently established, particularly in environments with complex stochastic transitions and subsequent decisions. Second, although the authors clarify the differences between HSR and OPSD, the connection between hindsight-induced preference shifts and actual action quality remains largely heuristic.

More importantly, I remain concerned about the methodological novelty. The rebuttal mainly clarifies the motivations and roles of the three proposed components, but does not sufficiently demonstrate a fundamental methodological distinction from existing approaches to step-level credit assignment and hindsight-based learning.

Overall, I acknowledge the additional empirical evidence and clarifications, but these do not fully resolve my concerns regarding the reliability and novelty of the proposed approach. Therefore, I maintain my original score of 2 (Weak Reject).

Thank you for your follow-up. We further address the remaining concerns below.

HPG reliability.

HPG starts from state-aligned action groups, following the grouping principle validated in GiGPO. Realized futures are then used only to prune links between samples with the same state and action but inconsistent hindsight. Policy improvement in reinforcement learning relies on distinguishing the values of different actions within the same state, rather than attributing differences in subsequent outcomes to identical actions taken in that state. Moreover, in our statistics, the specific noise-induced false-pruning cases affected only approximately 0.1% of all samples. In Table 3, HPG also improves both benchmarks in the component ablation.

Interpretation of HSR.

The reward-derived advantage determines whether an action is reinforced or penalized. Hindsight only scales that advantage through a action-level weight bounded in [0.8,1.2]. HSR improves GRPO, GiGPO, and GraphGPO on both benchmarks (Table 4).

Methodological novelty.

The main novelty lies in AGAE and HSR. AGAE brings GAE-style advantage propagation mechanism into critic-free group-based agentic RL. The state-aligned action groups provide value estimates at intermediate actions, creating local TD signals that propagate forward to earlier actions rather than relying only on a terminal-outcome advantage. The propagation coefficient is also designed to adapt to each action group's effective support.

HSR uses hindsight differently from other hindsight-based learning approaches, such as token-level self-distillation. The advantage computed on reward returns determines whether an action is reinforced or penalized. The hindsight-induced preference shift supplies only a bounded weight on that action's update. HSR therefore keeps the per-action update sign anchored to task feedback while adjusting its magnitude.

Reviewer_GaBiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

HeRPO leverages realized future trajectories to retrospectively assess the contributions of individual actions, thereby improving step-level advantage estimation and policy optimization.

Critical strengths
  1. The paper addresses an important problem in long-horizon agentic RL, namely fine-grained credit assignment under sparse outcome rewards.

  2. The proposed method achieves strong empirical results across multiple benchmarks, with reasonably comprehensive ablation studies.

Critical weaknesses
  1. The proposed method mainly combines three incremental improvements to existing group-based credit assignment approaches, rather than introducing a fundamentally new mechanism for credit assignment.

  2. Different future outcomes do not necessarily imply that an earlier action was good or bad, since subsequent decisions and environmental stochasticity can significantly affect the outcome. Both HPG and HSR rely on heuristic hindsight signals without establishing a clear causal connection between individual actions and eventual outcomes.

Questions for authors
  1. HeRPO combines hindsight-pruned grouping, adaptive GAE, and hindsight self-reweighting. Could the authors clarify the key conceptual novelty beyond existing group-based credit assignment methods, such as GiGPO and GraphGPO?

  2. Different future outcomes may result from subsequent decisions or environmental stochasticity rather than the current action. How do the authors justify using hindsight inconsistency in HPG and preference shifts in HSR as reliable signals for assigning credit to individual actions?

  3. HSR requires additional policy evaluations with hindsight-augmented contexts. What is the computational overhead compared with existing methods, and how does it scale with trajectory length?

3 comments
AuthorsOpenReview ↗

Response to Reviewer GaBi (Part I)

Thank you for the constructive assessment and for highlighting the importance of step-level credit assignment under sparse outcome rewards. Below, we provide a detailed, point-by-point response to each listed weakness and question. For clarity, "W" denotes a numbered Critical Weakness and "Q" denotes a numbered Question for Authors from your review. When a weakness and a question address the same issue, we combine their labels (e.g., "W1&Q1").

W1&Q1: Conceptual novelty.

Thank you for the opportunity to clarify the conceptual contribution. While GiGPO uses state-aligned groups and GraphGPO estimates credit through state-transition graphs, HeRPO integrates adaptive GAE into critic-free agentic RL, using group-derived values to propagate credit across action steps. Its core innovation also lies in leveraging realized futures as privileged information to refine action comparisons, adapt temporal credit propagation, and calibrate policy updates.

Specifically, HPG refines state-aligned comparisons by recognizing that different future outcomes following the same action at the same state may result from subsequent decisions or environmental stochasticity rather than the shared action itself. HPG therefore removes comparison links between same-action samples with inconsistent hindsight and estimates values from the retained returns, preventing downstream outcome variations from being attributed directly to the current action. AGAE further incorporates these group-derived values into GAE, adaptively adjusting temporal credit propagation according to the effective sample support of each retained group. Limited support favors longer-horizon Monte Carlo returns, while stronger support emphasizes local TD estimates. Finally, HSR measures how access to the realized future changes the old policy's preference for the executed action and translates this shift into a bounded weight on the advantage, calibrating update magnitude without altering its direction.

The ablations validate the contributions of these components. Among individual extensions to GiGPO, AGAE delivers the largest improvements on both benchmarks (ALFWorld: 86.7% → 93.0%; WebShop: 65.0% → 74.2%), while integrating all three components further increases performance to 95.8% and 80.7%, respectively (Table 3). Moreover, HSR consistently improves GRPO, GiGPO, and GraphGPO on both benchmarks (Table 4), demonstrating its effectiveness beyond the HeRPO framework. We will revise the introduction to state these contributions and distinctions more clearly.

W2&Q2: Hindsight mechanism reliability.

We thank the reviewer for highlighting this concern. HPG focuses step-level credit assignment on comparisons between different actions taken at the same state. As the reviewer notes, divergent future outcomes following the same action may arise from subsequent decisions or environmental stochasticity rather than the shared action itself. Grouping such samples together would therefore introduce downstream outcome variation into the value estimation of the current action. HPG addresses this issue by removing comparison links between samples that share the same state and action but exhibit inconsistent hindsight, while preserving comparisons between different actions at the same state. Importantly, hindsight is used solely to determine group membership, whereas the value estimate in Eq. (4) is computed from the retained samples' reward returns. This design prevents differences arising after an identical action at an identical state from being directly attributed to that action during step-level comparison.

HSR leverages hindsight to calibrate the update magnitude rather than determine update direction. Specifically, HSR measures how access to the realized future changes the policy's preference for the executed action and uses this shift to reweight its update. The estimated advantage remains the sole determinant of the update direction. When hindsight increases the preference for an action, HSR amplifies its positive-advantage update or attenuates its negative-advantage penalty, and a decreased preference produces the opposite effect. Clipping further bounds this adjustment within [1−ϵ,1+ϵ][1-\epsilon,1+\epsilon]. Thus, HSR incorporates privileged future information as a bounded reweighting signal without overriding the advantage-based credit assignment.

The HPG and HSR ablations in Table 3, together with the consistent improvements from integrating HSR into GRPO, GiGPO, and GraphGPO in Table 4, demonstrate the effectiveness and generalizability of these designs. We will clarify it in our revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer GaBi (Part II)

Q3: HSR computational cost.

We thank the reviewer for this valuable comment. To quantify the computational overhead of HSR, we evaluated GiGPO and GiGPO + HSR under identical WebShop training settings. The following metrics were averaged over training iterations:

MethodTime per iteration (s)Action steps/sEffective training tokens/s
GiGPO259.834.76525.36
GiGPO + HSR307.623.70426.49

HSR reweights each executed action based on its hindsight-augmented context under the rollout policy (Eq. 9). Despite introducing moderate computational overhead, HSR yields a 3% performance improvement on WebShop. The additional computation scales approximately linearly with trajectory length, as does the baseline training cost. Consequently, the relative overhead remains bounded and is estimated to approach 25% of baseline training time for longer trajectories. We will supplement the computational-cost analysis in the revised Appendix.

Dear Reviewer,

We are following up on our response to your review. We clarified HeRPO’s contributions relative to GiGPO and GraphGPO, explained how HPG and HSR use hindsight, and provided new measurements of HSR's computational cost. We hope these details address your concerns. If any questions remain, we would be grateful for the opportunity to respond.

Thank you for your time and consideration.

The authors

Reviewer_Xn8cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HeRPO, which uses future information from completed trajectories to improve credit assignment for long-horizon agents without a separately trained critic. Within groups aligned by state, the method removes comparisons involving the same action but inconsistent hindsight information. It adapts the GAE coefficient according to the effective sample support in the retained group, then compares the likelihood of the same action with and without hindsight context and uses bounded weights to rescale the sum of trajectory-level and step-level advantages. The authors evaluate the method on ALFWorld, WebShop, visual Sokoban, and SciWorld in the appendix. Most main experiments report means and standard deviations over three runs, with analyses of components, rollout group size, and the composition of hindsight information.

Critical strengths
  1. No additional critic training or new hyperparameters. HeRPO estimates values from within-group returns without separately training a critic. Its GAE coefficient is determined adaptively, while other settings follow the baseline configurations, reducing the burden of additional model training and hyperparameter selection.

  2. Repeated experiments strengthen confidence in the results. Most main results report means and standard deviations across three independent random seeds. Reporting both average performance and variation across runs provides stronger evidence for assessing the reliability of the gains. (Tables 1–4)

Critical weaknesses
  1. Many design choices are intuitive and lack corresponding theoretical analysis. For example, HSR computes a posterior preference shift using the log-likelihood ratio with and without hindsight information, then combines it with the advantage sign to construct exponential weights (Equations 9–10). AGAE maps effective sample support to an adaptive coefficient λ\lambda (Equation 7), but the theoretical justification for these particular choices remains insufficient. In particular, compared with GRPO, HSR introduces additional optimization bias through reweighting. The paper does not analyze whether the method can still converge to an optimal policy for the original expected-return objective after introducing this bias.

  2. The experimental settings appear outdated, and some metrics are already saturated. The main experiments still focus on older benchmarks such as ALFWorld and WebShop. In Table 1, success rates on several ALFWorld subtasks have already reached 100%, making it difficult for these evaluations to further distinguish the methods. Could the authors report results in more recent agent settings, such as coding or tool use? Comparisons on these more complex long-horizon tasks would help assess whether HeRPO generalizes beyond the existing benchmarks.

  3. The design of HPG appears to have theoretical flaws. In standard RL, the state value under a fixed policy depends only on the current state, not on the action subsequently taken. TD uses the current state value as a common baseline and combines it with the next-state value to identify actions that lead to better states. However, HPG filters samples according to the follow-up action and its hindsight information, making the value estimate for the same state depend on the follow-up action and hindsight rather than serving as a baseline shared across actions. The same state can therefore receive different value estimates in different trajectories. As a result, the TD signal may contain baseline shifts caused by filtering and could even unintentionally reinforce worse actions, but the paper does not analyze this additional bias.

Questions for authors
  1. In AGAE in Section 4.2, a larger number of within-group samples should also make Monte Carlo return estimates more accurate. Why does the method instead rely more on TD when effective sample support is larger (Equation 7)? Could the authors add an ablation using the reverse mapping, with greater reliance on Monte Carlo returns for larger support and on TD for smaller support?

  2. Can HeRPO be applied to coding and tool-use settings? Could the authors report how it compares with other methods in more recent agent settings?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Xn8c

Thank you for the careful review and for recognizing the critic-free design and repeated experiments. Below, we provide a detailed, point-by-point response to each listed weakness and question. For clarity, "W" refers to a numbered "Critical Weakness", and "Q" refers to a numbered "Question for Authors" in your review. When a weakness and a question concern the same issue, we combine their labels (e.g., "W2&Q2").

W1: Design rationale.

Thank you for the opportunity to clarify the design rationale. In AGAE, effective support measures how broadly the retained samples contribute to the group value estimate. The mapping in Eq. (7) favors local TD credit when that estimate has broader support and Monte Carlo returns when support is sparse.

In HSR, Eq. (9) measures the rollout policy’s change in likelihood for the executed action after seeing its realized futures. The exponential in Eq. (10) maps this change monotonically to a positive weight, and clipping bounds that weight within [1−ϵ,1+ϵ][1-\epsilon,1+\epsilon]. These properties limit the size of the reweighting and preserve each advantage sign. Let gwg_w and gg be the weighted and unweighted surrogate gradients on the same rollout distribution, evaluated at θ=θold\theta=\theta_{\mathrm{old}} with the weight ww and advantage AA detached. Then gw−g=E[(w−1)A∇θlog⁡πθ(a∣s)]g_w-g=\mathbb{E}[(w-1)A\nabla_\theta\log\pi_\theta(a\mid s)] and ∣∣gw−g∣∣≤ϵE[∣A∣∣∣∇θlog⁡πθ(a∣s)∣∣]||g_w-g||\leq\epsilon\mathbb{E}[|A| ||\nabla\theta\log\pi_\theta(a\mid s)||], provided the expectation is finite. Here ∣∣⋅∣∣||\cdot|| denotes the Euclidean norm. This bounds the local perturbation relative to the unweighted surrogate using the same advantage estimate.

The current evidence is empirical: adding HSR to GRPO, GiGPO, and GraphGPO consistently improves their performance (Table 4). We will strengthen the theoretical analysis in the revised Appendix.

W2&Q2: Evaluation coverage.

We appreciate the reviewer's attention to evaluation coverage. Our evaluation spans four benchmarks with different interaction demands: text-based embodied tasks in ALFWorld (Table 1), web interaction in WebShop (Table 1), visual planning in Sokoban (Table 2), and interactive scientific experimentation in SciWorld (Appendix C.1, Table 5). Among these, SciWorld requires multi-step interactions with simulated scientific tools and instruments. HeRPO achieves 61.5% success on SciWorld, compared with 57.0% for ScalingInter-7B and 50.5% for AgentGym-RL-7B. Thus, the evidence extends beyond ALFWorld and WebShop to visual and simulated tool-interaction settings. We will evaluate HeRPO on more complex long-horizon tasks in the future work.

W3: Clarify for HPG mechanism.

We appreciate the concern and would like to clarify the role of hindsight in HPG. For two samples with the same state and current action, HPG uses their realized futures only to decide whether each sample should contribute to the other’s value estimate. The estimate itself is calculated on the retained samples' reward returns without using the hindsight.

This pruning rule addresses a specific credit assignment problem. When the same current action is followed by different outcomes, the return difference cannot be attributed to a difference in that action. Using it to assess the shared action would conflate the current decision with subsequent decisions or environmental variation. HPG therefore removes the comparison link between such samples when their hindsight differs, while retaining comparisons between different current actions at the same state.

The HPG ablation supports the practical benefit of this pruning rule in both evaluated agentic tasks. We will elaborate this distinction in the revised manuscript to avoid ambiguity.

Q1: AGAE and the reverse mapping.

We thank the reviewer for highlighting the need for further analysis on AGAE. Effective support in Eq. (7) measures how well the group-derived value estimate is supported. When support is high, AGAE gives more weight to the local TD estimate; when support is low, it prefers the Monte Carlo return.

As suggested, we tested the reverse mapping variant, which instead favors Monte Carlo returns at high support and local TD estimates at low support. On ALFWorld/WebShop with the 1.5B backbone, the reverse mapping achieved 93.4±0.8/77.4±1.6, which underperforms the HeRPO's original mapping in Table 1 (95.8±0.5/80.7±1.2), providing direct evidence for the direction used in Eq. (7). We will include this ablation in the final revision.

Dear Reviewer,

We are following up on our response to your review. We clarified the rationale for AGAE, HSR, and HPG, highlighted the SciWorld results, and reported a new ablation of the reverse AGAE mapping. We hope these details address your concerns. If any questions remain, we would be grateful for the opportunity to respond.

Thank you for your time and consideration.

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.