- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RELACE, a critic-free credit-assignment method for multi-turn language agents. After collecting trajectories, it scores each executed action under its original context and a hindsight context conditioned on the observed terminal state. The resulting score ratios are normalized and clipped within each trajectory, then used to reweight discounted terminal returns. These targets enter state-conditioned advantage estimation, which is combined with trajectory-level GRPO supervision. Experiments on ALFWorld and WebShop report improvements over several baselines.
Critical strengths
-
The paper addresses an important problem: terminal rewards provide limited information about the contributions of individual actions in long-horizon interaction.
-
Comparing original-context and hindsight-context scores is a reasonable attempt to distinguish outcome-dependent changes in action plausibility from baseline plausibility. The method uses teacher-forced scoring without requiring a separately trained value model or additional environment rollouts for credit estimation.
-
Evaluation covers two environments and two model sizes. The authors also appropriately acknowledge that their scores represent retrospective relevance rather than causal contribution.
Critical weaknesses
-
The central interpretation of the weights lacks direct empirical validation. RELACE measures how changing the context alters the likelihood of an executed action. However, compatibility with the observed terminal state does not necessarily indicate usefulness for the original task. In particular, the hindsight templates reformulate the goal as reaching the actual final observation. The paper does not directly establish that necessary or useful actions in successful trajectories receive higher weights than redundant actions. Table 1 measures downstream performance, Table 2 compares training variants, and Figure 3 reports aggregate scoring dynamics; none directly evaluates this correspondence. Appendix E states that per-step scores and weights are reported, but the examples provide observations and actions without those numerical values. Consequently, the reported performance gains do not establish that the method improves action-level credit accuracy.
-
The retrospective weighting cannot distinguish actions within zero-reward failed trajectories. From Eq. (6), implies for every step, regardless of the retrospective weights. The macro advantage therefore still assigns the same negative signal to all actions in a failed trajectory when its task group contains successful trajectories. State-conditioned comparisons can provide different local corrections, but these remain based on downstream outcomes and need not identify where the failure actually occurred. If all trajectories for a task receive zero reward, both advantage components vanish. Thus, the proposed weighting does not resolve the motivating case in which a late mistake causes useful earlier actions to receive unfavorable credit. This is a substantive scope limitation rather than an implementation error and should be reflected more explicitly in the claims.
-
The ablation does not isolate the main proposed addition. Table 2 compares hindsight-score weighting using with RELACE using a trajectory-normalized and clipped likelihood ratio. This changes the original-context denominator together with normalization and clipping. The improvement from 75.78% to 79.43% therefore does not establish the denominator's independent contribution. A comparison that removes only the denominator while preserving the other transformations would more directly test the central design choice. Moreover, the success-protecting mask removes all negative local corrections from successful trajectories, so identifying a relatively low-value action does not necessarily lead to its local suppression.
-
Comparison settings and statistical reporting limit confidence in the gains. Appendix C.1 uses for RELACE and for GiGPO, directly affecting the step-level targets being compared. Table 1 also combines published baseline configurations without sufficiently establishing protocol equivalence. In addition, the cited HCAPO paper reports an ALFWorld 7B result of with smoothing, whereas Table 1 uses its result; the smoothed configuration is particularly relevant because RELACE also uses smoothing. Finally, the seed coverage and aggregation details referenced in the text and Figure 2 are not supplied in Appendix C. These omissions matter for interpreting the smaller performance margins.
Questions for authors
-
Can the authors provide per-step weights for successful trajectories containing both necessary actions and clearly redundant actions, together with an independent assessment of action usefulness? A controlled comparison involving redundant steps or action removal would directly test the proposed interpretation.
-
What specific benefit does retrospective weighting provide for zero-reward failed trajectories, beyond the existing macro and state-conditioned outcome comparisons? Please clarify the intended scope of the claim about protecting useful actions from late failures.
-
Does the original-context denominator improve performance when normalization, clipping, smoothing, masking, and the discount factor are held fixed? Please also clarify seed counts, checkpoint selection, and the comparison with smoothed HCAPO.
-
How sensitive are the retrospective weights to the hindsight prompt formulation? Since the current template replaces the original task with the observed final state, can the authors distinguish changes caused by outcome-relevant information from those caused by reformulating the goal? A comparison with a prompt that retains the original task and adds the final observation would help clarify this issue.
-
How much of the reported improvement depends on the success-protecting mask? Since this mask removes all negative local advantages from successful trajectories, does RELACE mainly reinforce selected actions rather than suppress redundant ones? An otherwise matched comparison with and without the mask would help separate the contribution of retrospective weighting from that of asymmetric protection.
3 comments
W3 and Q3. Misunderstanding
We would like to clarify an important point about the ablation design. Our denominator-removal ablation changes only the original-context denominator, while retaining trajectory normalization, clipping, temporal smoothing, success-protecting masking, the discount factor, and all remaining training settings.
Specifically, the only modification is
\quad\longrightarrow\quad u_{i,t}=p^H_{i,t}.$$ The controlled WebShop results are: - **GiGPO:** 73.83% success. - **Without original-context denominator:** 75.78%. - **Full RELACE:** 79.43%. We have also strengthened the theoretical motivation in Appendix C by deriving the hindsight likelihood-ratio identity. This explains why hindsight plausibility alone is insufficient: an action may receive a high hindsight score simply because it was already probable in its original context. The denominator accounts for this baseline plausibility. Our practical likelihood scores approximate this principle rather than constituting exact hindsight posteriors. Regarding reporting, we have clarified that validation occurs **every five training updates**, and each run reports its highest validation success rate within the **first 200 updates**. For multi-seed experiments, we compute this maximum independently for each seed and then aggregate the resulting values, reporting the mean and standard deviation. Single-seed ablations are explicitly distinguished from multi-seed results. ### **W4. Stronger HCAPO baseline and comparison fairness** We thank the reviewer for identifying the stronger published HCAPO configuration. **We have updated Table 1 to include the published smoothed HCAPO results** and revised the corresponding comparisons. To investigate discount-factor sensitivity, we additionally evaluated RELACE at $\gamma=0.95$, obtaining **77.34% WebShop success**, compared with 79.43% at its default $\gamma=0.6$. This indicates that RELACE's performance is not solely dependent on choosing the smaller discount factor. We are currently performing similar ablation studies on GiGPO as well. For $\gamma=0.6$ on GiGPO, the accuracy is $74.22\%$ and the validation success rate curve is very instable, often fluctuating between high and low values. ### **Q4. Sensitivity to hindsight prompt formulation** Our hindsight prompt intentionally conditions on the **observed terminal state** to measure how the outcome changes the likelihood of an executed action. This is used only for retrospective scoring, without changing rollout generation. Retaining the original task alongside the final observation could introduce conflicting objectives, particularly in failed trajectories, where the intended goal differs from the observed outcome. Nevertheless, we agree that comparing prompt formulations is an interesting direction for future analysis. ### **Q5. Contribution of success-protecting masking** We have added component ablations in **Appendix D.3**. Removing the success-protecting mask reduces WebShop success from **79.43% to 75.00%**, while removing temporal smoothing reduces it to **75.78%**. Importantly, **retrospective weighting remains effective even with the mask fixed**, as demonstrated by our case studies. Although the mask prevents negative local advantages on successful trajectories, RELACE still reduces the positive advantages of redundant actions (2.355 to 2.302 and 0.724 to 0.658), while increasing those of necessary actions. Thus, the mask does not eliminate RELACE's ability to distinguish useful from redundant actions. ## **Summary** We thank the reviewer for the constructive suggestions. We have now added **two detailed case studies, controlled ablations, stronger baseline comparisons, and additional theoretical justification**, while clarifying the limitations of our method. We believe these additions directly address the main concerns and substantially strengthen the paper. We hope the reviewer finds the revised manuscript satisfactory and welcome any further questions.Thank you very much for your response. I have read it carefully; however, my concerns mentioned earlier regarding the methodology and experiments remain, and therefore I have decided to maintain my score (2).
We thank the reviewer for the detailed and constructive feedback. We have substantially strengthened the manuscript with two new action-level case studies, controlled ablations, additional theoretical analysis, and clearer experimental reporting. We address each concern below.
W1 and Q1. Direct evidence for action-level credit assignment
We agree that improvements in aggregate success rates alone do not establish the quality of action-level credit assignment. To directly address this concern, we have added two extensive WebShop case studies (Appendix E) reporting original-context likelihood scores, hindsight scores, retrospective weights, discounted returns, and final advantages.
Importantly, we independently identify necessary and redundant actions from their effects on the environment, rather than inferring their usefulness from RELACE's weights. We then compare RELACE against an otherwise identical estimator with retrospective weighting disabled (ρt=1), using the exact same rollout batches and retaining all other components.
Case Study 1: Necessary purchase versus redundant action. Two trajectories successfully purchase the same product, but one performs an unnecessary color re-selection that leaves the environment unchanged.
click[buy now]receives ρt=1.053, increasing its advantage from 2.946 to 3.034.click[dark heather]receives ρt=0.961, decreasing its advantage from 2.355 to 2.302.Case Study 2: Identical actions with different usefulness. We compare the exact same action,
click[5x-large tall], across two successful trajectories. In one trajectory, the action selects the required size; in the other, it redundantly re-selects an already selected size, leaving the environment unchanged.Both actions have identical discounted returns (Gt=6) and receive identical advantages (0.724) without retrospective weighting. However, RELACE distinguishes them:
Thus, retrospective weighting changes the advantage gap from zero to 0.166, despite identical action text, identical discounted returns, and identical terminal success.
These examples directly demonstrate the mechanism the reviewer requested: RELACE can distinguish actions according to their usefulness in the observed environment, even when conventional discounted-return comparisons assign them identical credit. The distinction remains after all components of the implemented estimator, including success-protecting masking, are applied. Although these examples do not establish universally accurate causal attribution, they provide concrete action-level evidence beyond aggregate benchmark performance.
W2 and Q2. Zero reward trajectories.
We acknowledge that when all trajectories receive zero reward, RELACE's advantages vanish, as is also the case for GRPO and GiGPO. This is a general limitation of outcome-based learning, rather than a limitation specific to our retrospective weighting. Our contribution is to provide finer-grained credit assignment from available reward signals. Dynamic sampling and exploration-based methods are complementary approaches for handling all-failure groups.
Importantly, even when individual trajectories fail, RELACE can still provide informative local comparisons when the task group contains successful trajectories. The retrospectively weighted returns from successful trajectories influence the state-conditioned baselines, thereby affecting the advantages assigned to actions in failed trajectories. Unlike GiGPO, RELACE additionally incorporates hindsight information into these comparisons, allowing more informative credit allocation without requiring additional reward models.