acceptodds
Withdrawn from ICLR 2027

RAHL: Recursive Self-Improvement through Repairability-Aware Hindsight Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language agents can fail their assigned tasks while achieving other goals along the way. Hindsight relabeling can recover these successes, but changing the goal alone may leave recorded plans inconsistent with the new task. Goals achieved by the same trajectory can differ in how much plan revision they require. We introduce Repairability-Aware Hindsight Learning (RAHL), which selects verified goals according to the cost of repairing their associated plans. The acting policy revises plans in execution order, conditioning each edit on the previously repaired history. A policy-generated mask restricts edits to spans classified as goal-specific intentions, while recorded actions and observations remain unchanged. Among candidates that satisfy the editing constraints, RAHL favors those requiring fewer token changes. Each replay-verified demonstration is paired with new rollouts from the same initial state for training alongside original tasks. The updated policy then acts, proposes goals, and repairs demonstrations in the next round, forming a recursive actor–relabeler loop. Experiments show higher GRPO task success on ALFWorld, WebShop, and ScienceWorld at matched training rounds, with gains depending on the optimizer. On identical WebShop histories, the trained relabeler yields more replay-verified demonstrations, while demonstration yield per model call remains similar.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, language agents, hindsight relabeling, recursive self-improvement, plan repair, self-generated supervision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TyVAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RAHL, which converts incorrect rollouts generated by an agent during training into successful trajectories by reconstructing the tasks and agent sequences. This allows the method to utilize trajectories that fail their original tasks and aims to address the lack of training signals in GRPO when all rollouts in a group receive zero rewards. Through experiments and analyses on ALFWorld, WebShop, and ScienceWorld, and comparisons with a GRPO-only baseline, the paper demonstrates the effectiveness of the proposed method.

Critical strengths
  1. The paper addresses a practical challenge in agent reinforcement learning: groups of failed rollouts with identical fail rewards provide no group-relative reward advantage in GRPO. The motivation is clearly presented, and the examples effectively illustrate the method.

  2. The proposed method, RAHL, can be used to design tasks tailored to the agent and may also be extended to areas such as curriculum learning.

Critical weaknesses
  1. RAHL is a task synthesis method based on achieved outcomes. However, utilizing the newly synthesized tasks requires additional rollouts, and the method does not fully reuse all previously collected rollout trajectories, which still results in wasted computation. Previous work has already discussed the lack of training signals caused by groups of entirely failed rollouts [1][2][3][4]. The authors should clarify how RAHL differs from these methods and include them as baselines in the experiments.

  2. The results do not convincingly demonstrate the effectiveness of RAHL. For example, in Table 2, adding RAHL to GSPO and GiGPO results in worse performance than their respective baselines, while Figure 4 does not demonstrate the benefits of RAHL. The authors should also use appropriate visual formatting, such as boldface in the tables, to make the comparisons clearer to readers.

[1] "ExGRPO: Learning to Reason from Experience", Zhan, Li, et.al. [2] "ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay", Lu, Zhong, et.al. [3] "Learning from Hindsight for VLA Reinforcement Learning", Xu, Jiang, et.al. [4] "CRPS: Curriculum Replay via Progressive Suffixes from Successful Trajectories for Long-Horizon LLM Agents", Zhang, Yang.

Questions for authors
  1. How do the authors interpret “Recursive Self-Improvement” in the title? The paper does not improve the model’s capabilities as a goal proposer, span classifier, or in other such roles, but only improves its capabilities as an actor. Meanwhile, Figures 3 and 4 do not provide sufficient evidence of sustained improvement over longer training horizons.
Reviewer_EZEwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RAHL improves LLM training by converting failed trajectories into successful demonstrations through hindsight goal relabeling. It introduces repairability-aware goal selection, favoring achieved goals that require minimal edits. These demonstrations are paired with new rollouts for GRPO training, forming a recursive actor–relabeler learning loop.

Critical strengths
  1. Combining verified successful demonstrations with newly sampled rollouts provides additional learning signals and alleviates the zero-advantage problem in all-failure GRPO groups.
  2. One of the key distinctions between RAHL and existing hindsight relabeling methods is its repairability-aware goal selection.
Critical weaknesses
  1. The paper does not sufficiently justify or empirically validate repairability-aware goal selection. A lower editing cost primarily reflects textual similarity, rather than the potential training benefit, task difficulty, or informational value of the relabeled goal. Therefore, an ablation comparing random selection among valid, repairable goals against minimum-edit-cost selection is necessary to establish the effectiveness of the proposed strategy.
  2. Please report the distribution of the number of valid, repairable goals generated from each source trajectory, particularly the proportion of trajectories yielding multiple candidates within the same relation bucket. If most source trajectories yield only one eligible candidate, the proposed repairability-aware selection mechanism rarely makes a meaningful decision.
Questions for authors

Please see the weaknesses section.

Reviewer_sTHPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RAHL, which constructs hindsight demonstrations from unsuccessful interactions between language agents. A shared policy provides achieved goals and repairs goal-specific plan segments while preserving the recorded actions and observations. A native verifier identifies the first successful prefix, and a replay checks the preserved execution. The method selects a repairable goal with minimal token edit cost within a sampled relation compartment, then trains on a demonstration jointly verified with seven new rollouts and the original task data using GRPO. The method is based on HER is goal-conditioned RL. Though Experiments show improvements on ALFWorld and WebShop, I don’t see too much novelty of this method. Even the trick emphasized in the title that selects a repairable goal with minimal token edit cost, in my point of view, is mediocre because I cannot find any apparent advantages of minimal token edit cost.

Critical strengths
  1. Extensive experiments are done to show the effectiveness of their method. The paper provides substantial training details in the appendix.
Critical weaknesses
  1. The paper is poorly written. Some parts of the paper are not written from the reader's perspective. For instance, the “goal A”, “goal B”in Intro is so hard to follow for the first time of reading. The three rollouts in “Collect & propose”in Fig. 2 are the same, which makes no sense. There are too many equations in Sec 3.5, some of which are training details and should be placed to appendix.

  2. Some critical parts of the method lacks reasonable experimental validation. (1) The central repairability-aware selection criterion is insufficiently validated. No experiments show the advantage of minimum token edit cost against some basic methods like uniform selection from the truly repairable candidates. (2) The additional benefit of recursive relabeler updates remains unresolved. On Tab. 5, verified sources increase from 32 to 36, but model calls also increase from 307 to 346.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.