acceptodds
Under review as a conference paper at ICLR 2027

Beyond Episodes: Learning Behaviors That No Rollout Demonstrated

Abstract

An LLM agent’s rollouts can contain complementary fragments that together solve a task, even when no individual rollout succeeds. Single-episode supervision leaves these combinations undemonstrated. We investigate whether agents can learn these behaviors by composing their own fragmented interaction experience. We propose GraphStitch, which constructs goal-conditioned training trajectories by joining fragments from different rollouts at shared states. By pooling both successful and failed attempts, it recovers solutions absent from every original rollout. Specifically, GraphStitch organizes observed transitions into a shared graph, extracts composed paths, and assigns goals using environment-specific achievement rules. State identity or environment replay checks executability before ordinary supervised fine-tuning. This extends supervision beyond episode boundaries while reusing recorded action-labeled edges as its building blocks. We evaluate compositional goal reaching and generalization to held-out tasks. On held-out Blocksworld, GraphStitch improves greedy success over hindsight relabeling by 28.2 percentage points. Using one-quarter as many collected rollouts as Hindsight, GraphStitch also achieves higher mean held-out greedy success on Blocksworld (68.0% versus 46.8%), with additional replay during data construction. Gains also appear on Countdown, and controlled comparisons support the contribution of cross-episode composition. Composing fragmented experience thus enables LLM agents to learn beyond behaviors demonstrated in individual rollouts.

Primary area:
reinforcement learning
Keywords:
LLM agents, trajectory stitching, goal-conditioned learning, compositional generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Ko9WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GraphStitch, a data augmentation method that stitches together trajectory segments from failed long-horizon task attempts at intermediate states. The comparative experiments demonstrate substantial generalization gains over conventional data augmentation methods.

Critical strengths

The presentation is clear. The method itself is simple, intuitive, and easy to apply, with a straightforward rationale.

Critical weaknesses

The experimental settings are relatively simple: Applying the method to more complex environments may be challenging, since identifying suitable intermediate nodes and segmenting trajectories into appropriate phases can themselves be difficult. The benefits are also not obvious in alphaworld environment. The experimental comparisons are mostly limited to simple data augmentation baselines, without direct comparisons to existing trajectory stitching methods.

The novelty of the proposed method seems limited. Trajectory stitching has already been studied in prior work. The main contribution appears to be applying these ideas to LLM agents, while the methodological differences and additional insights remain relatively limited.

Questions for authors

See Weaknesses.

Reviewer_JsS8OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

GraphStitch joins fragments from different agent episodes at shared states, searches for useful paths, and checks them by replay. The resulting goal-labeled trajectories are used for supervised training. It improves results on Blocksworld and Countdown, including solutions absent from every original episode. On ALFWorld, it does not significantly improve over hindsight relabeling.

Critical strengths

The Blocksworld result is concrete: 98 of 301 collection tasks obtain a solution that no individual episode provides, and the trained model improves on tasks with more blocks. The zero-junction and strict-composition controls help show that joining episodes matters. The paper also reports the negative ALFWorld result, replay costs, and Countdown results after removing training overlap. I lean toward acceptance for this evidence about learning from composed experience.

Critical weaknesses

The controls do not completely separate composition from the amount and distribution of supervision. The token-matched Blocksworld condition has 16,370 assistant tokens versus 8,379 for Hindsight, and the exhaustive-prefix baseline was audited but not trained. Zero-junction has more assistant tokens than matched GraphStitch and still performs worse, so token count alone cannot explain the result, but goal and trajectory-length differences remain.

The benefit is less clear when state matching is ambiguous. ALFWorld has only 14.94% random candidate replay acceptance and no significant gain over Hindsight. The no-gate ablation also changes candidate selection and observation provenance, making it hard to isolate the effect of replay filtering.

Questions for authors
  1. Can you train the exhaustive-prefix baseline under a matched assistant-token budget and update schedule, and report the goal and trajectory-length distributions?

  2. In ALFWorld, can the replay-gate comparison use the same candidates and observation representation? A breakdown of rejection causes and valid new goal coverage would help explain the null result.

Reviewer_a5BAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GraphStitch, a framework for improving LLM-agent learning by constructing successful trajectories from fragments of different rollouts, including unsuccessful attempts. The key observation is that multiple failed rollouts may collectively contain the necessary transitions to solve a task, even when no individual rollout demonstrates a complete solution.

GraphStitch organizes collected state-action transitions into a shared graph and searches for paths that connect fragments across episode boundaries. Candidate trajectories are validated through exact state matching or environment replay, and successful compositions are used as additional goal-conditioned demonstrations for supervised fine-tuning.

The authors evaluate GraphStitch on Blocksworld, Countdown, and ALFWorld. Experimental results demonstrate substantial improvements over single-episode hindsight supervision on Blocksworld and Countdown, while no significant additional improvement is observed on ALFWorld.

Critical strengths

Interesting and practically meaningful idea. The paper investigates an interesting approach to constructing successful training trajectories from fragmented experiences, including failed rollouts. Reusing unsuccessful attempts to recover undemonstrated successful behaviors is potentially valuable for improving the data efficiency of LLM-agent learning.

Simple and intuitive framework. The proposed method combines state-based graph construction, trajectory stitching, environment verification, and supervised fine-tuning. It does not require training an additional world model or generating new state transitions, making the overall framework relatively straightforward and interpretable.

Promising experimental results and useful analyses. The paper reports substantial improvements on Blocksworld and Countdown. The authors also provide ablation studies, including Zero-junction and Strict-stitch, as well as analyses of data budgets, rollout collection costs, and replay validation. These experiments provide useful evidence that cross-episode composition can contribute to downstream policy learning.

Critical weaknesses

W1. The methodological novelty is relatively limited, as the core idea has already been explored in several existing trajectory stitching approaches.

The central idea of GraphStitch is to combine fragments from different trajectories through shared states to construct new successful demonstrations. However, similar ideas have already been extensively explored in earlier studies on trajectory stitching, offline reinforcement learning, and imitation learning, including BATS, Model-based Trajectory Stitching, DiffStitch, and state-level trajectory stitching approaches.

The authors clearly acknowledge these studies and explain the differences between GraphStitch and existing methods. In particular, GraphStitch constructs complete goal-labeled trajectories from recorded state-action transitions and uses them for LLM-agent supervised fine-tuning, rather than generating connecting transitions or modifying the policy-learning objective.

While these distinctions are valid, they appear to be primarily differences in implementation choices, supervision format, and application setting, rather than fundamental methodological innovations. The core trajectory composition principle remains largely unchanged.

Therefore, the methodological novelty appears relatively narrow, and the paper does not sufficiently demonstrate what new technical challenges are addressed beyond adapting existing trajectory stitching techniques to LLM-agent training.

W2. Lack of direct experimental comparisons with existing trajectory stitching-based methods.

Although the paper extensively evaluates GraphStitch against Won-only and Hindsight baselines, it does not provide direct comparisons with representative trajectory stitching methods.

This omission is particularly important given that the proposed method shares its central motivation and construction mechanism with several existing approaches. The current experiments mainly demonstrate that cross-episode trajectory composition can outperform single-episode hindsight relabeling, but do not establish whether GraphStitch offers advantages over existing trajectory stitching techniques.

For example, prior work on state-level trajectory stitching and combinatorial generalization has already investigated how to construct or exploit behaviors absent from individual demonstrations.

A controlled comparison with representative trajectory stitching methods, adapted to the same LLM-agent environments and training settings, would be necessary to better establish the incremental contribution of GraphStitch.

W3. The performance improvements may be partially influenced by differences in training data volume and supervision distribution.

The main Blocksworld experiment uses substantially different amounts of training data. GraphStitch constructs 7,274 training trajectories, compared with only 557 trajectories for Hindsight.

The authors provide useful controls, including Zero-junction, Strict-stitch, and total-token-matched experiments. However, the total-token-matched GraphStitch dataset still contains approximately 1.95 times as many loss-bearing assistant tokens as the Hindsight dataset.

Furthermore, the constructed trajectories differ in length and goal distribution, which may also contribute to the observed improvements.

Although the existing ablations provide evidence supporting the value of cross-episode composition, they do not fully isolate its contribution from additional supervision or changes in the training distribution. More carefully matched comparisons would strengthen the experimental conclusions.

W4. The effectiveness and generalizability of GraphStitch remain limited in more realistic interactive environments.

GraphStitch achieves substantial improvements on Blocksworld and Countdown, where environment states and transitions can be represented relatively precisely. However, it does not demonstrate statistically significant improvements over Hindsight on ALFWorld.

This limitation is particularly concerning because GraphStitch relies on matching shared intermediate states across different rollouts. In more realistic LLM-agent environments, observations may be incomplete, and identical observations do not necessarily imply equivalent underlying states.

The ALFWorld experiments highlight this challenge: only 14.94% of uniformly sampled candidates are executable after refining the state keys, while almost all candidates constructed using observation-only keys fail replay validation.

These results suggest that the applicability of GraphStitch may depend heavily on accurate state representations and efficient environment replay. Additional evaluation or analysis would help establish when the proposed method provides reliable benefits in practical agent environments.

W5. The claimed experience efficiency does not fully account for trajectory construction and verification costs.

The paper emphasizes that GraphStitch achieves higher performance than Hindsight while using fewer collected rollouts. However, constructing and verifying composed trajectories requires additional environment replay.

For example, in the Blocksworld comparison, GraphStitch with 2,408 collected episodes requires 17,541 total environment invocations, whereas Hindsight with 9,632 collected episodes requires 14,818 invocations.

Although environment replay can be considerably cheaper than collecting new LLM-generated rollouts, the current comparison does not fully characterize the end-to-end computational and interaction costs.

A more comprehensive efficiency evaluation would help clarify the practical benefits of GraphStitch over simply collecting additional rollouts.

Questions for authors
  1. Novelty and methodological contribution: The authors acknowledge several closely related trajectory stitching methods. Beyond differences in implementation choices, supervision format, and application to LLM agents, what are the fundamental methodological contributions of GraphStitch over existing approaches?

  2. Comparison with prior trajectory stitching methods: Why are representative trajectory stitching-based methods not included in the experimental comparisons? Can the authors adapt and evaluate existing state-level trajectory stitching approaches under the same environments, rollout pools, and training budgets?

  3. Controlled evaluation: Can the authors provide experiments matched on loss-bearing assistant tokens, trajectory lengths, and goal distributions to better isolate the benefits of cross-episode composition from those of additional supervision?

  4. Generalization to interactive environments: Why does GraphStitch not demonstrate significant improvements over Hindsight on ALFWorld? Is the limitation primarily due to inaccurate state matching, low replay validity, or insufficient additional goal coverage?

  5. State matching and applicability: How could GraphStitch be extended to partially observable or dynamic agent environments where exact state representations and deterministic replay are unavailable?

  6. End-to-end efficiency: Can the authors provide comparisons under matched total computational or environment-interaction budgets, including rollout collection, graph construction, and replay validation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.