- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ReStitch, a retrieval-augmented sequence modeling approach for offline RL that aims to explicitly realize trajectory stitching by retrieving and injecting trajectory continuations into a Decision Transformer via cross-attention. Retrieval uses a utility that combines relevance in a learned IQL value representation, a feasibility gate based on value, and an advantage-weighted score aggregated over the candidate segment. A dual-stream Transformer with distance-aware selective stitching preserves a BC pathway while allowing retrieval-conditioned actions. On D4RL, ReStitch yields substantial gains on sparse-reward AntMaze tasks, with more modest or limited improvements on dense-reward MuJoCo control. Ablations indicate the joint importance of relevance/feasibility/advantage and of representation quality from IQL.
Critical strengths
The paper introduces a clear, instance-based perspective on policy improvement: retrieving explicit, feasible, and advantageous continuations, rather than only conditioning on scalar signals or optimizing auxiliary objectives.
Critical weaknesses
- The stitching claim is not established on the comparison or the benchmark that would test it.
The substantial gains' are large relative to DT and QT, which are weak on AntMaze. Relative to the relevant stitching baselines the margin is small. The per-task gaps versus reproduced HIQL are approximately , , , , , and . On medium-diverse, QCS-RTG is slightly higher (79.1 versus 78.6). On large-play and large-diverse, the gaps versus HIQL (3.1 and 2.4) are about one standard error given the reported seed standard deviations. The average of 83.5 versus 79.3 therefore does not, by itself, show a reliable stitching improvement.
The benchmark choice leaves the stitching claim under-tested. D4RL AntMaze is a fixed-goal, sparse-reward task, and behavior cloning fails on its medium and large mazes, so it still measures classical policy improvement over the datasets. It does not isolate the trajectory composition claimed in the title. The play and diverse episodes are long and already chain several commanded goal-reaching segments, and evaluation uses a single goal. Park et al.\ (OGBench, ICLR 2025, Table~1) therefore mark D4RL AntMaze as not testing goal stitching: many tested state-goal pairs already occur together in the training data (Ghugare et al., 2024), and the dataset does not force the agent to assemble short, disjoint segments. OGBench's singletask variants preserve ReStitch's problem formulation, while its stitch datasets limit each trajectory to four cells and its giant mazes extend the horizon. Results on \texttt{antmaze-large-stitch-singletask} and \texttt{antmaze-giant-stitch-singletask} are the direct test of whether retrieval composes segments or imitates long logged paths. They are absent, and the paper does not discuss this choice.
[1] OGBench: Benchmarking Offline Goal-Conditioned RL
[2] Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of View
- The implementation does not isolate trajectory-level stitching. The stitching loss regresses only the first action of the highest-utility retrieved segment. The remaining horizon is context for cross-attention. At inference, utility is replaced by latent one-nearest-neighbor cache inheritance. The paper therefore compares a full system with IQL pretraining, reranking, dual heads, and a distance gate against external baselines, without an ablation that separates advantage-weighted action relabeling from the use of an explicit continuation. There is also no measurement that the executed behavior is composed from different dataset trajectories.
Questions for authors
-
Add two controls on AntMaze: regress the stitching head toward the retrieved action with the trajectory context removed, and train an advantage-weighted behavior-cloning head with no retrieval. Also compare one-nearest-neighbor cache inheritance with recomputing the utility at the query state.
-
On at least \texttt{antmaze-large-stitch-singletask} and \texttt{antmaze-giant-stitch-singletask}, report success under the OGBench single-task reward.
-
After a successful AntMaze episode, what fraction of visited state--action pairs comes from distinct source trajectories, and for how many steps does behavior continue to follow the retrieved segment?
-
Appendix C refers to a distance-gate ablation in Section 5 and contains an unresolved ``Proposition ??''. Please add the ablation or remove the claim. Proposition 2 restates the threshold algebra of . For Proposition~1, please measure how much varies across retrieved segments.
-
The indicator filters by value. Please provide a quantitative check that accepted anchors are reachable, beyond the single pair in Figure~2.
3 comments
Response to WeFn (2/2): Stitching Mechanism, Ablations, and Theoretical Clarification
W2 / Q1. Isolating trajectory-level stitching from target supervision and cache approximations.
- Fixed-target controls (Table R1): Holding and rankings fixed (), AWR (advantage-weighted regression on dataset actions without retrieval) obtains
52.6and25.2, Target-only (regressing onto without retrieved context) obtains74.2and58.4, Anchor-only () obtains82.6and71.8, and Full ReStitch () obtains88.2and80.9. - Cache inheritance vs. query-time reranking (Table R2): Recomputing utilities or re-retrieving online at test time yields
88.3and88.5(vs.88.2for cache inheritance) while increasing latency by +10.4% and +128.9%. - Conclusion: ReStitch is not reducible to AWR or memoryless target regression: overall retrieved context conditioning contributes +14.0 to +22.5 over identical target supervision alone (
Target-only), of which +5.6 to +9.1 specifically come from the subsequent continuation tokens beyond the initial anchor pair (FulloverAnchor-only).
Q3. Evidence of composition and how retrieved segments are used at inference. To address the reviewer's concerns regarding evidence of trajectory composition and the use of retrieved segments at inference, we clarify how ReStitch operates during rollouts, emphasizing its closed-loop, retrieval-conditioned action generation mechanism and providing additional empirical evidence of trajectory stitching on benchmarks with fragmented offline trajectories:
- Closed-loop continuous action generation rather than trajectory copying: At inference (Algorithm 1, Phase 3), the agent does not copy dataset actions or execute a retrieved segment open-loop for multiple steps. Instead, at every step , the policy observes a continuous environment state , queries FAISS for the nearest cache owner , and attends jointly over all inherited segments ( tokens weighted by ) via Cross-Attention to decode a continuous action before re-querying at step . Consequently, visited pairs are continuously synthesized rather than drawn verbatim from individual dataset trajectories, and the horizon serves as a lookahead conditioning window rather than a multi-step execution plan.
- Evidence of multi-trajectory composition via OGBench
stitch(Table R5): Because actions are continuously synthesized from a weighted mixture of candidates at each step, the cleanest way to verify genuine multi-segment composition—without relying on heuristic post-hoc matching of generated actions to dataset episodes—is to evaluate on benchmarks where no single dataset trajectory spans the task. In OGBenchlarge-stitchandgiant-stitchexperiments (Table R5 above), every offline trajectory is strictly restricted to short local corridor segments (never connecting start to goal). Achieving 48.89% and 13.33% success on these benchmarks (vs. 28.33% and 0.00% for IQL) provides direct task-level evidence that the policy can leverage guidance from fragmented offline trajectory segments to reach the goal.
Q4. Appendix C and Proposition 1.
- We fixed the broken reference, added the distance-gate ablations (Tables R3a and R3b), and downgraded Proposition 2 to a brief remark characterizing gate locality.
- Proposition 1 and : IQL does not fit an explicit behavior density model . We agree that is not strictly constant across segments and cannot be verified without an auxiliary behavior density model. We have therefore removed the formal “Proposition 1” header and rephrased the derivation as an informal heuristic extension of IQL’s unnormalized AWR weight to -step segments.
Q5. Value indicator and reachability. We agree with the reviewer that the condition evaluates value compatibility rather than certifying dynamical reachability. In the revised manuscript, we have renamed this component from "feasibility gate" to "value-compatibility condition" throughout the text and removed all claims that it guarantees physical reachability. As detailed in the General Response (Section 3, Table R4), this indicator acts solely as a soft ranking preference that controls the assignment of the advantage bonus (while relevance in the learned value representation and the distance gate handle local proximity).
Follow-up Response: Baseline Comparisons
2. Scope of Contribution and Comparison with Stronger Single-Task Baselines (ReBRAC, FQL, GFP)
We agree that ReBRAC, FQL, and GFP are essential performance references for single-task antmaze-large-stitch-singletask-task1-v0.
To clarify how these results relate to our core claim, we distinguish between two questions:
- Can retrieval unlock trajectory stitching in return-conditioned sequence models (RCSL / DT-style methods)? (Our primary focus)
- How does the resulting method compare against state-of-the-art TD-based and flow-based actor–critic algorithms? (Broader performance context)
As established in theoretical and empirical analyses of RCSL (Brandfonbrener et al. [1]; R²CSL [2]), sequence-level return-conditioned methods require return coverage at the initial state distribution and struggle fundamentally when stitching suboptimal fragments. On antmaze-large-stitch, standard RCSL collapses completely (0.0% success for DT). By combining segment retrieval, value-guided scoring, and conditional action prediction, ReStitch raises RCSL performance from 0.0% to 48.89% (and outperforms the standard single-task value-based IQL baseline of 28.2% which we used to guide the retrieve).
At the same time, we explicitly acknowledge that ReStitch (48.89%) remains well below heavily regularized TD actor–critic methods (ReBRAC: 88.4%) and expressive flow-policy actor–critics (FQL: 71.8%, GFP: 90.3%). We do not claim that ReStitch outperforms state-of-the-art actor–critic methods; rather, our goal is to bridge the severe stitching deficit within the RCSL/DT paradigm. We will explicitly include ReBRAC, FQL, and GFP in our comparison table and discuss this paradigm gap as a concrete limitation and direction for future work.
3. Direct RCSL Comparison on OGBench (DT & QCS vs. ReStitch)
To directly test our central thesis on OGBench, we evaluated standard DT and QCS on Large-stitch and Giant-stitch Task1:
| Method | Large-stitch Task1 | Giant-stitch Task1 |
|---|---|---|
| DT (context length ) | 0.00 ± 0.00% | 0.00 ± 0.00% |
| QCS | 25.89 ± 11.98% | 0.00 ± 0.00% |
| ReStitch (Ours) | 48.89 ± 9.38% | 13.33 ± 2.73% |
These results confirm that when complete successful demonstrations from the start state are absent in the offline dataset, standard DT fails to achieve any task completions (0.00%), whereas ReStitch consistently stitches fragments to achieve non-trivial success on both Large-stitch (48.89%) and Giant-stitch (13.33%). Together with our controlled ablation studies in the main paper (which isolate the effect of the retrieval module under identical backbones), this provides transparent positioning of both ReStitch's specific improvement over RCSL baselines and its current performance gap relative to strong baselines( HIQL, ReBRAC, FQL).
References
[1] Brandfonbrener et al. When Does Return-Conditioned Supervised Learning Work for Offline Reinforcement Learning? NeurIPS 2022. [2] R²CSL: Return- and Reachability-Conditioned Supervised Learning. arXiv:2506.08463, 2025.
Response to WeFn (1/2): Empirical Margin and OGBench Evaluation
We thank the reviewer for the rigorous critique and especially for pointing us to OGBench, which we agree provides a cleaner testbed for strict trajectory stitching.
W1a. Margin over strong baselines. We agree that the improvements over HIQL on antmaze-large-play (74.5 vs. 71.4) and antmaze-large-diverse (80.9 vs. 78.5) are modest relative to cross-seed variability and should not be overstated as statistically conclusive. However, ReStitch achieves a higher average normalized return across the six AntMaze tasks (83.3 vs. 79.3 for HIQL), suggesting a broader performance advantage rather than an isolated improvement. More importantly, our additional experiments on the OGBench stitching benchmarks (Table R5) provide further evidence of ReStitch's effectiveness in settings that explicitly require composing short trajectory segments. On large-stitch, ReStitch achieves 48.89% success compared with 37.22% for HIQL and 28.33% for IQL, while on giant-stitch, it achieves 13.33% compared with 9.89% and 0.00%, respectively. Although these results also exhibit variability, they provide complementary evidence supporting the effectiveness of retrieval-based trajectory stitching. We have revised the manuscript accordingly to moderate our claims and clearly distinguish consistent empirical improvements from statistically conclusive gains.
W1b / Q2. Benchmark choice and OGBench evaluation. We agree with the reviewer and recent literature (Park et al., 2025; Ghugare et al., 2024) that standard D4RL AntMaze datasets contain some long trajectories, making OGBench
stitchdatasets a cleaner testbed for strict compositional stitching (where dataset trajectories are capped at short segments and never connect start to goal). Following your suggestion, we evaluated ReStitch againstIQLandHIQL (fixed-task adaptation)on bothantmaze-large-stitch-singletask-task1-v0andantmaze-giant-stitch-singletask-task1-v0(Table R5 below).Evaluation protocol: To match our fixed-task setting, all methods are trained and evaluated on the default fixed task (
task1) across 3 random seeds (seeds 0, 1, 2) for 1M gradient steps per seed. For each seed, we evaluate the final 3 checkpoints (800k,900k, and1Msteps) over 100 full rollout episodes per checkpoint, yielding 3×100=300 evaluation episodes per seed and 900 episodes per method per task (1,800 episodes total across both tasks). We report the mean success rate and standard deviation across the 3 seeds:Table R5. Evaluation on OGBench
large-stitchandgiant-stitch(task1, 900 episodes per method/task)Interpretation:
Large-stitch: When offline demonstrations are strictly fragmented into short local segments, ReStitch achieves a higher mean success rate (48.89 ± 9.38%) than both IQL (28.33 ± 15.95%, +20.56 percentage points) and HIQL fixed-task adaptation (37.22 ± 20.21%, +11.67 percentage points), with a smaller cross-seed standard deviation onLarge-stitch(9.38%vs.20.21%for HIQL).Giant-stitch: On the extreme-horizonGiant-stitchbenchmark, standard IQL fails (0.00 ± 0.00%), whereas both HIQL (9.89 ± 2.34%) and ReStitch (13.33 ± 2.73%) achieve non-trivial success (+3.44 percentage points higher mean for ReStitch, though absolute success remains modest for all methods). Because both HIQL and ReStitch rely on IQL-style expectile value learning for long-horizon credit assignment, value degradation over extreme horizons likely remains a shared bottleneck onGiant-stitch.Overall, Table R5 demonstrates that ReStitch achieves the highest mean success rates on both OGBench stitching benchmarks, providing additional evidence of its effectiveness in settings that explicitly require composing short trajectory segments. The improvements are more pronounced on Large-stitch, while the relatively low absolute success rates on Giant-stitch highlight the remaining challenges of extreme-horizon trajectory stitching.