acceptodds
Under review as a conference paper at ICLR 2027

ReStitch: Offline Trajectory Stitching via Retrieval-Based Policy Improvement

Abstract

Return-Conditioned Supervised Learning (RCSL), notably Decision Transformers (DT), has emerged as a stable paradigm for offline reinforcement learning. However, these methods can struggle to stitch suboptimal trajectories, limiting their performance in sparse-reward environments. While recent methods incorporate value-based guidance into DTs, they typically represent policy improvement through conditioning signals or optimization objectives rather than explicit trajectory-level continuations. In this work, we propose ReStitch, which formulates retrieval as an instance-based mechanism for offline policy improvement. ReStitch retrieves trajectory segments that are relevant, feasible, and advantageous, and injects them as explicit state-action guidance through cross-attention. Experiments on D4RL show substantial gains on sparse-reward AntMaze tasks, while improvements are more limited on dense-reward MuJoCo control. Mechanism analyses show that these gains require an informative value representation and the joint use of relevance, feasibility, and advantage in retrieval. These results highlight both the effectiveness and the empirical capability boundary of retrieval-based trajectory stitching.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, trajectory stitching, decision transformer, sequence modeling, retrieval-augmented reinforcement learning, policy improvement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WeFnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReStitch, a retrieval-augmented sequence modeling approach for offline RL that aims to explicitly realize trajectory stitching by retrieving and injecting trajectory continuations into a Decision Transformer via cross-attention. Retrieval uses a utility that combines relevance in a learned IQL value representation, a feasibility gate based on value, and an advantage-weighted score aggregated over the candidate segment. A dual-stream Transformer with distance-aware selective stitching preserves a BC pathway while allowing retrieval-conditioned actions. On D4RL, ReStitch yields substantial gains on sparse-reward AntMaze tasks, with more modest or limited improvements on dense-reward MuJoCo control. Ablations indicate the joint importance of relevance/feasibility/advantage and of representation quality from IQL.

Critical strengths

The paper introduces a clear, instance-based perspective on policy improvement: retrieving explicit, feasible, and advantageous continuations, rather than only conditioning on scalar signals or optimizing auxiliary objectives.

Critical weaknesses
  1. The stitching claim is not established on the comparison or the benchmark that would test it.

The substantial gains' are large relative to DT and QT, which are weak on AntMaze. Relative to the relevant stitching baselines the margin is small. The per-task gaps versus reproduced HIQL are approximately +10.6+10.6, +3.1+3.1, +2.9+2.9, +1.8+1.8, +3.1+3.1, and +2.4+2.4. On medium-diverse, QCS-RTG is slightly higher (79.1 versus 78.6). On large-play and large-diverse, the gaps versus HIQL (3.1 and 2.4) are about one standard error given the reported seed standard deviations. The average of 83.5 versus 79.3 therefore does not, by itself, show a reliable stitching improvement.

The benchmark choice leaves the stitching claim under-tested. D4RL AntMaze is a fixed-goal, sparse-reward task, and behavior cloning fails on its medium and large mazes, so it still measures classical policy improvement over the datasets. It does not isolate the trajectory composition claimed in the title. The play and diverse episodes are long and already chain several commanded goal-reaching segments, and evaluation uses a single goal. Park et al.\ (OGBench, ICLR 2025, Table~1) therefore mark D4RL AntMaze as not testing goal stitching: many tested state-goal pairs already occur together in the training data (Ghugare et al., 2024), and the dataset does not force the agent to assemble short, disjoint segments. OGBench's singletask variants preserve ReStitch's problem formulation, while its stitch datasets limit each trajectory to four cells and its giant mazes extend the horizon. Results on \texttt{antmaze-large-stitch-singletask} and \texttt{antmaze-giant-stitch-singletask} are the direct test of whether retrieval composes segments or imitates long logged paths. They are absent, and the paper does not discuss this choice.

[1] OGBench: Benchmarking Offline Goal-Conditioned RL

[2] Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of View

  1. The implementation does not isolate trajectory-level stitching. The stitching loss regresses only the first action of the highest-utility retrieved segment. The remaining horizon is context for cross-attention. At inference, utility is replaced by latent one-nearest-neighbor cache inheritance. The paper therefore compares a full system with IQL pretraining, reranking, dual heads, and a distance gate against external baselines, without an ablation that separates advantage-weighted action relabeling from the use of an explicit continuation. There is also no measurement that the executed behavior is composed from different dataset trajectories.
Questions for authors
  1. Add two controls on AntMaze: regress the stitching head toward the retrieved action with the trajectory context removed, and train an advantage-weighted behavior-cloning head with no retrieval. Also compare one-nearest-neighbor cache inheritance with recomputing the utility at the query state.

  2. On at least \texttt{antmaze-large-stitch-singletask} and \texttt{antmaze-giant-stitch-singletask}, report success under the OGBench single-task reward.

  3. After a successful AntMaze episode, what fraction of visited state--action pairs comes from distinct source trajectories, and for how many steps does behavior continue to follow the retrieved segment?

  4. Appendix C refers to a distance-gate ablation in Section 5 and contains an unresolved ``Proposition ??''. Please add the ablation or remove the claim. Proposition 2 restates the threshold algebra of wt=exp⁡(−dt/σ)w_t=\exp(-d_t/\sigma). For Proposition~1, please measure how much log⁡Z(s)\log Z(s) varies across retrieved segments.

  5. The indicator V(sk,0)>V(sq)−δV(s_{k,0})>V(s_q)-\delta filters by value. Please provide a quantitative check that accepted anchors are reachable, beyond the single pair in Figure~2.

3 comments
AuthorsOpenReview ↗

Response to WeFn (1/2): Empirical Margin and OGBench Evaluation

We thank the reviewer for the rigorous critique and especially for pointing us to OGBench, which we agree provides a cleaner testbed for strict trajectory stitching.

W1a. Margin over strong baselines. We agree that the improvements over HIQL on antmaze-large-play (74.5 vs. 71.4) and antmaze-large-diverse (80.9 vs. 78.5) are modest relative to cross-seed variability and should not be overstated as statistically conclusive. However, ReStitch achieves a higher average normalized return across the six AntMaze tasks (83.3 vs. 79.3 for HIQL), suggesting a broader performance advantage rather than an isolated improvement. More importantly, our additional experiments on the OGBench stitching benchmarks (Table R5) provide further evidence of ReStitch's effectiveness in settings that explicitly require composing short trajectory segments. On large-stitch, ReStitch achieves 48.89% success compared with 37.22% for HIQL and 28.33% for IQL, while on giant-stitch, it achieves 13.33% compared with 9.89% and 0.00%, respectively. Although these results also exhibit variability, they provide complementary evidence supporting the effectiveness of retrieval-based trajectory stitching. We have revised the manuscript accordingly to moderate our claims and clearly distinguish consistent empirical improvements from statistically conclusive gains.

W1b / Q2. Benchmark choice and OGBench evaluation. We agree with the reviewer and recent literature (Park et al., 2025; Ghugare et al., 2024) that standard D4RL AntMaze datasets contain some long trajectories, making OGBench stitch datasets a cleaner testbed for strict compositional stitching (where dataset trajectories are capped at short segments and never connect start to goal). Following your suggestion, we evaluated ReStitch against IQL and HIQL (fixed-task adaptation) on both antmaze-large-stitch-singletask-task1-v0 and antmaze-giant-stitch-singletask-task1-v0 (Table R5 below).

Evaluation protocol: To match our fixed-task setting, all methods are trained and evaluated on the default fixed task (task1) across 3 random seeds (seeds 0, 1, 2) for 1M gradient steps per seed. For each seed, we evaluate the final 3 checkpoints (800k, 900k, and 1M steps) over 100 full rollout episodes per checkpoint, yielding 3×100=3003 \times 100 = 300 evaluation episodes per seed and 900 episodes per method per task (1,800 episodes total across both tasks). We report the mean success rate and standard deviation across the 3 seeds:

Table R5. Evaluation on OGBench large-stitch and giant-stitch (task1, 900 episodes per method/task)

MethodLarge-stitch successGiant-stitch success
IQL28.33 ± 15.95%0.00 ± 0.00%
HIQL fixed-task adaptation37.22 ± 20.21%9.89 ± 2.34%
ReStitch (Ours)48.89 ± 9.38%13.33 ± 2.73%

Interpretation:

  1. Empirical improvements on Large-stitch: When offline demonstrations are strictly fragmented into short local segments, ReStitch achieves a higher mean success rate (48.89 ± 9.38%) than both IQL (28.33 ± 15.95%, +20.56 percentage points) and HIQL fixed-task adaptation (37.22 ± 20.21%, +11.67 percentage points), with a smaller cross-seed standard deviation on Large-stitch (9.38% vs. 20.21% for HIQL).
  2. Performance on Giant-stitch: On the extreme-horizon Giant-stitch benchmark, standard IQL fails (0.00 ± 0.00%), whereas both HIQL (9.89 ± 2.34%) and ReStitch (13.33 ± 2.73%) achieve non-trivial success (+3.44 percentage points higher mean for ReStitch, though absolute success remains modest for all methods). Because both HIQL and ReStitch rely on IQL-style expectile value learning for long-horizon credit assignment, value degradation over extreme horizons likely remains a shared bottleneck on Giant-stitch.

Overall, Table R5 demonstrates that ReStitch achieves the highest mean success rates on both OGBench stitching benchmarks, providing additional evidence of its effectiveness in settings that explicitly require composing short trajectory segments. The improvements are more pronounced on Large-stitch, while the relatively low absolute success rates on Giant-stitch highlight the remaining challenges of extreme-horizon trajectory stitching.

AuthorsOpenReview ↗

Response to WeFn (2/2): Stitching Mechanism, Ablations, and Theoretical Clarification

W2 / Q1. Isolating trajectory-level stitching from target supervision and cache approximations.

  • Fixed-target controls (Table R1): Holding astitcha_{\text{stitch}} and rankings fixed (H=5H = 5), AWR (advantage-weighted regression on dataset actions without retrieval) obtains 52.6 and 25.2, Target-only (regressing onto astitcha_{\text{stitch}} without retrieved context) obtains 74.2 and 58.4, Anchor-only (H=1H=1) obtains 82.6 and 71.8, and Full ReStitch (H=5H=5) obtains 88.2 and 80.9.
  • Cache inheritance vs. query-time reranking (Table R2): Recomputing utilities or re-retrieving online at test time yields 88.3 and 88.5 (vs. 88.2 for cache inheritance) while increasing latency by +10.4% and +128.9%.
  • Conclusion: ReStitch is not reducible to AWR or memoryless target regression: overall retrieved context conditioning contributes +14.0 to +22.5 over identical target supervision alone (Target-only), of which +5.6 to +9.1 specifically come from the subsequent H−1H-1 continuation tokens beyond the initial anchor pair (Full over Anchor-only).

Q3. Evidence of composition and how retrieved segments are used at inference. To address the reviewer's concerns regarding evidence of trajectory composition and the use of retrieved segments at inference, we clarify how ReStitch operates during rollouts, emphasizing its closed-loop, retrieval-conditioned action generation mechanism and providing additional empirical evidence of trajectory stitching on benchmarks with fragmented offline trajectories:

  1. Closed-loop continuous action generation rather than trajectory copying: At inference (Algorithm 1, Phase 3), the agent does not copy dataset actions or execute a retrieved segment open-loop for multiple steps. Instead, at every step tt, the policy observes a continuous environment state stenvs_t^{\text{env}}, queries FAISS for the nearest cache owner sNNs_{\mathrm{NN}}, and attends jointly over all K=5K=5 inherited segments (M=K×H=25M = K \times H = 25 tokens weighted by wcandw_{\text{cand}}) via Cross-Attention to decode a continuous action astitch=Φstitch(hfused(L))a_{\text{stitch}} = \Phi_{\text{stitch}}(h_{\text{fused}}^{(L)}) before re-querying at step t+1t+1. Consequently, visited (stenv,at)(s_t^{\text{env}}, a_t) pairs are continuously synthesized rather than drawn verbatim from individual dataset trajectories, and the horizon H=5H=5 serves as a lookahead conditioning window rather than a multi-step execution plan.
  2. Evidence of multi-trajectory composition via OGBench stitch (Table R5): Because actions are continuously synthesized from a weighted mixture of K=5K=5 candidates at each step, the cleanest way to verify genuine multi-segment composition—without relying on heuristic post-hoc matching of generated actions to dataset episodes—is to evaluate on benchmarks where no single dataset trajectory spans the task. In OGBench large-stitch and giant-stitch experiments (Table R5 above), every offline trajectory is strictly restricted to short local corridor segments (never connecting start to goal). Achieving 48.89% and 13.33% success on these benchmarks (vs. 28.33% and 0.00% for IQL) provides direct task-level evidence that the policy can leverage guidance from fragmented offline trajectory segments to reach the goal.

Q4. Appendix C and Proposition 1.

  • We fixed the broken reference, added the distance-gate ablations (Tables R3a and R3b), and downgraded Proposition 2 to a brief remark characterizing gate locality.
  • Proposition 1 and log⁡Z(s)\log Z(s): IQL does not fit an explicit behavior density model πβ(a∣s)\pi_\beta(a\mid s). We agree that 1H∑j=0H−1log⁡Z(sj)\frac{1}{H}\sum_{j=0}^{H-1}\log Z(s_j) is not strictly constant across segments and cannot be verified without an auxiliary behavior density model. We have therefore removed the formal “Proposition 1” header and rephrased the derivation as an informal heuristic extension of IQL’s unnormalized AWR weight exp⁡(βA(s,a))\exp(\beta A(s,a)) to HH-step segments.

Q5. Value indicator and reachability. We agree with the reviewer that the condition I[V(sk,0)>V(sq)−δ]\mathbb{I}[V(s_{k,0}) > V(s_q) - \delta] evaluates value compatibility rather than certifying dynamical reachability. In the revised manuscript, we have renamed this component from "feasibility gate" to "value-compatibility condition" throughout the text and removed all claims that it guarantees physical reachability. As detailed in the General Response (Section 3, Table R4), this indicator acts solely as a soft ranking preference that controls the assignment of the advantage bonus (while relevance R(sq,sk,0)R(s_q, s_{k,0}) in the learned value representation ϕV\phi_V and the distance gate wtw_t handle local proximity).

AuthorsOpenReview ↗

Follow-up Response: Baseline Comparisons

2. Scope of Contribution and Comparison with Stronger Single-Task Baselines (ReBRAC, FQL, GFP)

We agree that ReBRAC, FQL, and GFP are essential performance references for single-task antmaze-large-stitch-singletask-task1-v0.

To clarify how these results relate to our core claim, we distinguish between two questions:

  1. Can retrieval unlock trajectory stitching in return-conditioned sequence models (RCSL / DT-style methods)? (Our primary focus)
  2. How does the resulting method compare against state-of-the-art TD-based and flow-based actor–critic algorithms? (Broader performance context)

As established in theoretical and empirical analyses of RCSL (Brandfonbrener et al. [1]; R²CSL [2]), sequence-level return-conditioned methods require return coverage at the initial state distribution and struggle fundamentally when stitching suboptimal fragments. On antmaze-large-stitch, standard RCSL collapses completely (0.0% success for DT). By combining segment retrieval, value-guided scoring, and conditional action prediction, ReStitch raises RCSL performance from 0.0% to 48.89% (and outperforms the standard single-task value-based IQL baseline of 28.2% which we used to guide the retrieve).

At the same time, we explicitly acknowledge that ReStitch (48.89%) remains well below heavily regularized TD actor–critic methods (ReBRAC: 88.4%) and expressive flow-policy actor–critics (FQL: 71.8%, GFP: 90.3%). We do not claim that ReStitch outperforms state-of-the-art actor–critic methods; rather, our goal is to bridge the severe stitching deficit within the RCSL/DT paradigm. We will explicitly include ReBRAC, FQL, and GFP in our comparison table and discuss this paradigm gap as a concrete limitation and direction for future work.


3. Direct RCSL Comparison on OGBench (DT & QCS vs. ReStitch)

To directly test our central thesis on OGBench, we evaluated standard DT and QCS on Large-stitch and Giant-stitch Task1:

MethodLarge-stitch Task1Giant-stitch Task1
DT (context length K=20K=20)0.00 ± 0.00%0.00 ± 0.00%
QCS25.89 ± 11.98%0.00 ± 0.00%
ReStitch (Ours)48.89 ± 9.38%13.33 ± 2.73%

These results confirm that when complete successful demonstrations from the start state are absent in the offline dataset, standard DT fails to achieve any task completions (0.00%), whereas ReStitch consistently stitches fragments to achieve non-trivial success on both Large-stitch (48.89%) and Giant-stitch (13.33%). Together with our controlled ablation studies in the main paper (which isolate the effect of the retrieval module under identical backbones), this provides transparent positioning of both ReStitch's specific improvement over RCSL baselines and its current performance gap relative to strong baselines( HIQL, ReBRAC, FQL).

References

[1] Brandfonbrener et al. When Does Return-Conditioned Supervised Learning Work for Offline Reinforcement Learning? NeurIPS 2022. [2] R²CSL: Return- and Reachability-Conditioned Supervised Learning. arXiv:2506.08463, 2025.

Reviewer_hTw3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the difficulty of composing suboptimal trajectories in return-conditioned sequence models for offline reinforcement learning. This paper first pretrains IQL and constructs a retrieval index using hidden representations from the value network. It then selects trajectory continuations based on state similarity, a value-based condition, and the average advantage over each segment. The retrieved context is incorporated into a Decision Transformer through parallel, multi-layer cross-attention. Separate action heads and a distance-weighted loss are used to learn retrieval-conditioned actions. At inference time, the model inherits cached context from the nearest historical anchor and selects either the stitching head or the base head according to distance. Experiments report improvements over several baselines on AntMaze.

Critical strengths
  1. The method demonstrates clear empirical advantages on AntMaze navigation tasks. In Table 1, ReStitch achieves higher mean scores than HIQL on all six tasks and QCS-RTG on five tasks, reaching 74.5 and 80.9 on large-play and large-diverse, respectively. The authors use a unified evaluation protocol for several key baselines, providing a reasonable basis for comparison.

  2. The paper explicitly reports its limitations and computational costs. The authors acknowledge that ReStitch underperforms several strong baselines on MuJoCo and report end-to-end latency, cache requirements, and GPU memory usage. This allows readers to assess the method’s applicability beyond its positive results. I personally appreciate this transparency.

Critical weaknesses
  1. The central explanation of “feasible trajectory stitching” is not adequately supported by the method’s definition. The value-based condition in Equation (2) compares long-term values but does not establish that a candidate anchor is reachable from the current state. Moreover, the condition only removes the advantage bonus; it does not exclude candidates that fail the condition.

  2. Equations (3)–(5) do not specify temporal encoding. If the implementation likewise lacks such encoding, jointly permuting the retrieved tokens and their corresponding weights would leave the attention output unchanged. The existing sweep over H therefore does not, by itself, establish that the model exploits ordered trajectory continuations rather than simply accessing more state–action samples.

  3. The claim that a single candidate receives over 95% of the attention is directly contradicted by Figure 6. Together with the incorrect average score and the explanation of α that conflicts with the formula, this reduces my confidence in the conclusion that the proposed mechanism has been systematically validated.

Questions for authors
  1. Does the implementation explicitly filter out candidates that fail the value-based condition? If not, what proportion of the selected top-K candidates fail this condition? Does “feasibility” refer specifically to value compatibility or to reachability under the environment dynamics?

  2. Why does the inference-time gate use the distance to the proxy anchor rather than the actual stitching anchor? Please provide the distributions of both distances, the fallback rate, and a comparison against gating based directly on the actual stitching-anchor distance.

  3. Do the retrieved segments include temporal positional encodings or segment identity encodings? How does performance change when temporal order is shuffled while preserving the same tokens and scores, or when only the initial state–action pair is retained?

  4. Please verify the key reported results and evaluation settings, including the average in Table 1, the 95% attention claim associated with Figure 6, the discrepancy between the IQL scores of 71.2 and 47.2 for the same task in Tables 1 and 3, and whether the main tables report the final or the best evaluation window.

2 comments
AuthorsOpenReview ↗

Response to Reviewer hTw3

We thank the reviewer for the meticulous reading and for appreciating our transparency regarding limitations and overhead. We apologize for the inconsistencies and typos identified in the initial submission; all will be corrected in the revised manuscript.

W1 / Q1. Value compatibility vs. dynamical feasibility. The reviewer is completely right on both points: (a) the indicator I[V(sk,0)>V(sq)−δ]\mathbb{I}[V(s_{k,0}) > V(s_q) - \delta] only removes the advantage bonus rather than excluding candidates from the pool, and (b) value compatibility does not guarantee dynamical reachability.

  • As reported in the General Response (Section 3, Table R4), among the selected top-KK (K=5K=5) candidates, 3.4%–6.2% fail the condition relative to the cache owner and 9.6%–15.3% fail relative to the online query (for the top-1 candidate, violation rates are 0.6%–1.5% relative to the owner and 4.8%–8.4% relative to the query).
  • In the revised manuscript, we have renamed this term to "value-compatibility condition" throughout the paper and explicitly stated that it introduces a value-based ranking preference rather than a hard filter, and that it does not guarantee or certify dynamical reachability.

W2 / Q3. Temporal encoding and the role of horizon HH. Our implementation does not use temporal positional encodings in Eqs. (3)–(5); thus, as the reviewer correctly notes, the cross-attention operation is permutation-invariant if the retrieved tokens and their weights wcandw_{\text{cand}} are jointly permuted. We have revised Section 4.2 to explicitly clarify that cross-attention conditions on an unordered collection of state–action transitions extracted from multi-step continuations, rather than modeling their temporal ordering.

Regarding the role of HH (Table R1): We distinguish the ranking horizon (HrankingH_{\mathrm{ranking}}), used to compute candidate rankings and supervision targets, from the context horizon (HcontextH_{\mathrm{context}}), which determines the number of retrieved transitions provided to the model. When candidate rankings and supervision targets astitch∗a^*_{\text{stitch}} are computed once at Hranking=5H_{\text{ranking}}=5 and held fixed (Table R1), removing all context (Target-only, Hcontext=0H_{\text{context}}=0) drops returns from 88.2 ± 1.2 to 74.2 ± 2.4 on medium-play and from 80.9 ± 1.6 to 58.4 ± 3.6 on large-diverse. Providing the initial anchor pairs (sk,0,ak,0)(s_{k,0}, a_{k,0}) (Anchor-only, Hcontext=1H_{\text{context}}=1, sharing the same wcandw_{\text{cand}} as Full) recovers returns to 82.6 ± 1.6 and 71.8 ± 2.3, and including the subsequent H−1=4H-1=4 transition tokens (Full, Hcontext=5H_{\text{context}}=5) yields an additional +5.6 and +9.1 points. Thus, even without explicit temporal order encoding, supplying additional state–action transition samples drawn along the selected high-utility continuations provides denser local state–action context than single anchor pairs alone.

W3 / Q4. Verification of reported results.

  • Table 1 average: Corrected from 83.5 to 83.3.
  • Fig. 6 / App. G: We thank the reviewer for catching these inconsistencies. In Fig. 6, 95% was the combined sum of the two goal-directed candidates (τ3\tau_3 [45%] + τ4\tau_4 [50%]) rather than a single candidate. To avoid any confusion from this single-state K=4K=4 illustration, we have removed Appendix G (Fig. 6) in the revised manuscript and instead rely on our quantitative context ablations in Table R1 to validate context usage.
  • Role of α\alpha: Fixed App. E.4/E.5 to consistently state that larger α\alpha increases the weight on the advantage term.
  • IQL 71.2 (Table 1) vs. 47.2 (Table 3): 71.2 cites the published score from QCS. 47.2 is the score of our standalone Phase-1 IQL checkpoint evaluated under our unified 10-checkpoint running-average protocol without actor-specific action clipping or hyperparameter tuning. Both are now explicitly labeled.
  • Evaluation window: Clarified in Section 5.1 that all reproduced results report the final 10-checkpoint running average across 5 seeds.

Q2. Proxy vs. actual anchor for gating. Please see Tables R3a and R3b in the General Response. Gating on the actual continuation anchor yields 87.8 ± 1.3, 73.9 ± 1.9, and 80.2 ± 1.7 (within 0.4–0.7 of SNNS_{NN} owner-based gating).

Thanks for the authors' reply. Most of my concerns have been addressed. I will maintain my score.

Reviewer_pu7SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReStitch augments a Decision Transformer with offline trajectory retrieval to improve trajectory stitching. Candidate segments are ranked using similarity in a pretrained IQL value representation, a value-threshold indicator, and exponentiated segment-average advantage. A parallel cross-attention stream incorporates retrieved state–action sequences, while separate action heads predict dataset actions and retrieved-action targets. A distance-weighted auxiliary loss trains the stitching head. For efficient inference, the method matches the current state to a dataset neighbor and inherits that neighbor’s cached retrieval contexts.

Experiments cover D4RL AntMaze navigation and MuJoCo locomotion. ReStitch achieves an average AntMaze score of 83.5, compared with 79.3 for HIQL and 77.3 for QCS-RTG. On MuJoCo, its average score is 78.4, below QCS at 90.1. Additional analyses examine retrieval scoring, IQL representation quality, context-integration architecture, retrieval scale, hyperparameters, and computational overhead.

Critical strengths
  1. Meaningful AntMaze improvements over strong baselines. ReStitch improves over both value-guided sequence modeling and hierarchical policy extraction, rather than only outperforming vanilla DT. On large-play and large-diverse, it obtains 74.5 and 80.9, compared with 71.4 and 78.5 for HIQL and 69.2 and 70.3 for QCS-RTG. These results provide substantive evidence that the proposed retrieval-and-supervision mechanism is useful for offline navigation.
  2. Informative analysis of the role of learned value representations. On antmaze-medium-play-v2, the IQL checkpoint experiment shows poor ReStitch performance with an untrained representation and substantial improvement with trained representations. With the final checkpoint, ReStitch reaches 88.2 compared with 47.2 for the corresponding IQL policy. Together with the retrieval-utility ablations, this provides useful evidence about the importance of learned value information and how it is used for policy extraction. The paper also explicitly acknowledges weaker locomotion performance.
Critical weaknesses
  1. The contribution of future trajectory context is not sufficiently isolated from retrieved-action supervision. In Eq. (7), the stitching head is supervised by the first action of a selected segment, while subsequent states and actions enter through cross-attention. This is a reasonable learning formulation, but establishing the trajectory-level contribution requires distinguishing the benefit of future context from the benefit of selecting improved action targets.

    The horizon and architectural ablations provide partial support, but do not fully resolve this distinction. Changing the horizon also changes the segment-average advantage in Eq. (2), potentially altering candidate rankings and supervision targets. The horizon experiment therefore does not hold the retrieval decision fixed while varying future context. Similarly, the architectural comparisons test different context-integration designs rather than isolating the contribution of future continuation tokens under identical supervision. The current evidence does not determine how much improvement comes from retrieved-action supervision versus future trajectory context. This is consequential because explicit trajectory-level guidance is central to the claimed methodological contribution.

  2. The effect of inference-time cache inheritance on continuation compatibility is insufficiently characterized. Appendix F explicitly describes an efficient implementation in which the query inherits the cached candidates and rankings of a nearest-neighbor dataset state, rather than reranking candidates for the query itself. The computational profiling establishes the practicality of this implementation, but does not directly measure the performance impact of the approximation.

    In addition, Eq. (7) and Algorithm 1 use different distance references: training weights supervision using distance to the selected continuation’s anchor, whereas inference selects the action head using distance to the nearest-neighbor cache owner. These mechanisms can serve different purposes, so their difference does not itself imply an incorrect algorithm. However, proximity to the cache owner does not directly measure compatibility with the inherited continuation. The value-threshold term alone also does not establish reachability. The topology and attention examples provide illustrative support, but do not quantify how reliably cache inheritance preserves appropriate continuations across queries. This leaves the proposed stitching mechanism less firmly validated than the aggregate return improvements.

Questions for authors
  1. With candidate rankings and retrieved-action targets held fixed, how does the full model compare with a policy trained on the same targets without retrieval context, and with a model receiving only the retrieved anchor state–action pair? These controls would directly distinguish the contribution of future trajectory information from action-target selection.
  2. What is the performance difference between nearest-neighbor cache inheritance and reranking candidates for the current query at inference? Does the approximation have a larger effect near maze bottlenecks?
  3. Could you clarify the respective roles of the training and inference distance gates? Among decisions where the inference gate selects the stitching head, what is the distribution of distances to the selected continuation’s anchor? Does gating against that anchor change performance? This would help distinguish detection of departure from dataset support from control of continuation mismatch.

These questions concern the attribution and validation of the present method, rather than requests for broader future work.

1 comment
AuthorsOpenReview ↗

Response to Reviewer pu7S

We thank the reviewer for the precise and constructive questions, which directly shaped our new controlled ablations.

W1 / Q1. Isolating future context from action-target selection.

We agree that varying HH in the original sensitivity sweep simultaneously alters Aˉ(τ)\bar A(\tau) and therefore changes candidate rankings and stitching targets astitcha_{\text{stitch}}. In our new control (General Response, Table R1), candidate rankings and astitcha_{\text{stitch}} are computed once using H=5H=5 and held strictly fixed across variants. We compare:

  1. AWR (no retrieved-action supervision, no stitching loss): 52.6 ± 3.0 (medium-play) and 25.2 ± 7.2 (large-diverse).
  2. Target-only (trained on the exact same astitcha_{\text{stitch}} targets and gating rules, but with zeroed context inputs): 74.2 ± 2.4 and 58.4 ± 3.6.
  3. Anchor-only (retaining only the initial pair (sk,0,ak,0)(s_{k,0}, a_{k,0}) of each of the K=5K=5 segments): 82.6 ± 1.6 and 71.8 ± 2.3.
  4. Full ReStitch (retaining all H=5H = 5 steps per segment): 88.2 ± 1.2 and 80.9 ± 1.6.

Conclusion: While retrieval-guided target supervision accounts for a substantial primary jump over AWR (+21.6 and +33.2 points), removing retrieved context entirely (Target-only) still causes a severe drop of 14.0 and 22.5 points relative to Full, and including the subsequent H−1H-1 continuation tokens improves over Anchor-only by +5.6 and +9.1 points. We have calibrated our claims in the Abstract, Section 1, and Section 4 to reflect that ReStitch's gains arise synergistically from both retrieval-based target supervision and multi-step context conditioning.

W2 / Q2. Effect of cache inheritance. Query-time utility recomputation requires only ϕV(sq)\phi_V(s_q) and V(sq)V(s_q) since segment advantages Aˉ(τk)\bar A(\tau_k) are precomputed offline, making it inexpensive (+0.223 ms/step for cached-candidate reweighting and +2.772 ms/step for full online re-retrieval and reranking). Table R2 compares the strategies on antmaze-medium-play-v2: updating the normalized utility weights wcandw_{\text{cand}} on the inherited set yields 88.3 ± 1.2 (Δ=+0.1\Delta = +0.1), and full online re-retrieval yields 88.5 ± 1.3 (Δ=+0.3\Delta = +0.3, well within 1 std of 88.2 ± 1.2). Because the owner-based distance gate already restricts stitching to queries within a tight local neighborhood of sNNs_{\mathrm{NN}}, the inherited candidates and weights closely match the query-specific optimum, confirming that cache inheritance achieves virtually the same performance at less than half the latency.

Q3. Roles of the training and inference distance gates. The two gates serve complementary purposes: during training, wtw_t (measured to sstitchs_{\text{stitch}}) downweights supervision from distant anchors to prevent target mismatch; at inference, the gate (measured to sNNs_{\mathrm{NN}}) restricts the policy to state–context pairings supported by the offline cache. As reported in Tables R3a and R3b:

  • Among active stitching decisions, the distance to the cache owner has a median (P50) of 1.54–1.86 and a 90th percentile (P90) of 2.86–3.19 (bounded by r=3.684r = 3.684), while the distance to the actual continuation anchor sstitchs_{\text{stitch}} has a P50 of 1.82–2.26 and a P90 of 3.79–4.42, exceeding rr in 10.90%–17.53% of accepted steps.
  • Gating directly on the actual continuation anchor (Actual-anchor gate, without recalibrating σ\sigma and τsafe\tau_{\text{safe}}) yields 87.8 ± 1.3, 73.9 ± 1.9, and 80.2 ± 1.7 across the three tasks—within 0.4–0.7 points (well inside one standard deviation) of the default owner-based gate (88.2 ± 1.2, 74.5 ± 1.8, and 80.9 ± 1.6).
  • The overall fallback rate to the base head is 11.84%, 16.42%, and 18.75% across the three tasks. Removing inference fallback (Always stitch head) drops performance by 4.4 to 7.4 points with higher variance, confirming the protective role of distance-gated fallback.

Comments on the paper

8 comments
AuthorsOpenReview ↗

General Response (1/6): Overview

We thank all reviewers for their careful and constructive feedback. We were encouraged that reviewers found the AntMaze improvements meaningful over strong baselines (pu7S, hTw3), the value-representation analysis informative (pu7S, WeFn), and the reporting of limitations and costs transparent (hTw3).

The reviews converge on three core questions. We address each with new controlled experiments below. All changes in the revised PDF will be highlighted in blue.

AuthorsOpenReview ↗

General Response (2/6): Fixed-target Context Ablation

1. Is the gain due to retrieved trajectory context or to retrieved-action supervision? (pu7S, hTw3, WeFn)

We agree that the original horizon sweep (HH) confounds context length with candidate ranking, since Aˉ(τ)\bar A(\tau) depends on HH. To isolate the contribution of retrieved trajectory context from that of retrieved-action supervision, we conducted an additional fixed-target experiment. Candidate rankings and stitching targets astitch∗a^*_{\text{stitch}} are computed once (H=5H=5) and held fixed across all variants. Only the conditioning information given to the model changes.

For these supplementary experiments, we implement three variants to disentangle the contribution of retrieved context and action supervision:

  • Target-only replaces all retrieved state–action inputs with zeros and sets candidate weights uniformly to 1/K1/K, while retaining the retrieval branch, the original retrieved-action targets, the training loss weights, and the inference head-selection rule. Thus, retrieval still determines supervision and gating, but candidate-specific information is removed from the model’s context inputs.
  • Anchor-only retains only the initial state–action pair (sk,0,ak,0)(s_{k,0}, a_{k,0}) of each of the K=5K=5 retrieved segments. Candidate ranking still uses the complete H=5H=5 segments, and the selected action targets and gating rules remain unchanged, isolating the contribution of subsequent context tokens (j=1,…,H−1j=1,\dots,H-1).
  • AWR uses the same causal backbone and base action head to predict dataset actions, minimizing an advantage-weighted mean squared error with w(s,a)=min⁡{exp⁡(10A(s,a)),100}w(s,a)=\min\{\exp(10A(s,a)),100\}, where A(s,a)=min⁡(Q1,Q2)(s,a)−V(s)A(s,a)=\min(Q_1,Q_2)(s,a)-V(s) is computed using the frozen critic. These weights are not additionally normalized within a batch. This deterministic AWR-style baseline uses neither retrieved-action supervision nor a stitching loss and always executes the base head without retrieval-based gating.

All three variants are trained from scratch with the exact same policy training budget and evaluated over 5 seeds. The results are summarized in Table R1. Table R1 reveals a consistent performance hierarchy across both AntMaze tasks: AWR < Target-only < Anchor-only < Full ReStitch. These results help distinguish the contributions of retrieval-guided supervision, retrieved anchor information, and additional continuation context.

Table R1. Fixed-target context ablation (normalized return, mean ± std over 5 seeds)

DatasetFullTarget-onlyAnchor-onlyAWR
antmaze-medium-play-v288.2 ± 1.274.2 ± 2.482.6 ± 1.652.6 ± 3.0
antmaze-large-diverse-v280.9 ± 1.658.4 ± 3.671.8 ± 2.325.2 ± 7.2

Analysis of Table R1:

  1. Retrieval-guided target supervision vs. in-trajectory AWR (Target-only vs. AWR): Even with zeroed context inputs, training the stitching head on retrieved cross-trajectory targets astitch∗a^*_{\text{stitch}} with distance gating (Target-only: 74.2 and 58.4) substantially outperforms in-trajectory advantage-weighted regression (AWR: 52.6 and 25.2), suggesting that retrieval-guided target supervision and its associated stitching mechanism provide a strong policy-improvement signal.
  2. Necessity of retrieved context conditioning (Full vs. Target-only): Under the exact same targets astitch∗a^*_{\text{stitch}} and gating rules, removing context inputs (Target-only) degrades performance by 14.0 on medium-play and 22.5 on large-diverse while substantially increasing cross-seed variability relative to Full (88.2 and 80.9). This demonstrates that the policy does not merely memorize target actions parametrically; it actively relies on candidate-specific context conditioning via Cross-Attention to resolve multi-modal ambiguity and adapt to state offsets during cache inheritance.
  3. Contribution of multi-step continuation tokens (Full vs. Anchor-only): Providing initial anchor pairs (sk,0,ak,0)(s_{k,0}, a_{k,0}) (Anchor-only: 82.6 and 71.8) recovers most state-level location information (+8.4 and +13.4 over Target-only), while incorporating the remaining H−1=4H-1=4 continuation steps (Full) yields a further gain of +5.6 and +9.1 by providing richer trajectory-level state–action context.

Overall, these results support the central design of ReStitch: retrieval-guided supervision provides a strong learning signal, while candidate-specific conditioning and additional continuation transitions contribute further improvements. Importantly, the fixed-target comparisons show that ReStitch's gains cannot be explained solely by changing retrieved-action targets; the information provided to the policy through retrieval also plays an important role.

AuthorsOpenReview ↗

General Response (3/6): Cache Inheritance vs. Query-time Reranking

2. Inference-time cache inheritance and distance gating (pu7S, hTw3, WeFn)

2.1 Cache inheritance vs. query-time reranking

The utility in Eq. (2) requires only ϕV(sq)\phi_V(s_q) and V(sq)V(s_q) for the online query, while segment advantages Aˉ(τk)\bar A(\tau_k) are defined on static dataset segments and precomputed offline. Consequently, query-time utility recomputation is computationally feasible: reranking the inherited candidates adds 0.223 ms per step (+10.4%), while full online ANN re-retrieval plus reranking adds 2.772 ms per step (+128.9%).

To assess whether the computational savings of cache inheritance come at the expense of policy performance, we evaluate inference-time retrieval on antmaze-medium-play-v2, keeping the trained policy, critic, and offline cache fixed without retraining:

  • Cache inheritance (ReStitch baseline): Inherits the K=5K=5 candidates and normalized weights stored for the nearest-neighbor cache owner sNNs_{\mathrm{NN}}.
  • Cached-candidate reranking: Loads the owner's cached K=5K=5 candidate segments, recomputes their utilities using the current query's feature similarity ϕV(sq)\phi_V(s_q) and value estimate V(sq)V(s_q) (paired with precomputed candidate values and segment advantages), and updates their ordering and normalized weights without an additional ANN search.
  • Re-retrieval and reranking: Performs a fresh ANN search over the full dataset index using the current query (retrieving 30 candidates, filtering out incomplete segments or those from the owner's episode, and doubling the pool if fewer than 5 valid candidates remain), recomputes current-query utilities over the pool, and selects and normalizes the top-KK (K=5K=5) segments.

Table R2. Cache inheritance vs. query-time reranking on antmaze-medium-play-v2 (K=5,H=5)

Inference methodNormalized scoreInference latency, ms/stepLatency increase
Cache inheritance (ReStitch)88.2 ± 1.22.150—
Cached-candidate reranking88.3 ± 1.22.373+10.4% (1.10×)
Re-retrieval and reranking88.5 ± 1.34.922+128.9% (2.29×)

Takeaway: Because the owner-based distance gate already restricts stitching to queries within a tight neighborhood of sNNs_{\mathrm{NN}}, inherited utilities closely match query-specific utilities. Online re-retrieval and reranking yields negligible score differences (+0.1 to +0.3 points, well within 1 std) while more than doubling per-step latency.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.