acceptodds
Withdrawn from ICLR 2027

Mitigating the Need for Bellman Completeness: Multi-Fragment Trajectory Stitching for Decision Transformer

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The Decision Transformer (DT) has achieved promising results in offline reinforcement learning by reframing it as sequence modeling. However, DT fundamentally struggles with trajectory stitching—the ability to combine trajectory fragments into globally superior trajectories. This limitation stems from two intertwined factors: (i) DT is a supervised learning model whose policy space is confined to the support of the training data distribution, and (ii) offline datasets, especially suboptimal ones, often lack coverage of the compositional trajectories that stitch together high-return fragments. In contrast, dynamic programming (DP)-based approaches, such as Q-learning, naturally support trajectory stitching through value iteration. However, they rely on Bellman completeness and accurate function approximation, which can be challenging to satisfy in practice. To bridge this gap, we propose Multi-Fragment Trajectory Stitching (MFTS), a diffusion-based data augmentation framework that constructs synthetic trajectories by stitching high-return fragments from suboptimal data. MFTS stitches high-return fragments together by generating forward and backward state trajectories with a pre-trained conditional diffusion model, then fusing them via a time-aware asymmetric blending strategy. The resulting bridge is completed with actions and rewards through auxiliary models. MFTS thus grants DT the benefits of dynamic programming while mitigating the need for Bellman completeness. Extensive experiments on the D4RL benchmark demonstrate that MFTS achieves superior or competitive performance in most settings, with the largest gains on challenging medium-replay datasets where stitching matters most.

Primary area:
reinforcement learning
Keywords:
Offline Reinforcement Learning, Decision Transformer, Trajectory Stitching, Diffusion Models, Data Augmentation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nyuuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MFTS, a diffusion-based data augmentation method for improving trajectory stitching in Decision Transformers. It connects high-return trajectory fragments through bidirectional diffusion and asymmetric fusion, then trains DT on the augmented data. The method shows improvements on several D4RL tasks, particularly in medium-replay settings.

Critical strengths

The paper addresses a meaningful limitation of Decision Transformers. The motivation is clear, and the results on medium-replay datasets are promising.

Critical weaknesses
  1. Limited algorithmic novelty. The overall approach is closely related to existing works such as DiffStitch [1] and BiTrajDiff [2], which have already explored diffusion-based trajectory stitching and bidirectional generation. MBRCSL [3] has also studied trajectory stitching without Bellman completeness. The proposed multi-fragment design mainly extends pairwise stitching, while the asymmetric fusion uses a simple, predefined linear weighting scheme. I find the technical novelty rather limited.

  2. Lack of global consideration. The method independently generates bridges between adjacent fragments without considering the quality of the complete trajectory. Moreover, fragments are ordered solely by local returns, which does not necessarily guarantee their temporal or dynamical compatibility.

  3. Insufficient ablations. The paper does not examine how the number of fragments affects performance. In particular, the trajectory-level stitching gain analysis only considers two fragments, leaving the actual benefit of multi-fragment stitching unclear.

  4. Incomplete comparisons. Although BiTrajDiff [2] is discussed in the paper, it is not included in the experiments. The evaluation is also limited to MuJoCo locomotion tasks, making it difficult to assess the generality of the method.

References

[1] Li et al. DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-Based Trajectory Stitching. 2024.

[2] Qing et al. BiTrajDiff: Bidirectional Trajectory Generation with Diffusion Models for Offline Reinforcement Learning. 2026.

[3] Zhou et al. Free from Bellman Completeness: Trajectory Stitching via Model-Based Return-Conditioned Supervised Learning. 2023.

Questions for authors
  1. How does performance change as the number of fragments increases? Does multi-fragment stitching consistently outperform pairwise stitching?
  2. Why are fragments ordered solely by local returns? Would considering their compatibility lead to better results?
  3. Can the authors provide a comparison with BiTrajDiff to better establish the contribution over existing methods?
  4. How sensitive is the method to the predefined fusion weights? Would other weighting schedules or learned weights achieve similar results?
Reviewer_XnMpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a generalization problem of offline reinforcement learning: Because sequence models like decision transformer are confined to the support of the training data distribution, it cannot combine locally optimal fragments from suboptimal datasets into globally superior trajectories. This inability of the sequential model to connect sub trajectories into new ones is addressed through so called trajectory stitching, which attempts to create new and correct trajectories from existing imperfect ones. Dynamic programming methods can stitch trajectories, but have strict assumptions such as the Bellman completeness. The method introduced bypasses these with a diffusion-based data augmentation framework for Multi-Fragment Trajectory Stitching. To do this, MTFS performs:

1- Fragment Extraction: Selects top−ktop-k trajectories based on discounted return to form an expert buffer, then samples and sorts fragments by local return. 2- State Trajectory Generation: Uses a conditional diffusion model to generate forward and backward state sequences between the terminal state of one fragment and the initial state of the next. 3- Time-Aware Asymmetric Fusion: Fuses the forward and backward state sequences using a monotonically increasing weight profile to resolve bidirectional conflicts. 4- Action & Reward Completion: Employs auxiliary MLPs (inverse dynamics and reward predictors) to infer missing actions and rewards for the fused state bridge. 5- Dual Filtering: Applies a post-hoc filter using a 1-step dynamics model and a reward discriminator to discard physically implausible transitions.

In a nutshell, MTFS extracts high-return fragments, generates bridging states using a conditional diffusion model with a novel time-aware asymmetric fusion of forward and backward trajectories, and completes the actions and rewards via auxiliary models. A dual-filtering mechanism ensures dynamic consistency, effectively granting DT the stitching capabilities of DP without requiring value iteration or Bellman completeness. The newly generated trajectory is stored in the augmented dataset used to train the DT.

The authors report experiments in the D4RL showing that the

Critical strengths

1- The problem is well motivated. DT’s inability to stitch is real, and the data-augmentation route is an active area of research 2- Keeping source fragments as real transitions and generating only bridges is a sensible design. It limits model error to the bridge, which is an advantage over full-model-rollout approaches. 3- The ablation design is reasonable: fusion strategy, window length, data ratio, and a trajectory-level metric. Results are reported with std over seeds, and the appendix has implementation detail and code. 4- The Hopper-MR gain over DiffStitch+DT, and the stability argument, is at least plausible given the bidirectional conditioning

Critical weaknesses

Method limitations. 1- MFTS grounds its motivation in the theoretical limitations of supervised sequence modeling, specifically highlighting the "Bellman Completeness" requirement. Yet, its core generation mechanism—the time-aware asymmetric fusion—is mostly a heuristic convex combination designed to mitigate conflicting signals in diffusion generation. 2- Fragments come from top-k whole trajectories by return. This cannot harvest good segments from poor trajectories, which is the classic stitching scenario. The method then largely bridges between already-good trajectories. Adaptive fragment selection is deferred to future work. 3- The diffusion, inverse-dynamics and reward models are themselves trained on D and confined to its support. The limitation (i) the authors attribute to DT applies to them as well. Using inverse dynamics on generated states can lead to action-state misalignment and compounding errors. The issue has been identified in (Yu et al. 2025). Inverse dynamics is ill-posed for locomotion, and wrong actions become behavior-cloning targets for DT. Action clipping is not discussed.

Yu, H., Zhang, D., Du, Q., Zhao, Y., Zhang, H., Chen, G., ... & Zhao, J. (2025). Astro: Adaptive stitching via dynamics-guided trajectory rollouts. arXiv preprint arXiv:2511.23442.

4- MFTS relies on a Temporal U-Net for its state diffusion model. The auxiliary models (inverse dynamics, reward predictor, dynamics checker, and reward discriminator) are implemented as standard MLPs.

Experimental limitations 5- It is evaluated exclusively on the D4RL benchmark (MuJoCo locomotion tasks) using Decision Transformer as the downstream algorithm. 6- No value-based baselines (CQL, IQL) appear despite the DP framing. No DT-variant baselines are compared (CGDT, which is discussed, or EDT). BiTrajDiff, the most closely related method, is also not compared. 7- The reported scores e.g., in Hopper-MR seem lower than the originally reported ones 8- Table 1 mixes numbers from the original papers (RvS, QDT) with reproductions (DT, DiffStitch+DT). 9- There seem to be heavy per-dataset tuning (r_test from 0.35 to 0.92, n_opt, k_top, similarity threshold), with no indication that DT and DiffStitch received equal tuning. 10- ASTRO’s “SI” baseline is essentially MFTS’s design. ASTRO’s “state planner + inverse dynamics” ablation (Table 2, Table 4, Fig. 5c) is the same family as MFTS (generate states, then recover actions with inverse dynamics). ASTRO reports it as the weakest completion strategy: 17.4% dynamics violations and action MSE of 0.226, against 5.3% and 0.103 for its full method

Questions for authors

1- The rationale about the Bellman completeness, brought up as a main reasoning is deferred to the Appendix, where “MFTS does not rely on Bellman completeness because there is no value iteration” only holds in the narrow sense. The method depends on the realizability and accuracy of the diffusion, inverse-dynamics, dynamics and reward models, which is where the assumptions went. The aspects of the inverse-dynamics raised above should be considered. It is also necessary to clarify how MTFS differs from ASTRO's SI baseline.

2- Only Gym-MuJoCo medium, medium-replay and medium-expert are used. QDT, MBRCSL and DiffStitch use Maze2D, AntMaze, Kitchen or point-maze, where stitching is required. The authors should describe the support for the claim that the method “grants DT the benefits of dynamic programming”.

3- It seems MFTS only manages high return trajectories and would not use information from imperfect trajectories. ( Top-k trajectories by return, fragments of length W sorted by local return) Please clarify.

Reviewer_5SajOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method for trajectory synthesis by stitching together individual expert trajectories. Experimental results show that the synthesized data achieves certain effectiveness on DT.

Critical strengths

Well-written and very easy to follow

Critical weaknesses
  1. This work is fundamentally about data synthesis, yet the authors in the introduction continuously emphasize how DT performs and highlight its limitation of lacking stitching capability, causing the core content of the paper to deviate somewhat from what was discussed in the introduction.
  2. The authors claim that their method can stitch multiple trajectory segments while DiffStitch cannot, but I find this statement inaccurate. DiffStitch stitches segments within a dataset, and the stitched trajectories themselves can be divided into multiple parts. Therefore, I believe DiffStitch is essentially equivalent to multi-segment stitching. Thus, the only difference between this work and DiffStitch lies in using a bidirectional diffusion model when generating intermediate transition segments. Based on the above discussion, I think there is no fundamental distinction between this work and DiffStitch.
  3. The experimental section includes relatively few baselines and RL methods for comparison. From the perspective of data augmentation, this work only compares with DiffStitch and tests solely on DT.
Questions for authors

Is this paper focused on an algorithm improved based on DT, or is it a data augmentation work?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.