acceptodds
Under review as a conference paper at ICLR 2027

Trajectory Editing as Posterior Inference for Offline Reinforcement Learning

Abstract

Offline RL performance depends on the quality of trajectories in a fixed dataset. Nevertheless, practical datasets often contain structural deficiencies along two axes: spatial gaps, where useful locomotion segments lack connecting transitions, and temporal sparsity, where manipulation demonstrations under-sample critical phases such as contact, grasping, and alignment. These deficiencies limit downstream policy learning, reducing returns or success rates. We introduce trajectory editing to repair offline data without environment interaction by bridging disconnected segments or densifying critical phases. The key challenge is to satisfy the prescribed anchors of each edit while remaining within the support of the offline dataset and consistent with its dynamics. We propose Bayesian Inference for Trajectory Editing (BITE), a unified framework that treats the edited trajectory as a latent variable to be inferred. BITE learns a prior over plausible trajectories from the offline dataset and constructs an anchor-consistency likelihood from the edit specification, which induces a posterior that trades off data plausibility and edit satisfaction. We instantiate this prior with a diffusion trajectory model and derive a lifted posterior sampler that propagates clean-level edit observations through the reverse diffusion chain using backward sequential Monte Carlo. The resulting posterior inference procedure achieves sampling consistency as the particle population grows and its edit posteriors are provably as accurate as the learned prior, with no dependence on any edit distribution. Experiments on D4R locomotion and RoboMimic manipulation tasks show that BITE delivers the strongest performance gains among baselines. Our code is publicly available at https://anonymous.4open.science/r/bite-055D.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, Diffusion Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cB7ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Bayesian Inference for Trajectory Editing (BITE), a framework for improving offline RL by repairing structural deficiencies in fixed datasets. BITE addresses two settings: spatial editing, which connects disconnected trajectory fragments, and temporal editing, which densifies critical phases of manipulation demonstrations. The method combines a diffusion trajectory prior, an anchor-consistency likelihood, and a transition-consistency potential, and uses backward Sequential Monte Carlo (SMC) for posterior inference. Experiments on D4RL, RoboMimic, and OGBench demonstrate improvements across several offline RL algorithms.

Critical strengths
  1. An important problem and an interesting unified formulation. The paper addresses an important problem in offline RL: improving the quality of fixed datasets without additional environment interaction. The distinction between spatial gaps and temporal sparsity is well motivated. The proposed Bayesian formulation unifies spatial stitching and temporal densification through a reusable trajectory prior, with editing constraints imposed at inference time rather than through a separately trained conditional editor. Although several components build on existing diffusion posterior sampling and SMC techniques, this unified perspective is interesting.

  2. Strong empirical results with useful controls. The empirical results are strong across multiple downstream learners and benchmarks, which suggests practical value. The paper also includes useful ablations and controls, such as matched-data-budget experiments, reward relabeling, dynamics guidance, and computational cost analysis. The theoretical analysis provides useful insights into posterior sampling consistency and the approximation-error differences between Bayesian and amortized trajectory editing.

Critical weaknesses
  1. The theoretical claims appear stronger than what is established.

    • Lifted versus original posterior. The relationship between the lifted posterior and the original Bayesian editing posterior is unclear. Equation (8) introduces additional likelihoods and transition potentials across diffusion levels, which may change the clean-level marginal. Theorem 1 only establishes consistency for the clean-level marginal of this constructed lifted posterior, without showing that it recovers the originally intended editing posterior.
    • Average-case bound. Proposition 1 controls the expected posterior approximation error under the data-induced edit distribution m⋆, rather than providing guarantees for individual rare or unseen edits. Moreover, the practical editing requests are constructed through return-based cross-trajectory selection, whose distribution is not shown to match m⋆.

    As a result, the theoretical guarantees do not fully support the paper's claims, particularly regarding the lifted posterior and rare editing requests.

  2. The baseline comparison and the attribution of the empirical gains are unresolved.

    • Baseline provenance. The baseline results in Table 1 appear identical to those reported in DiffStitch [1], including their standard deviations. However, DiffStitch's Table 1 caption reports three random seeds, whereas this paper's Table 1 caption states four. It should be clarified which results were reused or rerun, and whether the comparison protocols were consistent.
    • Data selection and reweighting. Spatial editing selects fragments from return-ranked trajectory pools, with augmented data accounting for 70% of samples in each training minibatch, and the matched-data-budget control does not isolate this selection effect. A comparison against oversampling the same selected original fragments without editing would better establish the contribution of the proposed method.
    • Temporal baselines. Table 2 compares BITE against the original datasets but does not include simpler alternatives such as uniform densification, interpolation, or oversampling selected critical segments. These comparisons would help clarify whether the improvements come from Bayesian editing or simply from emphasizing critical phases.

    As a result, the experiments do not sufficiently isolate the contribution of trajectory editing from data selection and reweighting.

[1]: Li, Guanghe, et al. "Diffstitch: Boosting offline reinforcement learning with diffusion-based trajectory stitching." ICML 2024.

Questions for authors
  1. Does the lifted posterior in Equation (8), after marginalizing over the intermediate diffusion levels and averaging over the auxiliary observations, recover the original clean-level Bayesian editing posterior? If so, can the authors provide a formal justification?
  2. How does Proposition 1 apply to the practical return-selected editing requests, whose distribution is not shown to match m⋆?
  3. The baseline results in Table 1 match those reported in DiffStitch, but the reported seed counts differ. Can the authors clarify which results were reused or rerun, and whether the evaluation protocols were matched?
  4. On one representative mixed-quality setting, can the authors compare spatial editing against oversampling the same selected high-return fragments without generating new transitions?
  5. On one representative RoboMimic task, can the authors include one simpler temporal augmentation baseline, such as uniform densification or oversampling the selected original segments, at a matched training budget?
Reviewer_mmDbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses structural deficiencies in offline reinforcement learning datasets by proposing Bayesian Inference for Trajectory Editing (BITE). The method combines a diffusion-based trajectory prior with an anchor-consistency likelihood and a learned dynamics model to generate plausible trajectories through posterior inference. Experiments on D4RL demonstrate improvements in downstream policy performance.

Critical strengths

1.The paper addresses an important practical challenge in offline RL by repairing incomplete trajectories without additional environment interaction. 2.The Bayesian formulation is clearly presented, and the experiments demonstrate promising performance improvements across multiple downstream RL algorithms.

Critical weaknesses

1.The fundamental concern is that the method attempts to repair missing transitions using a trajectory prior and dynamics model learned from the same incomplete dataset. Since these models may be inaccurate in under-covered regions, there is no guarantee that the generated trajectories satisfy the true environment dynamics. More direct evaluations of physical feasibility in unseen or poorly covered regions are needed. 2. The theoretical analysis provides an expected posterior approximation error bound under a data-induced distribution of editing instances. However, this does not guarantee reliable performance for individual rare or unseen editing conditions. Further theoretical clarification and empirical validation are needed to support the claimed advantages in these challenging settings. 3.The proposed framework closely resembles existing diffusion posterior sampling and inpainting approaches, which combine learned generative priors with observation constraints. The authors should better clarify the technical novelty beyond adapting these techniques to offline RL and provide more direct comparisons with related methods. 4.The proposed backward sequential Monte Carlo sampler introduces additional computational overhead, particularly as the number of particles and trajectory horizon increase. More comprehensive scalability and efficiency analyses are needed to demonstrate its practicality in high-dimensional, long-horizon tasks.

Questions for authors

Please refer to the Weaknesses section for my questions and concerns.

Reviewer_wxzEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies trajectory editing for offline reinforcement learning, addressing two structural deficiencies in offline data: missing connections between useful behavioral fragments and insufficient temporal coverage of critical phases in manipulation demonstrations. The proposed method, BITE, unifies spatial stitching and temporal expansion as posterior inference combining a trajectory prior with an anchor-observation likelihood. The method learns a diffusion prior over state-action trajectories, uses a learned one-step dynamics model to construct a transition-consistency potential, and introduces auxiliary observations at different diffusion levels to impose editing constraints. A backward sequential Monte Carlo sampler with an analytically tractable Gaussian proposal then generates edited trajectories. Rewards are subsequently predicted, and the resulting trajectories augment the data used to train downstream policies. The theoretical analysis establishes asymptotic sampling consistency for the defined lifted target and compares the average posterior errors of Bayesian and amortized editing. The evaluation includes PointMaze reconstruction, D4RL locomotion, RoboMimic manipulation, and additional long-horizon OGBench tasks in the appendix. The reported results show improvements across multiple downstream learners. The appendix also examines dynamics guidance, particle count, reward sources, and computational cost.

Critical strengths
  1. A unified problem formulation with a concrete, implementable inference procedure. Expressing anchor constraints through a linear observation operator allows spatial stitching and temporal expansion to share a trajectory prior and an inference framework. Combining Gaussian reverse transitions with a linear-Gaussian likelihood gives an analytically tractable edit-guided proposal, while the local dynamics potential supplies additional one-step consistency information. This design has practical value for tasks that must reuse existing offline data.

  2. Broad downstream evaluation with meaningful control experiments. In the main D4RL table, BITE achieves the best or tied-best result in 26 of the 27 task-learner combinations. On RoboMimic, the overall average success rate increases from 61.5% to 67.6%. The appendix additionally includes controls using equally sized additional datasets, alternative reward sources, dynamics-guidance ablations, and long-horizon tasks. These results support the method's ability to improve data utility, with benefits extending beyond reconstruction metrics or a single downstream learner.

Critical weaknesses

The temporal-editing experiments do not sufficiently isolate the contribution of BITE's particular inference mechanism. The main RoboMimic experiments compare the original data with BITE-augmented data. This demonstrates that the overall pipeline can be beneficial, but does not distinguish gains from high-entropy segment selection, slowing down critical phases, reallocating training samples, or the proposed posterior sampler. The matched-size data control in the appendix focuses on Hopper and therefore does not directly address attribution in temporal editing for manipulation. To substantiate the central claim of repairing temporal sparsity, targeted comparisons are needed under matched edited segments, dataset sizes, and training budgets. Relevant controls include simple temporal expansion, increased sampling weight for high-entropy segments, or interpolation with actions re-estimated. Without such comparisons, the evidence more strongly supports the effectiveness of the complete augmentation pipeline than the necessity of the proposed inference mechanism.

Questions for authors

The following questions request clarification or validation of the current claims. Their answers could materially affect my assessment.

  1. What is the relationship between Eq. (8) and the original editing posterior? Please provide the complete joint distribution of the auxiliary observations and a derivation showing that marginalizing the lifted path target recovers the clean-level editing posterior. If the two targets are not equivalent, please explain the additional reweighting introduced by the lifted target and whether Proposition 1 still applies to the actual BITE sampling distribution.
  2. Does Proposition 1 provide only a distribution-averaged guarantee? If no additional instance-wise error bound is available, should the claims concerning "every edit posterior" and rare edits be narrowed? How does the edit-request distribution produced by return-based and similarity-based anchor selection relate to m* in the proposition?
  3. Do the benefits of temporal editing persist under targeted controls? With segment selection, augmentation size, and training budget held fixed, does BITE retain a clear advantage over simple temporal expansion or increased sampling of high-entropy segments? Also, what exactly does "replaying" segments at a slowdown rate in Appendix F.2 entail? Is it implemented entirely through offline index remapping and model-based generation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.