- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Bayesian Inference for Trajectory Editing (BITE), a framework for improving offline RL by repairing structural deficiencies in fixed datasets. BITE addresses two settings: spatial editing, which connects disconnected trajectory fragments, and temporal editing, which densifies critical phases of manipulation demonstrations. The method combines a diffusion trajectory prior, an anchor-consistency likelihood, and a transition-consistency potential, and uses backward Sequential Monte Carlo (SMC) for posterior inference. Experiments on D4RL, RoboMimic, and OGBench demonstrate improvements across several offline RL algorithms.
Critical strengths
-
An important problem and an interesting unified formulation. The paper addresses an important problem in offline RL: improving the quality of fixed datasets without additional environment interaction. The distinction between spatial gaps and temporal sparsity is well motivated. The proposed Bayesian formulation unifies spatial stitching and temporal densification through a reusable trajectory prior, with editing constraints imposed at inference time rather than through a separately trained conditional editor. Although several components build on existing diffusion posterior sampling and SMC techniques, this unified perspective is interesting.
-
Strong empirical results with useful controls. The empirical results are strong across multiple downstream learners and benchmarks, which suggests practical value. The paper also includes useful ablations and controls, such as matched-data-budget experiments, reward relabeling, dynamics guidance, and computational cost analysis. The theoretical analysis provides useful insights into posterior sampling consistency and the approximation-error differences between Bayesian and amortized trajectory editing.
Critical weaknesses
-
The theoretical claims appear stronger than what is established.
- Lifted versus original posterior. The relationship between the lifted posterior and the original Bayesian editing posterior is unclear. Equation (8) introduces additional likelihoods and transition potentials across diffusion levels, which may change the clean-level marginal. Theorem 1 only establishes consistency for the clean-level marginal of this constructed lifted posterior, without showing that it recovers the originally intended editing posterior.
- Average-case bound. Proposition 1 controls the expected posterior approximation error under the data-induced edit distribution m⋆, rather than providing guarantees for individual rare or unseen edits. Moreover, the practical editing requests are constructed through return-based cross-trajectory selection, whose distribution is not shown to match m⋆.
As a result, the theoretical guarantees do not fully support the paper's claims, particularly regarding the lifted posterior and rare editing requests.
-
The baseline comparison and the attribution of the empirical gains are unresolved.
- Baseline provenance. The baseline results in Table 1 appear identical to those reported in DiffStitch [1], including their standard deviations. However, DiffStitch's Table 1 caption reports three random seeds, whereas this paper's Table 1 caption states four. It should be clarified which results were reused or rerun, and whether the comparison protocols were consistent.
- Data selection and reweighting. Spatial editing selects fragments from return-ranked trajectory pools, with augmented data accounting for 70% of samples in each training minibatch, and the matched-data-budget control does not isolate this selection effect. A comparison against oversampling the same selected original fragments without editing would better establish the contribution of the proposed method.
- Temporal baselines. Table 2 compares BITE against the original datasets but does not include simpler alternatives such as uniform densification, interpolation, or oversampling selected critical segments. These comparisons would help clarify whether the improvements come from Bayesian editing or simply from emphasizing critical phases.
As a result, the experiments do not sufficiently isolate the contribution of trajectory editing from data selection and reweighting.
[1]: Li, Guanghe, et al. "Diffstitch: Boosting offline reinforcement learning with diffusion-based trajectory stitching." ICML 2024.
Questions for authors
- Does the lifted posterior in Equation (8), after marginalizing over the intermediate diffusion levels and averaging over the auxiliary observations, recover the original clean-level Bayesian editing posterior? If so, can the authors provide a formal justification?
- How does Proposition 1 apply to the practical return-selected editing requests, whose distribution is not shown to match m⋆?
- The baseline results in Table 1 match those reported in DiffStitch, but the reported seed counts differ. Can the authors clarify which results were reused or rerun, and whether the evaluation protocols were matched?
- On one representative mixed-quality setting, can the authors compare spatial editing against oversampling the same selected high-return fragments without generating new transitions?
- On one representative RoboMimic task, can the authors include one simpler temporal augmentation baseline, such as uniform densification or oversampling the selected original segments, at a matched training budget?