- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper combines historical trajectory encoding and probabilistic prediction of future representations with a Decision Transformer, using AdaLN to inject the predicted latent into action generation. It first performs behavior cloning and JEPA pretraining on reward-free trajectories, then uses critic guidance to adapt online to tasks with new rewards. The theoretical results comprise a variational lower bound on mutual information and, for a fixed predictor, a finite-sample return bound for convergence toward the population behavior-cloning policy.
Critical strengths
- Table 1 on p. 8 covers 19 D4RL locomotion and navigation datasets and reports means and standard deviations over five random seeds. Based on the reported means, the method performs best on 10 of 12 locomotion datasets and exceeds PDT on 18 of 19 datasets.
- The component comparisons are reasonably extensive. Table 2 on p. 9 examines a deterministic predictor, AdaLN versus token conditioning, critic guidance, and a frozen backbone. Table 7 on p. 34 additionally equips PDT with AdaLN and the same type of critic-based adaptation.
Critical weaknesses
- Equation (67) on p. 26 defines the HalfCheetah backward reward as −max{vₓ,3} − 0.1‖a‖². For every vₓ≤3, the velocity term is constantly −3, so standing still and running backward receive the same velocity reward. The formula therefore does not encourage backward locomotion, contradicting the subsequent description of opposite horizontal directions.
- Theorem 2 analyzes a fixed predictor under assumptions including a finite policy class, bounded occupancy mismatch, and independent sampling. It bounds the learned policy's error relative to a population predictive-conditioned policy. It does not analyze joint predictor–policy learning or online optimization under a new reward, and therefore does not directly establish an advantage over RTG/PDT or improved downstream sample efficiency.
Questions for authors
- What backward reward does the actual code use? Do the main transfer conclusions hold after correcting the formula or implementation?
- Please specify, for each table, the offline and online stages, environment steps, gradient steps, resources available for baseline adaptation, and normalization references for the new rewards.
Response to Reviewer CmXq
We thank the reviewer for the constructive comments.
Thank you for highlighting this ambiguity. We clarify that our implementation does not introduce leakage of action targets. For each position m, the historical encoder processes only the available causal history τm−K+1:m=(sm−K+1,am−K+1,…,sm−1,am−1,sm) to obtain Hm. Then, the predicted future latent are given by F^m∼qη(⋅∣Hm). The action is finally predicted as a^m=πϕ(τm−K+1:m,F^m). Accordingly, the BC objective applies position-specific action supervision, i.e., K1∑m=t−K+1tℓBC(a^m,am).
We clarify that it does not use a shared window-end latent to supervise earlier actions. We have revised the ambiguous loss formulation and its description to explicitly clarify this causal alignment.
We thank the reviewer for identifying these issues regarding the comparability of the adaptation results and the backward reward specification.
(a) Reward Access and Online Interaction Budgets.
Following the evaluation protocol of PDT, we aim to investigate whether our reward-free pretraining-based DT-JEPA can achieve competitive downstream performance compared with reward-supervised pretraining methods. We thus include reward-supervised baselines, such as Prompt DT and ODT, which can access reward label during offline pretraining phase. During downstream online adaptation, all methods adopt same interaction budget of 200k online transitions to ensure a fair comparison. We have clarified the evaluation protocols, baseline selection, and interaction budgets in the revised manuscript.
Table R1: Comparison of training resources and online adaptation protocols.
(b)Verification of Backward Locomotion.
For the backward-motion reward, we have corrected this typographical error. The implemented HalfCheetah backward reward is rbackward=max(−vx,3)−0.1∣a∣22 to encourage backward locomotion, which has been corrected in the revised manuscript. We evaluate the policies trained under the forward and backward rewards using their mean forward velocity and backward-step ratio, demonstrating the effectiveness of our reward settings.
Table R2: Forward/Backward locomotion evaluation on HalfCheetah.
We hope our clarification is helpful, and we would be grateful if you could consider increasing your score!