acceptodds
Under review as a conference paper at ICLR 2027

DT-JEPA: Learning to Predict Future Representations for Decision-Making

Abstract

Decision Transformer (DT) has demonstrated strong performance in various offline reinforcement learning (RL) tasks by conditioning action generation on expected future returns and historical trajectories. However, treating expected future returns as the condition provides an incomplete description of the future, as trajectories with different state transitions and behaviors may share similar returns. Moreover, expected future returns inherently depend on task-specific reward functions, which makes the policy learning reliant on reward-labeled datasets and restricts its scalability to tasks with similar but unseen rewards. Recently, Joint Embedding Predictive Architectures (JEPA) has emerged as a promising paradigm of self-supervised learning-based world models, which learns predictive representations of future dynamics in latent space rather than reconstructing future states in the original space. Inspired by JEPA, we propose DT-JEPA, an offline pretraining and online finetuning framework that learns general knowledge from reward-free offline trajectories and achieve excellent adaptation to the unseen tasks. Instead of predicting a single deterministic future embedding, we design a novel probabilistic predictor to model the conditional distribution of future representations in the latent space, thereby capturing both predictive information and uncertainty about future dynamics. The predicted future representation is then utilized to guide DT's action generation. To generalize to tasks with unseen reward objectives, we further develop a critic-guided fine-tuning mechanism to efficiently adapt the pretrained policy to unseen reward functions with limited online interactions. Theoretically, we show that 1) the JEPA objective seeks to maximize a variational lower bound on the mutual information between historical representations and future trajectories; and 2) predictive future-conditioned policy learning admits a finite-sample performance bound with an estimation term. Experiments on D4RL benchmarks demonstrate that DT-JEPA achieves efficient adaptation across varying offline RL tasks, outperforming traditional DT-based and other offline RL methods.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Decision Transformer, Joint-Embedding Predictive Architectures, World Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YpLZOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper combines historical trajectory encoding and probabilistic prediction of future representations with a Decision Transformer, using AdaLN to inject the predicted latent into action generation. It first performs behavior cloning and JEPA pretraining on reward-free trajectories, then uses critic guidance to adapt online to tasks with new rewards. The theoretical results comprise a variational lower bound on mutual information and, for a fixed predictor, a finite-sample return bound for convergence toward the population behavior-cloning policy.

Critical strengths
  1. Table 1 on p. 8 covers 19 D4RL locomotion and navigation datasets and reports means and standard deviations over five random seeds. Based on the reported means, the method performs best on 10 of 12 locomotion datasets and exceeds PDT on 18 of 19 datasets.
  2. The component comparisons are reasonably extensive. Table 2 on p. 9 examines a deterministic predictor, AdaLN versus token conditioning, critic guidance, and a frozen backbone. Table 7 on p. 34 additionally equips PDT with AdaLN and the same type of critic-based adaptation.
Critical weaknesses
  1. Equation (67) on p. 26 defines the HalfCheetah backward reward as −max{vₓ,3} − 0.1‖a‖². For every vₓ≤3, the velocity term is constantly −3, so standing still and running backward receive the same velocity reward. The formula therefore does not encourage backward locomotion, contradicting the subsequent description of opposite horizontal directions.
  2. Theorem 2 analyzes a fixed predictor under assumptions including a finite policy class, bounded occupancy mismatch, and independent sampling. It bounds the learned policy's error relative to a population predictive-conditioned policy. It does not analyze joint predictor–policy learning or online optimization under a new reward, and therefore does not directly establish an advantage over RTG/PDT or improved downstream sample efficiency.
Questions for authors
  1. What backward reward does the actual code use? Do the main transfer conclusions hold after correcting the formula or implementation?
  2. Please specify, for each table, the offline and online stages, environment steps, gradient steps, resources available for baseline adaptation, and normalization references for the new rewards.
Reviewer_CmXqOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

DT-JEPA replaces return-to-go conditioning in a Decision Transformer with predicted future representations. A probabilistic predictor learns embeddings from an EMA target encoder, and adaptive layer normalization incorporates the predicted latent into the policy. Reward-free pretraining is followed by reward-guided online adaptation using selected high-return trajectories and twin critics. Experiments cover locomotion, navigation, and changed-reward tasks, with component ablations and strengthened PDT baselines testing the proposed design.

Critical strengths
  1. The pipeline combines reward-free trajectory learning with later reward-guided adaptation. Predicting the future representation from available history avoids requiring realized future observations at deployment. This addresses settings where trajectories are available before the downstream objective is specified.

  2. The main evaluation covers 19 datasets with five independent training seeds, and DT-JEPA exceeds PDT's mean on 18 of them. The enhanced PDT baseline, which includes both adaptive layer normalization and critic guidance, also has a lower reported mean. This control provides evidence for a contribution beyond these two shared components.

Critical weaknesses
  1. The conditioning and training-loss descriptions leave the causal visibility of action targets unclear. The window history includes past actions, while the window-level formulation appears to supervise actions using a latent derived from the end of that window; for earlier positions, this could expose the action target through the auxiliary pathway despite causal masking in the transformer. Position-specific causal latents or current-action-only supervision could address this issue; the written formulation alone does not establish that the implementation leaks action targets.

  2. The reported protocol leaves uncertainty about the comparability of the adaptation results. Methods differ in reward access and adaptation settings, while the reported episode-based curves do not fully specify the resources used for each main comparison; additionally, the printed backward-motion reward has a velocity term that is constant below the stated threshold. These details affect the interpretation of the implemented task and the attribution of performance differences to the representation under comparable resources.

Questions for authors

See Weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer CmXq

We thank the reviewer for the constructive comments.

[W1]

Thank you for highlighting this ambiguity. We clarify that our implementation does not introduce leakage of action targets. For each position mm, the historical encoder processes only the available causal history τm−K+1:m=(sm−K+1,am−K+1,…,sm−1,am−1,sm)\tau_{m-K+1:m}=(s_{m-K+1},a_{m-K+1},\dots,s_{m-1},a_{m-1},s_m) to obtain HmH_m. Then, the predicted future latent are given by F^m∼qη(⋅∣Hm).\hat F_m\sim q_\eta(\cdot|H_m). The action is finally predicted as a^m=πϕ(τm−K+1:m,F^m)\hat a_m=\pi_\phi(\tau_{m-K+1:m},\hat F_m). Accordingly, the BC objective applies position-specific action supervision, i.e., 1K∑m=t−K+1tℓBC(a^m,am).\frac{1}{K}\sum_{m=t-K+1}^{t} \ell_{\mathrm{BC}}(\hat a_m,a_m).

We clarify that it does not use a shared window-end latent to supervise earlier actions. We have revised the ambiguous loss formulation and its description to explicitly clarify this causal alignment.

[W2]

We thank the reviewer for identifying these issues regarding the comparability of the adaptation results and the backward reward specification.

(a) Reward Access and Online Interaction Budgets.

Following the evaluation protocol of PDT, we aim to investigate whether our reward-free pretraining-based DT-JEPA can achieve competitive downstream performance compared with reward-supervised pretraining methods. We thus include reward-supervised baselines, such as Prompt DT and ODT, which can access reward label during offline pretraining phase. During downstream online adaptation, all methods adopt same interaction budget of 200k online transitions to ensure a fair comparison. We have clarified the evaluation protocols, baseline selection, and interaction budgets in the revised manuscript.

Table R1: Comparison of training resources and online adaptation protocols.

MethodOffline RewardsOffline TrainingOnline Fine-tuningOnline Transitions
TD3-BC✓✓✓200k
TD-JEPA✗✓✓200k
ODT✓✓✓200k
QT✓✓✓200k
Prompt DT✓✓✓200k
VDT✓✓✓200k
LSDT✓✓✓200k
PDT✗✓✓200k
DT-JEPA✗✓✓200k

(b)Verification of Backward Locomotion.

For the backward-motion reward, we have corrected this typographical error. The implemented HalfCheetah backward reward is rbackward=max⁡(−vx,3)−0.1∣a∣22r_{\mathrm{backward}}=\max(-v_x,3)-0.1|a|_2^2 to encourage backward locomotion, which has been corrected in the revised manuscript. We evaluate the policies trained under the forward and backward rewards using their mean forward velocity and backward-step ratio, demonstrating the effectiveness of our reward settings.

Table R2: Forward/Backward locomotion evaluation on HalfCheetah.

RewardMean Velocity (m/s)Backward-step Ratio (%) ↑
Forward5.6460.81
Backward-5.57599.34

We hope our clarification is helpful, and we would be grateful if you could consider increasing your score!

Reviewer_ZP33OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DT-JEPA predicts future trajectory representations from observed history and uses them to condition a Decision Transformer. It pretrains without reward labels, then adapts to new rewards using online data and critic guidance. The paper reports D4RL results, changed-reward experiments, and ablations of prediction, conditioning, and adaptation.

Critical strengths
  1. The ablations support combining prediction with policy adaptation. Table 2 reports an average normalized score of 87.3 for DT-JEPA, compared with 75.8 for deterministic prediction and 80.4 without critic guidance. Table 7 also strengthens PDT with the same conditioning and critic-guided adaptation, but its average remains 79.5 (pages 9 and 34). This is a helpful control for the comparison with PDT.

  2. The experiments cover several offline dataset qualities, new reward objectives, and data with high-return trajectories removed. This gives a broader test of the method than evaluation only on expert demonstrations (pages 8–9 and 31).

Critical weaknesses
  1. The paper needs evidence that the learned distribution predicts meaningful futures. Slowly updating the target encoder does not by itself prevent representations from collapsing. The training loss also omits the target-entropy term in the information bound while the target changes. Showing that different latent samples change actions establishes that the policy uses them, but not that they accurately describe possible futures. Held-out prediction results and a clear account of how collapse is prevented would support the central interpretation (pages 5 and 27–28).

  2. The evaluation has unresolved definitions. Table 1 does not clearly specify online interaction budgets or reward access for each method, and the changed-reward results do not give their normalization references. The HalfCheetah backward reward in Equation 67 has a constant velocity term whenever forward velocity is at most 3, so it does not favor backward motion over standing still (page 26). Whether this is a printing error or the reward actually used matters directly to the claimed adaptation result.

Questions for authors
  1. What prevents representation collapse, and how well does the predictor perform on held-out trajectories? Does each action use a latent computed only from the history available before that action?

  2. What reward access and interaction budgets are used for each Table 1 method, and how are returns normalized for the new rewards? Can you provide variation across runs for Tables 2 and 7?

  3. Is Equation 67 the implemented HalfCheetah backward reward? If so, what evidence shows backward locomotion rather than standing still?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.