acceptodds
Under review as a conference paper at ICLR 2027

VE-JEPA: Joint Latent and Value Learning for Offline Reinforcement Learning

Abstract

Predictive representations are typically assessed by how accurately they predict. We examine whether predictive accuracy is informative about downstream control in offline reinforcement learning. We introduce VE-JEPA, a planning-free joint-embedding predictive architecture in which a single encoder is trained through latent prediction, reward prediction, and expectile value learning, with an advantage-weighted policy operating directly on encoded observations. Across three medium-replay locomotion tasks and five seeds per condition, allowing Q and V losses to update the encoder substantially improves control over prediction-only and prediction-plus-reward encoder training. Ablations show that latent prediction improves reward prediction from frozen encoder features, while its incremental control benefit varies across tasks and seeds. Encoder learning rate also affects control performance. Reducing the encoder learning rate tenfold raises mean AntMaze medium-play success from 5.0% to 36.0%, averaged across seven training seeds. Measurements on fixed observations show reduced representation movement despite larger encoder-gradient norms and higher predictive loss. Extensions across locomotion datasets reveal a training-budget tradeoff: slower encoder updates can delay performance improvements at a fixed budget. Across nine D4RL locomotion datasets with ten seeds per method and dataset, VE-JEPA achieves an aggregate interquartile mean normalized return of 74.7 against Decision ConvFormer's 71.5 at 200,000 training steps under fixed-target evaluation, a difference of with a 95% bootstrap confidence interval of . This comparison varies across data regimes: VE-JEPA leads by 13.8 points on medium-replay and trails by 8.8 on medium-expert; the overall ordering is sensitive to target-return selection. These results highlight the importance of value-informed encoder training and task-dependent encoder adaptation when learning predictive representations for offline control.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, representation learning, joint-embedding predictive architectures, value learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ixSvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VE-JEPA, an offline RL architecture that jointly trains an encoder through latent prediction, reward prediction, and IQL value learning. A separate policy is trained using advantage weights and acts directly on encoded observations without planning. Experiments on D4RL locomotion and AntMaze demonstrate the importance of value supervision and examine how the benefits of latent prediction and slower encoder updates vary across tasks and training budgets.

Critical strengths
  • The paper compares different encoder training objectives while continuing to train all component heads. These experiments provide useful evidence that reward prediction alone does not recover the control performance obtained when Q and V losses also update the encoder.

  • The main locomotion evaluation uses ten seeds per method and dataset, and the selected encoder learning rate is evaluated on seven additional seeds on AntMaze medium-play. Confidence intervals and analyses of sensitivity to training budget and target return selection help clarify the robustness and scope of the findings.

Critical weaknesses
  • The additional control benefit of latent prediction is not clearly established. In Table 5, the confidence intervals include zero on Hopper and HalfCheetah, while most of the improvement on Walker2d comes from one seed. It remains unclear what latent prediction contributes beyond value supervision.

  • The representation diagnostics do not sufficiently explain how Q and V supervision changes the encoder to improve control. Moreover, the findings are limited to IQL on D4RL locomotion and AntMaze, leaving their generality across other value learning objectives and tasks unclear.

Questions for authors
  • Could you compare VE-JEPA with an existing JEPA or predictive representation method under the same downstream IQL objectives and access to task rewards? The separate T-JEPA studies in Appendix D do not establish the relative benefit of the predictive objective within the VE-JEPA architecture. A comparison that controls for encoder architecture and training budget would help clarify what the proposed approach contributes beyond existing representation learning methods.

  • How do Q and V losses separately affect the information preserved by the encoder? Could you evaluate reward prediction from frozen representations when Q and V losses are allowed to update the encoder separately, while keeping all heads trained? Controlling for shared gradient clipping would help distinguish representation changes from effects on other model components.

  • Do the findings on value supervision and encoder learning rate hold with another offline RL method and on OGBench [1] single task manipulation environments? Repeating the central encoder supervision and prediction removal comparisons would help establish whether the findings extend beyond IQL and D4RL.

  • Appendix E reports an IQM of 74.7 for DC with selected target returns, compared with 74.2 for VE-JEPA at the exact 200K checkpoint. Does the reported advantage over DC persist when conditioning targets are selected using separate validation evaluations and both methods use the same checkpoints and number of evaluation episodes?

[1] Park, Seohong, et al. "Ogbench: Benchmarking offline goal-conditioned rl." International Conference on Learning Representations. 2025.

Reviewer_6PsXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

VE-JEPA combines latent prediction, reward prediction, IQL-style value learning, and an advantage-weighted policy. On state-based D4RL tasks, the paper investigates which objectives should supervise the encoder and how its learning rate affects control. Value supervision produces substantial improvements, while the additional benefit of latent prediction and slower encoder updates is task dependent.

Critical strengths
  • Clear evidence for value supervision within the proposed architecture. Adding Q/V gradients to prediction-plus-reward encoder training improves Hopper from 9.5 to 93.9 and Walker2d from 9.2 to 82.5, with improvements across all matched seeds.

  • Careful uncertainty and sensitivity analysis. The main comparison uses ten seeds, and the encoder-rate finding is confirmed on seven additional seeds: AntMaze medium-play success increases from 5% to 36%. The paper transparently reports dependence on training budget, evaluation protocol, and individual seeds.

Critical weaknesses
  • The motivation does not establish a clear unresolved research problem. Lambert et al., Objective Mismatch in Model-based Reinforcement Learning (2020), already demonstrate that predictive accuracy need not correlate with control. Babaeizadeh et al., Models, Pixels, and Rewards (2020), systematically investigate image/reward prediction versus control, including fixed-data experiments, and find that prediction quality can be informative in their setting. Neither is cited. The relevant open question is therefore which conditions determine this relationship. The present collection of ablations does not identify those conditions or explain what uncertainty in the existing literature it resolves.

  • Closely related representation-learning methods are omitted. TCRL (Zhao et al., ICML 2023) already studies planning-free control from predictive representations, joint versus separate value training, prediction objectives, and EMA versus shared-encoder targets. Its experiments show that joint value training can introduce instability. TD7/SALE (Fujimoto et al., NeurIPS 2023) learns predictive embeddings for state-based online and offline RL, separates encoder optimization from value learning, and uses periodically updated encoder copies. Neither is cited or compared. These works make the missing scientific question concrete: when does joint value supervision help, and when does separating representation and value learning work better? The paper does not isolate the roles of target updates, normalization, raw-state access, or optimization in explaining its findings.

  • The benchmark coverage is insufficient to establish a substantial empirical contribution. Nine datasets derived from three D4RL locomotion environments, plus three AntMaze datasets, provide limited diversity in observations, dynamics, and task structure. I would require evaluation on the complete single-task OGBench suite used by FQL: 50 state-based and five pixel-based tasks covering AntMaze, HumanoidMaze, AntSoccer, Cube, Scene, and Puzzle. Comparisons should include current baselines such as FQL and DFQL, alongside ReBRAC, TD7, and capacity-matched IQL where applicable, with comparable training and tuning budgets. The principal encoder-supervision, prediction-removal, and learning-rate ablations should be repeated across environment families. The separate Atari pipeline does not establish visual VE-JEPA performance.

  • The additional value of the predictive architecture remains unclear. Removing latent prediction produces uncertain differences on Hopper and HalfCheetah, while approximately 74% of the Walker2d gain comes from one seed. The large improvements from enabling Q/V gradients therefore establish the importance of value supervision in this implementation more convincingly than the usefulness of its predictive component. Without a matched shared-encoder IQL control and comparisons to the omitted predictive-encoder designs, the paper does not establish a transferable representation-learning finding.

Questions for authors
  • Can you explain the different outcomes of joint versus separate value/representation training in VE-JEPA, TCRL, and TD7/SALE through controlled comparisons of encoder targets, raw-state access, normalization, and gradient clipping within the same offline setting?

  • Can you evaluate VE-JEPA on the complete 55-task OGBench suite used by FQL, against contemporary baselines, and repeat the central ablations across navigation, manipulation, and visual tasks?

  • Across this broader evaluation, under what identifiable conditions does latent prediction improve control over a matched value-learning baseline, and which finding goes beyond the established observation that prediction quality alone is an insufficient proxy?

Reviewer_Hp5COpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether predictive fit of a learned representation is informative about downstream control in offline RL. The authors propose VE-JEPA, which trains a state encoder jointly with a latent dynamics model, a reward head, twin critics, and an expectile value function (IQL-style), and extracts a policy by AWR on detached features. An encoder-objective ablation on three medium-replay tasks shows that letting Q and V losses update the encoder is what drives performance, while the latent prediction loss contributes little to control. The authors’ central takeaway is that predictive accuracy is not informative about control: value supervision, not latent prediction, is what makes the encoder useful, and a slower encoder can improve AntMaze success even while predictive loss rises and effective rank falls. They therefore argue that predictive representations should be evaluated by downstream control rather than predictive fit.

Critical strengths

The attempt to address how the representation objective affects offline RL performance is a reasonable question.

Critical weaknesses
  1. The literature review is extremely insufficient, and as a result, the novelty and significance of the paper are unclear. References contain only 14 papers in total, and none of them engage directly with the paper’s core questions. It should have been the authors’ responsibility to position the paper against the thorough and proper literature, but below I note some classic works where the overlap lies.

a. [1] studied which objectives on offline D4RL data produce useful state representations for downstream offline RL, including latent forward prediction, reward prediction, and frozen vs. fine-tuned vs. jointly trained encoders. Several of its conclusions are in tension with Sections 5.2–5.3 and Appendix D. More broadly, the question of which auxiliary objective should shape an RL encoder has been studied systematically, e.g., by searching the space of auxiliary losses in [2]; the three hand-picked encoder objectives compared in Section 5.2 are not positioned against that line of work.

b. [3] and [4] analyze the optimization of self-predictive representations: stop-gradient targets, the two-timescale relationship between predictor and encoder, and representation collapse. The encoder learning-rate result in Section 5.4 is presented as a discovery, but it is the configuration those works study. [4] (Theorem 2) also argues that a reward head is redundant once a value loss and latent prediction train the encoder end-to-end, which is the conclusion of Section 5.2.

c. The architecture, a latent-dynamics and reward objective on the encoder with value learning on top and no planning, is the design of MR.Q [5], TD7 [7], and the more recent [6]. In those online methods, the encoder trained by prediction and reward alone performs well.

Given [1]–[7], I could not identify a finding in the paper that is new, and the paper itself does not tell the reader which of its results it considers new relative to this literature.


  1. The empirical evaluation is weak and does not support the claims.

a. All methods are trained for 200K steps rather than the offline conventional 1M. The paper’s own Figure 8 shows VE-JEPA still improving at 200K, and the 1M-step extension in Appendix E reverses the DC ordering. The reported IQL numbers are far below CORL’s benchmarking [8], and the DC reproduction is far below the published DC (Table 10).

b. The baselines are outdated. There is no comparison with at least ReBRAC [8] or QT [9], both of which report D4RL locomotion scores well above VE-JEPA’s 74.7 IQM at the standard budget.

c. The analysis experiments are too narrow for the stated conclusions. The encoder-objective ablation in Section 5.2, which carries the paper’s main message, uses only the three medium-replay tasks with five seeds each; whether the finding holds on medium, medium-expert, or AntMaze data is never tested. The encoder learning-rate multiplier in Section 5.4 is selected on a three-seed sweep on AntMaze medium-play only.


  1. The paper is hard to read.

a. There is no formal problem setup; Section 3.1 opens with loss functions. The rollout weight ρtρ^t and validity mask mtm_t in Equations (2)–(4) are never explained in the text.

b. Nearly every result is followed by a disclaimer about what it does not show, to the point where I could not identify what the paper claims. For example, Sections 5.3, 5.5, and Appendix D read like lists of negations.

c. The content needed to judge the main claims (1M-step results, DC target sensitivity, encoder-rate extension, learning curves) is in the appendix.


References:

[1] Representation Matters: Offline Pretraining for Sequential Decision Making (Yang and Nachum, 2021)

[2] Reinforcement Learning with Automated Auxiliary Loss Search (He et al., 2022)

[3] Understanding Self-Predictive Learning for Reinforcement Learning (Tang et al., 2023)

[4] Bridging State and History Representations: Understanding Self-Predictive RL (Ni et al., 2024)

[5] Towards General-Purpose Model-Free Reinforcement Learning (Fujimoto et al., 2025)

[6] NASDAQ: Normalized Observation Space Dynamics-Augmented Q-Learning (Liu et al., 2026)

[7] For SALE: State-Action Representation Learning for Deep Reinforcement Learning (Fujimoto et al., 2023)

[8] CORL: Research-oriented Deep Offline Reinforcement Learning Library (Tarasov et al., 2023)

[9] Q-value Regularized Transformer for Offline Reinforcement Learning (Hu et al., 2024)

Questions for authors
  1. Regarding Weakness 1, could the authors state which findings they consider new relative to [1-7], and in particular explain why their conclusions may or may not reconcile with those references?

  2. Regarding Weakness 2, could the authors report all methods at 1M steps on all nine locomotion datasets and six AntMaze datasets, including at least ReBRAC and QT, and account for the gap between their IQL and DC numbers and those in CORL and the DC paper?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.