acceptodds
Withdrawn from ICLR 2027

JEPA-x: Cross-Predictive Physics Grounding for Forecastable Latent Dynamics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Latent world models plan by predicting the outcomes of candidate actions in a learned representation space. In self-predictive models, however, the encoder and predictor are optimized jointly and can co-adapt to latent transitions that are easy to predict in the training distribution but weakly constrained by the physical evolution of the scene. We introduce the cross-predictive JEPA (JEPA-x), which grounds latent dynamics in privileged physical trajectories. JEPA-x treats visual observations and physical states as corresponding views of the same action-conditioned trajectory, advances both through a shared predictor, and matches each prediction to the future representations of both modalities. This encourages the action-conditioned predictor to learn a common transition rule across the two views. Privileged physical state is used only during training, leaving a visual-only model at deployment. Empirical results show that JEPA-x reduces the rollout drift of a newly fitted predictor from 0.361 to 0.104 and increases mean control success from 53.6% to 78.2% on a multi-task suite spanning six evaluation subfamilies. We additionally show that direct full-state regression improves position decodability without improving forecastability or control, indicating that the benefit comes from shaping latent dynamics rather than merely encoding physical variables.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
JEPA, physical grounding, world model

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_g1NxOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes the cross-predictive JEPA (JEPA-x), which incorporates privileged physical states into JEPA training to constrain the model to follow physical evolution. Its learning objective consists of four source–target terms that let the encoder and predictor see representations from both observations and physical states. JEPA-x also adopts a unified privileged-state interface, which allows it to operate across different task environments without task-specific labels. Experiments show that JEPA-x achieves better prediction ability and planning performance in several simulated control environments.

Critical strengths

The unified privileged-state interface is an interesting design. It enables a single trained JEPA-x model to handle diverse manipulation scenes without task-specific labels.

Critical weaknesses
  1. The assumption of access to privileged physical states, even if only during training, is strong and limits the real-world applicability of JEPA-x.
  2. The experimental evidence is insufficient. JEPA-x is never tested on unseen environments or objects, and I do not think the current architecture can handle them. The ablations also lack the simplest variant, which only learns an encoder from Z^o_t to s_t and a predictor s{t+1} = gφ(s_t, a_t); this variant does not need SIGReg to avoid collapse. Moreover, comparing only against LeWM is not convincing: since JEPA-x has access to more information, its better performance is expected.
Questions for authors
  1. You designed the privileged-state interface on the encoder side so that JEPA-x can adapt to diverse environments. However, the ground-truth dynamics should also differ across environments, even with the same objects. Should the same design also be considered for the dynamics?
  2. Some related work (Klindt et al., 2026; Zhang et al., 2026) has shown that, under certain assumptions, JEPA can recover the physical states up to a linear transformation from observations alone, whereas your work requires the physical states to be known. How do you view this difference?
Reviewer_BWifOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a physics-grounded JEPA model that uses privileged physical states during training. The method introduces a physical encoder alongside the visual encoder, with a shared action-conditioned predictor trained to predict future representations across both modalities. Experiments demonstrate improved control success and latent-dynamics forecastability compared with visual-only and alternative physical-supervision baselines.

Critical strengths
  1. Physics-grounded predictive representations. Rather than enforcing physical feasibility or making physical states decodable, the paper uses privileged physical trajectories to shape the evolution of visual latent representations. The supportive results in Figure 4 and Table 1 offers a useful insight for world-model learning.
  2. Intuitive formulation and clear logical chain. The cross-predictive objective is not practically heavy to implement and avoids a separate teacher–student distillation pipeline.
Critical weaknesses
  1. Limited generalizability of the rigid-body physical representation. The privileged-state interface relies on canonical SDF geometry and rigid-body poses. Such representations are less straightforward for deformable objects (e.g., cloth) or complex physical interactions (e.g., splashing). As far as I am concerned, all experiments use rigid-body simulation, leaving the applicability of the proposed approach to broader physical systems unclear.
  2. Limited evaluation under realistic physical-state uncertainty. All experiments use simulated trajectories, and obtaining such accurate privileged supervision from real-world data may be challenging. The paper does not evaluate sensitivity to noise or estimation errors in these training-time physical states, making it difficult to assess applicability beyond simulation.
  3. The effectiveness of cross-prediction requires further investigation. Table 2 shows that ALIGN-ONLY already achieves 72.6% control success, compared with 78.2% for JEPA-x. Furthermore, CROSS-ONLY achieves comparable forecastability with JEPA-x (0.101 vs 0.104) despite not sharing the predictor. It seems that the relative importance of cross-prediction and predictor sharing is not fully established, especially given the evaluation over only three training seeds.
Questions for authors
  1. How sensitive is JEPA-x to the choice of (1) privileged physical features and to (2) noise or estimation errors in training-time physical states?
  2. Could the authors provide stronger evidence for the incremental benefit of cross-prediction and predictor sharing over ALIGN-ONLY and CROSS-ONLY?
Reviewer_F9nEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes JEPA-x, a training objective for visual latent world models that uses privileged physical states during training. Visual observations and structured physical states are encoded into a common latent space. A shared action-conditioned predictor operates on either modality’s history, and each prediction is matched to both future representations. The physical encoder is discarded at deployment. The main evaluation trains one model across 22 manipulation configurations. JEPA-x improves average control success from 53.6% to 78.2% across six evaluation families. Freezing the visual encoders and fitting new predictors also reduces normalized rollout drift from 0.361 to 0.104. Ablations and four additional single-task benchmarks examine the contributions of alignment, cross-prediction, and predictor sharing, and distinguish position decodability from forecastability and control performance.

Critical strengths
  1. The control improvement is substantial under a well-matched deployment setup. All methods use the same visual backbone, action prior, planner, and search budget. JEPA-x improves every evaluation family, with particularly large gains on insertion, grasping, and stacking. The three training seeds and paired scenario-level tests provide useful support for the comparison with VISUAL. Achieving these gains while removing the physical branch at deployment makes the approach practically attractive.

  2. The representation analysis goes beyond evaluating the co-trained predictor. Freezing the encoder and fitting a fresh predictor directly addresses whether the learned representation supports prediction outside its original encoder–predictor pairing. Appendix C also reports the persistence denominator and examines the effective-rank explanation. Together with the decoding results, these checks provide credible evidence that the training objective changes the predictive structure of the visual representation. This analysis is a meaningful contribution beyond the aggregate success rates.

Critical weaknesses
  1. The comparisons leave uncertainty about the advantage over other ways of supervising dynamics with physical state. In Appendix G, FUT-ABS and FUT-DELTA receive a single visual latent and an action, whereas JEPA-x receives a three-frame history. This difference matters because the physical inputs deliberately exclude velocity. In addition, the regression loss averages over a target containing 512 geometry entries per object, without a reported comparison using balanced weights for geometry and dynamic variables. These choices leave plausible alternative explanations for the weak auxiliary-regression results. A matched comparison with the closely related PhyLatent, which also introduces training-time physical grounding and future alignment into a JEPA world model, would further clarify the contribution beyond the paper’s own baselines.

  2. The connection between forecastability and better action selection is supported indirectly. ALIGN-ONLY already reaches 72.6% success, recovering most of the improvement over VISUAL. CROSS-ONLY has essentially the same drift as JEPA-x, despite lower mean control success. These are informative results, but they leave open which changes in the representation actually improve planning. In particular, the statement in Section 5 that predictor sharing improves the consistency of goal-directed comparisons should remain a hypothesis unless those comparisons are measured. The fresh-predictor results establish improved forecastability under the chosen probe; direct evidence about candidate-action ranking would make the proposed explanation considerably stronger.

Questions for authors
  1. Baseline comparability: Have you evaluated FUT-ABS or FUT-DELTA with the same three-frame history as JEPA-x and separately weighted pose/motion and geometry targets? If so, does the forecastability and control gap remain? This would materially affect how I interpret the negative regression results.

  2. Objective sensitivity: How were the alignment and auxiliary-regression weights selected? Does JEPA-x retain its advantage over ALIGN-ONLY when alignment strength and the relative weighting of prediction and SIGReg are tuned comparably?

  3. Mechanistic evidence: Do you have measurements of candidate-action ranking accuracy or physical rollout error for JEPA-x, CROSS-ONLY, and ALIGN-ONLY under the same candidate sequences? Such evidence would help explain why comparable forecastability produces different control performance.

  4. Closest comparison: Is a matched comparison with PhyLatent available on the overlapping single-task benchmarks? If not, which experimentally demonstrated advantage should distinguish JEPA-x from this closely related approach?

Reviewer_GjK3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents JEPA-x, a training framework for improving the predictive structure of visual representations in latent world models. Building on LeWM, it introduces a physical-state encoder and a shared action-conditioned predictor. Visual and physical histories each predict future representations from both modalities, yielding four within-modal and cross-modal prediction terms. Both encoders remain jointly trainable, while separate isotropy regularizers discourage collapse. A unified object-based interface represents geometry, extent, pose, and effector configuration across manipulation settings. The physical branch is removed after training, preserving the visual baseline’s deployment architecture. The evaluation combines a multi-task manipulation suite with four additional single-task environments. On the multi-task suite, mean control success increases from 53.6% to 78.2%. To assess representations independently of their original predictors, the authors freeze the visual encoders and train new predictors, obtaining a reduction in normalized rollout drift from 0.361 to 0.104. State-regression and distillation comparisons suggest that position decodability alone does not explain these gains. Ablations indicate that frame-level alignment accounts for much of the control improvement, while cross-prediction further improves forecastability. Additional single-task results show consistent forecasting gains, while control improvements vary across environments. Overall, the work examines how privileged supervision shapes predictive representations and their usefulness for visual planning.

Critical strengths
  1. The reported improvement over the visual baseline is substantial. JEPA-x increases mean control success from 53.6% to 78.2%, with gains across all six interaction families under matched deployment conditions. Multiple training seeds and paired statistical tests strengthen this comparison. These results provide credible evidence that the proposed training scheme is useful within the evaluated setting.

  2. The fresh-predictor evaluation is a valuable diagnostic. By freezing the encoder and training a new predictor, the authors examine predictive structure beyond the original encoder–predictor pairing. Decodability probes and rank-controlled analyses provide complementary evidence. This analysis offers useful insight into why recovering physical variables from a representation does not necessarily imply reliable latent prediction or effective planning.

Critical weaknesses
  1. The distinct empirical contribution of symmetric cross-prediction is not sufficiently established. ALIGN-ONLY already achieves 72.6% control success compared with 78.2% for JEPA-x, while comparisons with stronger temporal supervision approaches related to TWIST and PhyLatent are absent. The results demonstrate gains over the selected baselines, but leave the advance over closely related methods uncertain. This limits the evidence supporting the significance of the proposed contribution.

  2. The comparison with direct physical supervision is not sufficiently controlled to support the mechanistic interpretation. The future-state heads in Appendix G receive a single visual latent, whereas JEPA-x receives histories. Averaging regression over many static geometry coordinates may also underweight motion. These differences leave uncertainty about how much of the gap comes from the learning objective itself, which is central to the paper’s contribution.

Questions for authors

The following questions are intended to resolve concerns that could change my recommendation.

  1. Can the authors provide a matched comparison with a trajectory-level distillation objective or the physical-grounding and future-alignment components of PhyLatent on an existing benchmark? Given the strong ALIGN-ONLY result, such a comparison would help establish the additional value of symmetric online cross-prediction beyond existing forms of privileged supervision.

  2. Do the conclusions of Appendix G persist when the future-state prediction heads receive the same length-three history as JEPA-x and dynamic target groups are weighted separately from static geometry? This control would help determine whether the observed gap reflects the proposed objective or differences in temporal information and supervision weighting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.