acceptodds
Under review as a conference paper at ICLR 2027

PhyLatent: Learning Dynamics-Relevant Representations for JEPA World Models

Abstract

We propose PhyLatent, a dynamics-relevant training objective for JointEmbedding Predictive Architecture (JEPA) world models. Our key observation is that preventing global latent collapse does not necessarily ensure that the learned representation preserves physically meaningful state and action relationships. We identify three failure modes: sensitivity to appearance changes that leave the physical state unchanged, insufficient separation of distinct physical states, and insufficient separation of different action-conditioned futures. We refer to these as Physical Invariance Collapse, Physical Distinguishability Collapse, and Counterfactual Dynamics Collapse, respectively. PhyLatent targets these failures through three coordinated training pathways, implemented with physical state grounding, future representation alignment, static visual invariance, counterfactual branch separation, and latent denoising. On OGBench-Cube, PhyLatent reduces the three collapse rates by 43.9%, 27.9%, and 47.1%, respectively, while improving MPC success by 12.0 percentage points (17.2% relative). Across four visual-control tasks, average planning success increases by 6.62 percentage points (8.3% relative). These results show that global non-collapse alone is insufficient for learning a reliable JEPA world-model state space, and that explicitly preserving dynamics relevant structure can improve closed-loop planning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
JEPA, World Models, Dynamics-Relevant Representation Learning, Representation Collapse, Latent Dynamics, Model Predictive Control, Visual Planning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_avvEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors present an argument that the global collapse regularization in JEPA-based world models, such as SIGReg in LeWM, does not guarantee that the latent space keeps the relevant dynamics-aware structure for planning. They define three key dynamics relevant failure modes: Physical Invariance Collapse, Physical Distinguishability Collapse, and Counterfactual Dynamics Collapse. Each of these modes of collapses focus on different issues and can be broken down into non-impactful visual changes resulting in shifts in the learned latent space and impactful visual change or different action applied to the same state do not result in large enough shifts in the latent space.

The authors propose PhyLatent which consists of adding 5 auxiliary losses to LeWM's training procedure. These 5 losses are defined as a static visual invariance penalty, physical state grounding, future representation alignment, counterfactual action separation penalty, and latent denoising. Across Cube, Two-Room, Reacher, and PushT benchmarks, PhyLatent not only improve the overall performance but also reduces the three collapse metrics.

Critical strengths

The motivation is well presented and useful for the overall field of world modeling and the authors have done a great job and defining what sort of collapses they notice in JEPA models and what they aim to fix. Additionally, the authors provide extensive analysis over their many experiments between PhyLatent and the baseline LeWM.

Critical weaknesses

W1. Gains over the existing methods, Sub-JEPA and SMWM, are marginal, and most of the improvement over LeWM appears to come from privileged state supervision. In table 1, PhyLatent's average is within 1 point of Sub-JEPA and SMWM which could potentially just be attributed to the standard deviation (SD is between 2-4 points across all tasks). Additionally, none of the baselines as far as I can tell received the same privileged state grounding which could attribute to higher performances. In table 3, the gains over LeWM seem mostly related to PSG other than Reacher where PhyLatent is still matched by the regular LeWM baseline. The goal-separation curves also show marginal improvements over LeWM. The displacement stratification is the most compelling result, but is not reported against LeWM + PSG, so it is unclear whether PhyLatent's gains would hold against a fairer baseline that is given the same privileged information during training. Overall, while the proposed method does seem to lower the proposed collapse metrics, the actual performance doesn't seem to be impacted as much which limits the scope of the paper as even though the learned representations might not necessarily preserve the best interpretable state and action relationships, maybe it is still useful enough for task at hand.

W2. The method integrates five additional auxiliary losses that each comes with their own weights and hyperparameters and the ablations are only done on Cube where some contradictions are shown. For example, removing SVIP somehow decreases invariance collapse and removing the counterfactual dynamics roughly leaves the CF collapse metric unchanged. Removing PSG seems to be the biggest impact on success and all 3 collapse metrics which makes it seem like that the privileged training time information is what really shapes the latent space to avoid these collapses.

Questions for authors

Q1. The reported numbers across Cube, TwoRooms, Reacher, and PushT for LeWM are all lower than those reported on the official LeWM paper. From what I can tell, the average performance across the task in their paper was 85.75 which is much higher than the reported 79.92 and much closer to PhyLatent's reported average of 86.54. Is there a difference from how the benchmarks were tested in this paper and LeWM's paper or can this be attributed to just high variance? Could you explain if there were any differences?

Q2. Can you report the displacement stratification for LeWM + PSG? Since this seems to be the strongest results in the paper, the gain from 26.67 to 60, it would be interesting to see whether the PSG objective is attributing to most of the gains.

Q3. How should one interpret table 4 on what each auxiliary objective does to LeWM? Are there any other more ablations or visualization done on how the different losses shape the latent space to avoid collapse?

Reviewer_stLEOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PhyLatent, a training objective for JEPA world models designed to preserve physical-state information and action-dependent dynamics. It identifies three failure modes that can persist despite global anti-collapse regularization and introduces five auxiliary losses to address them. Across four simulated environments, the method reduces the measured collapse rates and improves average planning success over LeWorldModel. The gains are concentrated on Cube and TwoRooms, with a modest improvement on PushT and effectively unchanged performance on Reacher.

Critical strengths
  • The paper addresses a meaningful limitation of existing approaches: preventing global representation collapse does not necessarily preserve the physical relationships needed for prediction and planning. The distinction between global non-collapse and locally useful latent structure provides a clear motivation.

  • The paper is well organized and generally easy to follow, with clear figures and concrete failure examples. The decision-level analysis helps distinguish inconsistencies in the encoded representations from errors introduced by the dynamics predictor.

Critical weaknesses
  • The largest gains over LeWM are mostly recovered by physical-state supervision alone. On Cube, LeWM + PSG reaches 80.67%, compared with 81.67% for the full method; on TwoRooms, the corresponding numbers are 96.33% and 96.83%. It is therefore difficult to attribute the main improvements to the proposed combination of failure-specific auxiliary losses rather than to the additional simulator-state supervision. The full method largely avoids the PSG-only regressions on Reacher and PushT, which is valuable. However, the experiments do not establish which additional components are necessary to obtain this benefit or whether a simpler combination would suffice

  • The relationship between the diagnostics, training pathways, and planning gains remains unclear. On Cube, removing the entire Counterfactual Dynamics pathway changes success only from 81.67% to 81.17%, while slightly improving the corresponding counterfactual diagnostic. Removing FRA also improves the distinguishability diagnostic while reducing success. Similarly, increasing all auxiliary-loss weights improves the diagnostic scores without consistently improving planning. These results suggest substantial interactions among the objectives and weaken a straightforward explanation in which each pathway repairs its associated failure and thereby improves control. The authors acknowledge this limitation, but it leaves the paper’s explanation of the behavioral gains less established than its empirical improvements.

  • The benefit over stronger alternatives is not clearly established relative to the added complexity. PhyLatent introduces five auxiliary objectives with manually adjusted, task-specific coefficients, as well as physical-state supervision during training. Yet its average planning success is 86.54%, compared with 86.33% for Sub-JEPA and 85.67% for SMWM, with other methods leading on individual tasks. These results support an improvement over LeWM, but provide limited evidence of a meaningful advantage over the strongest alternatives. The Cube experiment that scales all auxiliary weights together is useful, but does not establish robustness to their relative weighting or clarify the tuning effort required across tasks.

Questions for authors
  • The reported LeWorldModel results differ from those in the original paper. Could the authors clarify which differences in training, checkpoint selection, sampled start–goal pairs, or planner settings account for this discrepancy?

  • Under matched supervision and tuning budgets, how do simpler combinations, such as PSG + visual invariance or PSG + future alignment, compare with the full method across all four tasks?

  • What explains the large background and texture regressions on TwoRooms and Reacher? Could component ablations under these shifts identify whether the failures arise from the augmentation choices, the selected physical-state targets, or interactions among the objectives?

  • On Cube, success generally decreases with temporal goal offset before increasing at the largest offset. The paper correctly notes that temporal separation need not imply greater physical difficulty. Could the authors characterize how the sampled goals differ across offsets—for example, in physical displacement or task phase—to explain this rebound?

Reviewer_sKscOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper discusses three major types of representational collapse in JEPA style world models: physical invariance collapse, physical distinguishability collapse, and counterfactual dynamics collapse. Then, it offers a solution, PhyLatent, by enhancing the LeWM training objective through five auxiliary losses. Finally, experiments are performed on 4 benchmarks (Cube, TwoRooms, Reacher, and PushT) to compare against existing methods. PhyLatent improves significantly in collapse rates compared to LeWM but appears to only offer modest performance benefits closed-loop planning success.

Critical strengths

A major strength was that PhyLatent does appear to improve collapse rates in comparison to the baseline LeWM which was the main motivation for the paper.

Critical weaknesses

The first major weakness of the paper was the clarity. It was generally hard to follow and overlooked explaining some important information that could have enhanced readability. For instance, “order preserved” was the central point of figure 2 but the concept and definition of ordering did not appear to be explained at all in the paper, only referenced to as something that could be violated. The second weakness was that despite improvements in collapse rates it appears that actual planning success was marginal. In fact, in table 1 PhyLatent did not perform the best on any individual task.

Questions for authors

Why do you not observe more significant improvements in performance on downstream tasks? Why is there such a large gap between LeWM and PhyLatent on background and texture in TwoRooms and Reacher?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.