acceptodds
Withdrawn from ICLR 2027

AVL-JEPA: PREVENTING CAUSAL DYNAMICS INFORMATION COLLAPSE IN JOINT EMBEDDING PREDICTIVE ARCHITECTURE WORLD MODELS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

Joint embedding predictive architectures (JEPAs) predict future latent representations without reconstructing observations, enabling world models to focus on high-level semantic dynamics. However, a JEPA can preserve high-dimensional visual information while discarding information about the physical consequences of actions. We call this failure mode causal dynamics information collapse and propose action-grounded vision-invariance latent (AVL) to prevent this collapse. We first use the executed action as an auxiliary dynamics anchor that encourages the model to preserve dynamics information, and then use a vision-invariance pathway which aligns perturbed and clean latent predictions without discarding dynamics information, forcing the model to fully understand and utilize causal dynamics information. We validate AVL on four robotic control tasks (TwoRoom, PushT, OGBench Cube, and Reacher), showing that it substantially improves success rates under visual perturbations while preserving clean-environment performance. We further evaluate physical-consequence alignment, clean–noisy dynamics consistency, and the causal effect of targeted transition subspace erasure. Collectively, these results indicate that dynamic information causally relevant to planning is preserved from collapse under AVL.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Joint Embedding Predictive Architecture, Latent World Models, Self-Supervised Representation Learning, Latent Dynamics, Visual Invariance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AxMAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a failure mode termed causal dynamics information collapse, in which JEPA representations remain globally non-degenerate but insufficiently preserve action-conditioned physical change. It proposes AVL, a two-stage extension of LEWM. The first stage introduces an auxiliary action-reconstruction head operating on the current latent and the difference between encoded future and current observations, with a shared zero-transition branch subtracted from its response. The second stage freezes this head while retaining its loss and trains with visually perturbed current observations and clean future targets. The auxiliary head is removed for planning. Experiments cover four simulated control tasks and include visual-perturbation robustness, matched-state physical-consequence comparisons, clean–noisy transition consistency, and PCA-subspace erasure. The paper reports substantial robustness improvements, particularly on PushT and TwoRoom, although gains vary by task and perturbation.

Critical strengths

1. A practically relevant problem with substantial robustness gains.

The paper makes a useful distinction between avoiding global representation collapse and preserving information that is actually relevant for control. The improvements are substantial in several settings and are not limited to the unaugmented LEWM baseline. For example, on PushT with Gaussian noise at σ=0.10\sigma=0.10, AVL reaches 81.5% success, compared with 15.3% for LEWM and 31.3% for the vision-invariance variant. The gains also extend to brightness and saturation perturbations, even though training uses only mild Gaussian noise. Since the auxiliary head is removed at deployment, the method keeps the original planning interface.

2. The evaluation goes beyond latent-space visualization and relates learned transitions to physical outcomes.

Evaluating the same candidate actions from matched simulator states is a useful way to test whether the learned transition geometry reflects actual physical consequences. The clean–noisy comparison also helps separate action-dependent variation from perturbation-induced drift. Appendix B.5 includes same-dimensional random-subspace erasure controls, and the PCA visualization is treated as qualitative evidence rather than as the main support for the claim. These analyses make the evaluation more informative than success rates alone, although some of the conclusions still depend on how these diagnostics are interpreted.

Critical weaknesses

1. The specific roles of zero-transition subtraction and freezing the action head are not sufficiently isolated.

The current ablations show that combining action grounding with vision invariance is useful, but they do not isolate the two mechanisms that are central to the proposed explanation. In particular, it is unclear whether the transition-attributed head is better than an otherwise matched inverse-dynamics head without the zero-transition subtraction, and whether freezing the action head in the second stage is actually necessary compared with continuing to train it. The initialization and loss setup of the standalone vision-invariance variant should also be stated more clearly.

The constraint r(z,0)=0r(z,0)=0 is reasonable, but by itself it does not show that an action-decodable representation preserves all physical information needed for planning. Likewise, freezing one action readout constrains that readout, but not necessarily the full latent dynamics. Matched controls for the head design and for freezing versus continued training would make the mechanism claim much stronger. Training exposure should also be matched or clearly accounted for: the main text uses 10 + 2 epochs for AVL, whereas Appendix B.6 reports an 11-epoch LEWM run.

2. The diagnostics support improved transition geometry more directly than recovery of planning-relevant causal information.

The physical-consequence correlation measures agreement between Euclidean distance structures, and the regret oracle selects the candidate with the largest physical-response norm. Both are useful diagnostics, but neither directly shows that previously missing planning-relevant information has been recovered. In particular, a larger physical displacement is not necessarily equivalent to making progress toward the task goal. I think the paper should separate these more limited geometric findings from the stronger claim about preserving causal dynamics information for planning.

The erasure experiments also rescore cached candidate responses rather than rerunning closed-loop planning after the intervention. Although same-dimensional random-subspace controls are included, matching dimension does not match the amount of transition energy being removed. There is also a presentation inconsistency: Equation (11) defines a signed degradation, while Figure 5 and Section 4.5 discuss absolute differences. Figure 9 further shows that some LEWM erasures reduce regret. For these reasons, a larger erasure effect should not always be interpreted as evidence that more indispensable planning information has been removed.

Questions for authors

1. Matched augmentation and mechanism controls.

The robustness results seem to mix the proposed mechanism with additional corruption exposure during training. The vision-invariance stage uses clean and Gaussian-perturbed current observations with clean future targets, whereas the standard LEWM baseline does not appear to receive the same augmentation. Could the authors provide an LEWM control trained with the same perturbation distribution, clean-target objective, optimization steps, and other relevant hyperparameters, but without the auxiliary action loss? It would also be helpful to isolate the two main design choices: zero-transition subtraction and freezing the action head. In particular, how do (i) a matched inverse-dynamics head without zero-transition subtraction and (ii) the proposed head kept trainable during stage two compare with the final AVL model? Please also clarify how the existing “Vision-invariance” baseline is initialized and trained.

2. Physical consequences and planning relevance.

What exactly is the physical response vector Δx\Delta x for TwoRoom, PushT, Reacher, and Cube? The current analysis ranks candidates by the norm of their physical response, but the action causing the largest state change is not necessarily the one that makes the most progress toward the task goal. Do the same conclusions hold if candidates are evaluated using goal progress, cost reduction, or the actual goal-conditioned planning objective? A comparison with an action-only predictor would also help determine whether AVL captures state-dependent consequences rather than mainly separating action categories in latent space.

3. Interpretation of the targeted-erasure analysis.

Equation (11) defines signed changes, while Figure 5 and Section 4.5 appear to discuss absolute differences. Could the authors report the signed values and clarify whether all reported changes actually correspond to degradation? Some of the extended LEWM results appear to show reduced rather than increased regret after erasure. The random-subspace control is useful, but a same-dimensional PCA subspace can contain substantially more transition variance than a random one. Does the result hold when the removed variance or transition energy is matched? Also, does erasing these directions hurt the actual closed-loop CEM planner, rather than only changing the ranking of cached candidate responses?

Reviewer_gv62OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies causal dynamics information collapse in JEPA world models. The latent space remains high-dimensional and non-degenerate, but it selectively loses information about how different actions causally change the physical state. To address this, the authors propose AVL, which has two variants. The action-grounded variant uses a transition-attributed action head to infer the executed action sequence from latent transitions, forcing the latent space to retain dynamics information. The vision-invariance variant applies Gaussian noise perturbations to the current observation and aligns the predicted latents from the perturbed and clean branches to the same representation. The method is validated on counterfactual physical consequence alignment, normalized regret, and clean–perturbed consistency.

Critical strengths
  1. The paper is clearly written, and the method is described in detail.

  2. The experimental results are sufficient.

Critical weaknesses
  1. In Table 1, Figure 4, and Figure 5, few baselines are compared; for example, PLDM, SD-JEPA, and DINO-WM are missing. Without comparisons to strong baselines, it is difficult to demonstrate the robustness of the proposed method. In addition, except for Table 2, no variance is reported.

  2. Sections 4.4 and 4.5 provide little analysis of the experimental results. More thorough discussion and analysis should be provided.

  3. The visualization in Figure 7 is of limited value. Comparing one successful AVL rollout with one failed LEWM rollout is not very meaningful. Comparing rollouts starting from the same initial state would be more meaningful.

  4. Section 4.3 states that under the nuisance drift metric, the value decreases by 3.318 relative to action-grounded, but in Table 1 the difference is 1.9. This is inconsistent.

Questions for authors
  1. In Figure 3, what do the different colors represent?

  2. In Table 4, AVL is actually lower than action-grounded and vision-invariance on Reacher (82.6 / 84.0). Could the authors analyze why?

Reviewer_wdufOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work looks at "causal dynamics information collapse" in JEPAs, where high-dimensional representations fail to capture action-conditioned physical consequences. To address this, the author(s) propose Action-grounded Vision-invariance Latent (AVL), which uses an action-anchored pathway and vision-invariance alignment with a frozen action head to improve planning robustness across robotic tasks. evaluations across four robotic control tasks and multiple visual perturbations demonstrate its effectiveness.

Critical strengths
  1. empirical studies are strong. Rigorous testing shows substantial performance gains under severe visual corruptions across diverse control benchmarks, e.g. Gaussian noise, brightness shifts, and saturation changes.
  2. Dual pathway design. The architecture cleverly leverages a frozen action head during vision-invariance alignment to prevent shortcut solutions, which avoids the need for pixel-level reconstruction.
  3. failure mode analysis. It pointsouts an issue in standard JEPA world models where predictive objectives favor persistent visual factors over non-linear, action-driven physical dynamics.
Critical weaknesses
  1. heuristic hyperparameter tuning. Key values e.g. the action loss weight (λ2=5\lambda_2=5) rely heavily on empirical sweep tuning without formal theoretical guarantees for general environments.
  2. Simulation-only experiments. If I understand the experiments correctly, all of them are confined strictly to simulated environments, which leaves real-robot transfer feasibility and sim-to-real robustness unproven.
Questions for authors
  1. as the action head is frozen during the vision-invariance phase while its loss remains active, how do you prevent gradient conflict or representation stagnation in the shared encoder?
  2. In the targeted subspace erasure analysis, how do you ensure that the removed principal components isolate true causal dynamics rather than correlated static background features?
  3. how sensitive is the framework to the chosen action horizon (H=5H=5), and do longer-horizon prediction tasks exacerbate error accumulation in the action-grounded pathway?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.