acceptodds
Withdrawn from ICLR 2027

LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Compact JEPA world models enable efficient latent-space planning, but low-dimensional representation trained under reward-free self-supervision must encode both action-conditioned dynamics and predictable visual context. This competition can entangle controllable state with high-rank nuisance appearance and degrade planning as scenes become more complex. We introduce LRC-JEPA, a lightweight end-to-end world model that routes information into a compact predictive latent and learned-query residual-context embeddings . Only is propagated by the dynamics model and used for planning, while captures temporally persistent information for cross-attention reconstruction; a differentiable residual connection encourages the latent to retain complementary dynamic content. Under explicit assumptions, we show that the resulting representation is sufficient, minimal, nuisance-invariant, and disentangled. Across four simulated control environments, LRC-JEPA improves average planning success over a parameter-matched JEPA baseline by 9 percentage points and matches or exceeds substantially larger pretrained models. On the real-world Bridge-v2 set, its 5.5M-parameter active encoder outperforms DINO-WM (22.1M) and V-JEPA2 (303.9M) encoders while also enabling faster planning. Physical-state probes, reconstruction interventions, and ablations confirm the effectiveness of LRC-JEPA's representation disentanglement.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world model, representation learning, vision encoder, vision transformer, JEPA, model-based control

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VuAbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses information allocation in compact JEPA world models by separating predictive latent from learned-query residual-context embeddings. Action-conditioned prediction and SIGReg regularization shape the predictive latent, while temporal context regularization encourages the residual-context embeddings to capture persistent information. A joint reconstruction decoder provides additional supervision to both streams. Evaluation includes closed-loop planning in four simulated environments. The reported results show improved performance over a compact LWM baseline while retaining a similar computational cost.

Critical strengths
  1. The method improves performance while preserving compact latent-space planning. With the same number of parameters, LRC-JEPA outperform the baseline in four experiment environments.

  2. The paper provides direct evidence about information allocation. The physical-state probes and cross-trajectory context replacement support different functional roles for zz and uu. The PCA-compression and static-pattern perturbation experiments additionally test properties relevant to compactness and nuisance robustness. These analyses provide meaningful support for functional specialization beyond task-performance improvements.

Critical weaknesses
  1. The methodological advance over prior separation mechanisms is insufficiently explained. ContextWM [1] already separates context from latent dynamics, combines them through cross-attention reconstruction, and uses the dynamic state for control. Iso-Dream [2] also evaluates a reconstruction-only auxiliary branch that is discarded during imagination and deployment in its distractor setting. LRC-JEPA differs through its learned-query context, temporal regularization, and reward-free JEPA planning formulation. However, the paper does not clearly explain the methodological advances beyond the established separation principle.

  2. The planning gains are not sufficiently isolated as a consequence of the proposed functional separation. The comparison with LWM simultaneously introduces reconstruction supervision, a context branch, and temporal context regularization. Matching the active encoder therefore does not isolate disentanglement as the source of improvement.

[1] Wu et al., Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning. NeurIPS, 2023. [2] Pan et al., Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Models. NeurIPS, 2022.

Questions for authors
  1. What specific limitation of prior separation mechanisms does LRC-JEPA address, and how do its architectural and training changes resolve such limitation in the reward-free JEPA?

  2. How does LRC-JEPA compare with a LWM augmented with comparable reconstruction supervision and training resources?

Reviewer_qFEeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces LRC-JEPA, a world model with compact latent representations. It uses a small latent zz for predicting dynamics and planning actions, and auxiliary embeddings uu to capture background context for image reconstruction. Only zz is used for dynamics prediction and planning, while uu supports reconstruction during training. Experiments show improvements over several well-known baselines.

Critical strengths
  • Code and pretrained checkpoints are provided, which is good for reproducibility.
Critical weaknesses
  • Proof Issues. A major claim of the paper is the theoretical justification for LRC-JEPA. However, the proofs contain several significant flaws.

    1. Mixing Discrete and Continuous Entropy. In Eq. (22), the claim "Since UU is a deterministic function of VV, H(U)=I(U;V)H(U) = I(U; V)" holds for discrete variables, but not generally for continuous variables, which are allowed in the paper's setting. By definition, I(U;V)=H(U)−H(U∣V)I(U; V) = H(U) - H(U \mid V). Thus, I(U;V)=H(U)I(U; V) = H(U) follows from H(U∣V)=0H(U \mid V) = 0 in the discrete case. However, this does not hold for continuous variables. Consider U=V∼N(0,1)U = V \sim \mathcal{N}(0,1). The differential entropy H(U)H(U) is finite, but H(U∣V)=−∞H(U \mid V) = -\infty and I(U;V)=∞≠H(U)I(U; V) = \infty \ne H(U). Similar issues appear in several other equations such as Eq.(23) and Eq.(25).

    2. Counterexample to Assumption 5's Justification. Assumption 5 claims that minimum information rate is enforced by the dimensional constraints dz=dsd_z = d_s and Qdu=dvQd_u = d_v. Consider a simple example with ds=dz=1d_s = d_z = 1 and Q=du=dv=1Q = d_u = d_v = 1. Let the observation be Xt=(St,V)X_t = (S_t, V), where StS_t represents physical state and VV represents static background context, with both being independent standard Gaussian variables. The intended separation is Zt=StZ_t = S_t and Ut=VU_t = V. However, we can instead construct Zt=(St+V)/2Z_t = (S_t + V) / \sqrt{2} and Ut=VU_t = V. This representation still satisfies the dimensional constraints, has a standard Gaussian ZtZ_t, maintains temporally constant UtU_t, and allows perfect reconstruction since St=2Zt−UtS_t = \sqrt{2} Z_t - U_t. Yet ZtZ_t still contains background information. Thus, the dimensional constraints and stated representation properties do not enforce nuisance separation or establish minimum information rate. Instead, minimum rate is explicitly assumed in Assumption 5, without showing that the proposed architecture or training objective achieves it.

    3. Circular Proof. Remark 2 defines the prediction task as Y=Zt+1Y = Z_{t+1}, while Assumption 4 requires I(Y;V)=0I(Y; V) = 0. If the same definition of YY is used in Theorem 2, substituting it gives I(Zt+1;V)=0I(Z_{t+1}; V) = 0, which already assumes the nuisance-invariance property that Theorem 2 claims to prove. Thus, the invariance claim appears circular. If YY instead refers to the underlying physical state, the authors should clarify this distinction and explain why independence of the physical state from VV implies independence of the learned representation from VV.

  • Lacking Relevant Important Baselines. Separating physical dynamics from visual context is a well-established idea [1, 2]. Therefore, the comparisons should not be limited to common world-model baselines, but should also include methods adopting similar decompositions. DDP-WM [2] explicitly separates physical dynamics from background updates for efficient world-model planning, without requiring reward supervision. It is therefore directly relevant to the paper's reward-free setting and should be included as an experimental baseline.

[1] Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning, NeurIPS 2023

[2] DDP-WM: Disentangled Dynamics Prediction for Efficient World Models, ICML 2026

Questions for authors

See the "Critical Weaknesses section".

Minor weaknesses:

  • Figure 2 can be improved by showing ete_t and gag_a between action a1:Na_{1:N} and PϕP_\phi
Reviewer_7cFPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to learn latent representations for world models that disentangle the control/dynamics-relevant part of an observation from the visual/context part. The authors propose LRC-JEPA which splits the representation into two streams: a compact latent z (CLS token of a ViT encoder), which is the only input to the action-conditioned predictor (and the only representation used for planning) and a context embedding u (obtained by cross-attention of learned queries over patch tokens), which is used by an image reconstruction decoder. Their training loss combines next-latent prediction, SIGReg, a temporal VICReg-style loss, and a pixel reconstruction loss. Under assumptions, they claim that the learned representation is sufficient, minimal, nuisance-invariant and disentangled. They evaluate planning on PushT, TwoRoom, Reacher and Cube against several baselines and also report probes of z and u, and ablations.

Critical strengths
  • Their method only modifies training, so planning is not modified and still uses only the latent z, so no cost is added at planning time.
  • Code and checkpoints are provided.
Critical weaknesses
  • The method uses image reconstruction as supervision, with gradients into the encoder and into z (Eq. 12, App. A.4). It is therefore not a JEPA model.
  • LRC-JEPA differs from LWM in two ways: it adds the context stream u, and it adds a pixel reconstruction loss whose gradients reach z. It is not clear which of the changes gives the performance increase.
  • The theory part of the paper is hard to read, and is developed in more detail than the method itself. It is also hard to see the connection between the theory and the method or the experiments.
  • The introduction is too short and does not motivate the paper enough.
  • There are too many architectural details in the method section. BN, LN and GELU should not appear in function definitions.
  • The wording is sometimes imprecise, e.g. “functional routing architecture that assigns predictive dynamics to compact z” (contributions). - What does “functional” mean, and what does “assign predictive dynamics” mean?
  • Minor: use \citep for parenthetical citations.
Questions for authors
  • How does LWM with a reconstruction decoder on z alone (without u) compare to LRC-JEPA?
  • Why is LWM so low on Bridge-v2 far below all other models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.