acceptodds
Under review as a conference paper at ICLR 2027

Branch-JEPA: Learning Dynamics-Aware State Representations through Dual-Branch Patch Routing

Abstract

Predicting future embeddings offers a compact route to learning visual world models, but a well-dispersed latent space need not expose the state changes that distinguish actions. We present Branch-JEPA, a representation-level extension of an end-to-end joint-embedding world model. Two independent spatial routers aggregate shared Patch features into State and Act residuals, which augment rather than replace a global CLS pathway. A contrastive action-alignment objective supervises changes in the fused Act coordinates, while one shared Predictor models the complete representation. The design preserves latent prediction, global anticollapse regularization and goal-conditioned planning without a pretrained visual backbone or a denoising frontend. Across PushT, TwoRoom, Reacher and Cube, task-averaged planning success increases from 83.25% to 88.25% on clean observations and from 35.50% to 59.75% under Gaussian corruption. Direct actionregression controls show that action supervision and Patch access account for much of the gain. Paired intervals resolve smaller improvements from separate routing and Act-only alignment under Gaussian noise; their clean intervals include zero. Frozen readouts, interventions and success-aligned candidate ranking connect these behavioral results to relative specialization within the shared representation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
World models, joint-embedding predictive architectures, semantic collapse, dynamics-aware representation learning, patch routing, mutual information, visual robustness

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5F1nOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work aims to improve the JEPA representation space by decomposing it into state and action regions, each loosely trained to be informative of the current state and the action executed. They propose the Branch-JEPA, which partitions the feature space in State and Act and define a residual branch that is based on the transformer patch embeddings and a CLS branch. Branch-JEPA leads to improvement of robustness to image noise as improved success rate.

Critical strengths

Extensive empirical evaluation to establish performance, including simulator tasks and ablation studies.

Critical weaknesses
  • The writing does not hold up to academic standards. The language is very hard to read and descriptions are incomplete (e.g. 4.1 setting and overview). In general, the manuscript lacks clear flow of arguments and conclusions and often connections are missing. The connection of BranchJEPA design with distribution spread as mentioned is not clear (other than being an SSL method). The overall motivation is not clear.
  • Motivation section is not convincing. Fig.1a shows the widely established fact that SSL losses like SIGReg exist to prevent collapse - it is not clear how this is related to the noise experiment. The precise conclusion from the noise experiment that would support the motivation of BranchJEPA is also not clear. Moreover, it is not clear why specifically action prediction in JEPA space being robust to noise is important for a specific task or aspect. The authors claim that the experiments motivate the model design to "expose action-associated change" as an important requirement but it is not clear how that conclusion was reached with the experiments provided.
  • The action alignment loss entirely depends on the distance between two consecutive action embeddings z_{t+1}-z{t}. The contrastive loss simply learns to distinguish an action and does not guarantee a representation geometry that represents understanding how the action physically affects the environment. Moreover, not considering the state in which the action starts its execution is another important limitation, as how the action is executed is often dependent on the current state. q and k remain undefined in Sec 4.3 which breaks the connection between the displacement delta and the loss.
  • The State vectors suggest to be specialized to the state of the environment but there is no loss or mechanism that enforces that or disentagles those features from action. Therefore, the State may not be specialized and could just contain the information already available in the CLS/patches.
  • The technical contribution is rather low, as it boils down to an action alignment loss on part of the features. According to Tab.2 it can be replaced with a simple regression loss with minimal accuracy difference.
  • The ablation provided does not prove that the action alignment helps. The extra branches and layers simply provide more model capacity that can increase the performance and by choosing not to add the residuals in the ablation this capacity is removed from the model. A better ablation would be to retrain without the action alignment loss to show if it has an effect on performance.
Questions for authors
  • How is the gaussian noise parameter 0.05 chosen in the context of the scale of the latent/image space? What is the precise conclusion from the noise experiment that leads to in support of the contributions? Why should the readout be expected to be robust to noise? Is action prediction robustness to Gaussian noise a key problem to solving certain tasks?
  • What is the precise proof that Act/State separation itself is the source of improvement?
  • In what way is the representation space enhanced? Is the action encoded in a meaningful way in terms of geometry between states? How does that compare with the baseline?
Reviewer_SkMJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Branch-JEPA, an extension of LeWorldModel that aims to improve the accessibility of action-relevant information in latent representations. The method introduces dual spatial routers to construct State and Act representations, together with a contrastive objective that aligns Act latent differences with executed actions. Experiments across four control tasks demonstrate improved planning success, particularly under visual corruption.

Critical strengths
  • Well-motivated problem formulation. The paper identifies an important distinction between preventing representation collapse and learning action-relevant dynamics, supported by empirical diagnostics.
  • Simple and effective architecture. The proposed design preserves the global representation pathway and shared dynamics predictor while introducing lightweight spatial routing and action alignment, maintaining compatibility with the original planning framework.
Critical weaknesses
  • Limited methodological novelty. The main components build on existing spatial aggregation and action-supervised representation learning techniques. Moreover, direct action regression achieves comparable performance, leaving the specific benefit of contrastive alignment unclear.
  • Insufficient justification for dual routing. Single-router and shared-router variants already achieve competitive performance. The marginal benefits of independent routing are relatively small, particularly under clean observations.
  • Limited support from the motivating diagnostic. The degradation of a clean-trained action probe under corruption demonstrates reduced readout transferability rather than a definitive loss of action-relevant information.
  • Limited robustness generalization. Evaluation focuses on synthetic visual corruptions, without examining more challenging distribution shifts, such as viewpoint changes, occlusions, or unseen environments.
  • Unclear connection between representation quality and planning performance. Although the paper provides probing and candidate-ranking analyses, these results do not establish how the proposed representation organization leads to improved planning, particularly under visual corruption.
  • Incomplete computational analysis. The paper does not report parameter counts, training FLOPs, or runtime overhead, making it difficult to assess the computational trade-offs of the proposed architecture.
Questions for authors

See strengths and weaknesses

Reviewer_1nHmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Branch-JEPA, a world model built on top of LeWM. LeWM uses a ViT encoder and uses the CLS features as the latent representation, regularized with SIGReg. Besides the LeWM components, the proposed method learns an attentive probe on the patch features of the ViT encoder of LeWM to get state and action residuals, which is later added to the CLS features. It also adds a contrastive loss that aligns frame-to-frame changes in the Act partition with the executed actions. The paper shows marginal performance gain on the original LeWM environments, and larger gains under test-time corruptions.

Critical strengths
  • The problem of JEPA world models has no guarantee whether learned features contains all task-relevant information is relevant.
Critical weaknesses
  • The paper is poorly written. It's hard to understand the gist of the paper, and nearly every claim comes with a disclaimer like "does not establish", "not causal", etc. The method is simple but it's hard to extract from the text.
  • The paper presents only one baseline of LeWM, which is largely insufficient. How does Branch-JEPA compare to other state-of-the-art world models, and regularization objectives for world models?
  • Is Branch-JEPA only applicable on top of LeWM? The generality of the method is concerning given the current shape of the paper.
  • The performance gain of Branch-JEPA is quite weak, and the experiments are generally not informative. On the standard environments in LeWM, it achieves very marginal improvement (3-6%). The larger gains appear only under a few corruption types chosen by the authors, and even the paper admits “These results concern the specified corruptions; they do not establish robustness to arbitrary backgrounds or lighting”.
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.