acceptodds
Under review as a conference paper at ICLR 2027

JEPA-Anything: Learning Predictive Models across Different Worlds

Abstract

World models learn internal states for predicting how an underlying system changes. Cells, molecules, physical fields, control environments, and clinical trajectories differ in observation geometry, yet require predictions of hidden, intervened, or future states from available context. Joint-embedding predictive architectures (JEPAs) provide a natural framework for this problem, but their single target-prediction pathway does not explicitly organize predictive target coordinates. We therefore introduce \method, a framework that separates domain-specific observations and context–target construction from a shared predictive-state interface. Within this interface, we further propose orthogonal predictive factorization (OPF), which assigns target coordinates to separately parameterized factor-specific output heads and synthesizes them into a complete state for readout, intervention, planning, and multi-step rollout. Across a broad set of visual, biological, clinical, control, physical-field, weather, and molecular settings, domains retain their own encoders and observation geometries while using the same predictive core contract. Matched experiments show improvements in terminal readout, intervention prediction, and out-of-distribution or recursive forecasting; geometric analyses support well-conditioned synthesis, and factor interventions indicate functional use of the learned coordinate blocks. These results support a shared factorized predictive principle for heterogeneous worlds while preserving the structures needed for prediction, interaction, and analysis.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
World Models, Joint-Embedding Predictive Architectures, Predictive Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9EmROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces JEPA-Anything, a framework that reuses a common latent-state prediction procedure across independently parameterized domain models. Each domain retains its own input representation, encoder, and model parameters. Its core mechanism, Orthogonal Predictive Factorization, learns projections that transform target embeddings into coordinate blocks, predicts these blocks through a shared trunk with factor-specific output heads, and reconstructs a complete state through pseudoinverse synthesis. The training objective combines predictive regression with orthogonality and variance regularization.

Critical strengths
  1. Broad empirical coverage with substantial improvements on some tasks.

  2. The paper is very readable overall.

Critical weaknesses
  1. Limited Novelty in the Core Method The core idea lacks significant novelty. The method mainly adds learnable projections, partitioned outputs, and regularization to an existing JEPA. Since the model uses a shared backbone and linear heads, these partitioned heads can simply be combined into a single dense layer. A full-rank projection does not change the predictor's capabilities, and exact orthogonality makes the prediction error equivalent to that in the original space. Therefore, this method is mostly just a reparameterization and regularization of an existing predictor, not a fundamentally new mechanism. While these changes might affect training, the paper lacks deep analysis or new insights to prove this is a strong methodological contribution.

  2. Limited Contribution of the Cross-Domain Framework The contribution of the cross-domain framework is limited. Each domain still uses its own inputs, encoders, targets, and parameters; only the algorithmic steps are reused. JEPA already handles context encoding, target encoding, and latent prediction. While reusing a method across domains is reasonable, the paper does not clearly show what new technical ability this interface adds. Applying the same module to multiple domains shows it is broadly applicable, but this is not a new methodological innovation. The paper should more clearly separate the contribution of the unified framework from the specific changes made by OPF.

  3. Insufficient Experimental Proof of the Mechanism The experiments do not clearly prove that the improvements come from the proposed partitioned allocation of predictive capacity. Although ablation studies are provided, the effects of projection learning, output partitioning, and regularization are not well separated. Since partitioned linear heads are mathematically similar to a single dense layer, the paper should explain more clearly how partitioning changes the training process. Adding a control experiment with a fixed random orthogonal projection from the start (while keeping the variance regularization) would be very helpful. Also, performance drops after factor masking only show that the coordinates are being used, not that predictive information is allocated better. In the CITRIS analysis, the confidence interval compared to random rotations includes zero. Thus, the evidence shows the overall training design works, but it does not strongly support the proposed explanation of why it works.

  4. Theoretical Analysis Does Not Fully Explain the Improvements The theoretical analysis does not fully explain the predictive improvements. The results on orthogonality mainly limit coordinate-error amplification. They do not analyze how the encoders, predictors, and projections optimize together, or how errors accumulate during training. The post-hoc QR experiment shows that conditioning can improve without changing the prediction function, and the sensitivity analysis shows the lowest error actually occurs without orthogonality regularization. This does not mean orthogonality is useless, but it shows that better geometric structure does not always lead to better prediction. The paper needs to clearly explain when these benefits happen and how better geometry leads to better task performance during training.

Questions for authors

Please refer to Weaknesses.

Reviewer_9tNVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes JEPA-Anything, a cross-domain framework that keeps domain-specific encoders while sharing a JEPA-style predictive core, and introduces Orthogonal Predictive Factorization (OPF), which splits the target embedding into learned orthogonal blocks predicted by separate linear heads.

Critical strengths
  • Scope of empirical validation is broad
Critical weaknesses
  • Novelty

    • Fig. 1 and the Introduction (lines 076–087) describe situations where a standard JEPA is said to fall short, but they do not explain why it fails (i.e. challenges) or why the proposed method is the solution to resolve the challenge. This is the key part of a novelty claim, and it is missing.
      • Why OPF loss? Prior work already regularizes the structure of the latent embedding (e.g. VICReg, SparseJEPA) and organizing predictive coordinates into non-redundant blocks is not new. Moreover, line 218 states that concatenating factor heads “preserves the dense predictor’s function class”, so OPF adds no expressivity. If orthogonality is the key contribution, the authors should show how it differs from these existing methods and why it is the right solution to the stated challenges.
    • For a novelty claim built on combining components, the ablation should show why they must be combined this way.
      • Need independent component ablation. Now it’s additive ablation.
      • Most differences in Table 6 fall within one seed standard deviation
  • Clarity

    • Sections 3.1- 3.3: overall equations are not coherent. e.g. explicit definition of q_k is missing and the arguments in q_k from eq.2 vs eq.9 is not consistent.
    • Terminology is not convention/shared in the community nor clearly stated in the manuscript. Especially “pathway” is a vague term and it is unclear whether “pathway” refers to a domain-specific predictor (i.e. one for each world) or to a distinct computational branch within a predictor. The paper should clarify what constitutes a ‘pathway’.
Questions for authors

What are the results of (a) a dense predictor with only the variance hinges, (b) a fixed random orthogonal P, and (c) standard JEPA with VICReg-style regularization on the target?

Reviewer_To5MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces orthogonal predictive factorization (OPF), a framework that is designed to separate complementary predictive information and improve capacity allocation of standard JEPA models. The authors show that this framework improves over standard JEPA models on a selection of tasks and datasets.

Critical strengths

S1: Variety of datasets and tasks tested. The authors test 3 different levels/groups of task types and vary the datasets in each type, drawing from a variety of domains.

S2: The OPF framework is conceptually interesting and intuitive.

Critical weaknesses

W1: The framing of the paper is misleading. The title "JEPA-anything" and the guiding question ("Can different worlds be modeled through a common predictive learning principle while retaining their own observation geometry?", from Sec 1) lead the reader to believe that there is something about the existing JEPA framework that prevents it from being used on any domain. However, JEPAs themselves already use a common predictive learning principle and can in fact already be applied to anything. The paper's core contribution (OPF) is much narrower. In particular, the experiments test the paper's JEPA+OPF framework against standard JEPAs, which explicitly does not test the guiding question. I would strongly recommend the authors reconsider their framing of the paper to make it more accurate: for example, calling the method OPF-JEPA, factorizable JEPA... and changing the guiding question in the introduction.

W2: The evidence for OPF gains over standard JEPA are statistically weak. The key argument of the paper is that OPF helps in every domain/task, which will only be convincing with a set of complete, strong experimental results. More details in the questions section.

Questions for authors

Q1: Most dynamics results give means only: CITRIS, 8/9 Fig. 4a tasks, and the molecular table. Could stdevs/CIs be reported here? Can you also clarify how the CIs that are reported currently are calculated?

Q2: The paper does not report the Hopper and Walker2d returns or the KS six-step result. Can these be included?

Q3: Some of the baselines are suspiciously weak. On SigLIP2, standard JEPA does not beat the frozen checkpoint. Some dynamics baselines show no error growth over the rollout, indicating that the predictor could just be constant. Both planners get near-zero HalfCheetah returns.

Q4: This is just a suggestion and should definitely be lower priority than the above questions but the authors could consider looking into the effect of scaling models. The paper uses "capacity allocation" language, which could be strengthened by showing that JEPA+OPF is comparable to performance of larger standard JEPA models.

Minor:

  • All tasks and datasets could use more description, e.g., what is the format of MuJoCo/what is the task? The description for CITRIS is "CITRIS Interventional Pong (Lippe et al., 2022) supplies a pre-intervention observation and labels the externally changed state factors", which basically means nothing to me.

  • Fig 1 is incredibly confusing: nothing is defined ("operation", "region", s1, s2, ...)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.