acceptodds
Withdrawn from ICLR 2027

Learning Before Predicting: Physics-Guided Joint-Embedding Learning for Scientific Systems

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Scientific systems are heterogeneous twice: they live in different spatial dimensions, and their states contain quantities with different geometric laws. Density is a scalar, velocity is a vector, and orientational order is a tensor, yet learning pipelines often flatten them into interchangeable channels and ask a downstream predictor to recover both dynamics and structure from scarce labels. We ask a different question: can the physical organization of a state be learned before forecasting? We introduce UNIJEPA, a physics-guided joint-embedding framework that keeps field type explicit at the interface while a shared predictive core learns coupled spatiotemporal dynamics. During pretraining, lightweight field decoders expose predicted streams to system-specific equations, differential relations, realizability constraints, and transformation laws; the decoders are discarded before downstream prediction. Across 1D–3D scalar PDEs, 2D coupled flow, and scalar–vector–tensor Active Matter, the same parameterization principle improves few-shot transfer. On Active Matter with 16 labeled trajectories, UNIJEPA reduces five-seed full-state VRMSE and velocity error by 13.8% and 20.4%, respectively, and remains 10.1% better than a parameter-matched Scratch FNO. Frozen probes reduce mean physical readout error from 0.955 to 0.389, while typed-interface and exact- controls attribute these gains to physically organized representations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
joint-embedding predictive architecture, physics-guided representation learning, self-supervised learning, scientific machine learning, neural operators, few-shot forecasting, heterogeneous physical fields

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_RSkLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes UniJEPA, a pretraining scheme for few-shot forecasting of physical systems. Each field is routed by type (scalar, vector, tensor) into a shared Transformer, a predictor forecasts the next frame’s latent against an EMA target, and lightweight decoders expose the prediction to reconstruction, physics-residual, rotation-consistency and realizability losses during pretraining. The pretrained encoder and predictor are then fine-tuned inside an FNO on 16 trajectories. On Active Matter from The Well, this lowers full-state VRMSE by about 10% over a parameter-matched FNO trained from scratch; with some gains on PDEBench scalar systems and CFD2D.

Critical strengths

The following include the strengths of the manuscript: (S1) The structure of the paper is one of its strengths.

(S2) Multi-seed experiments are another positive point, commonly neglected in this area and present here.

Critical weaknesses

The following include the weaknesses of the manuscript (conceptual, technical and presentation):

(w1) The overall motivation of the work is not clear from the introduction or abstract. From the method it seems the method is acting a surrogate given all the predictive losses and hence the introduction must clarify this for the reader. Under the acceleration assumption it is expected so see plots comparing inference speed when comparing to baselines and even potentially the simulation itself.

(w2) If all the benchmarks are deterministic it is difficult for me to understand how JEPA or any representation encoder would facilitate the downstream objective which is forecasting in this work. The primary area where representation encoders shine are where observations are noisy and not deterministic to remove the noises not needed to model the state of a stochastic system.

(w3) The usage of term JEPA also fully seems out of place. Joint-Embedding Predictive Architecture generally refers to a case where we have joint embedding of the target and context and all predictions take place in this space. Once you add the pixel space (or physical space in this work) loss the JEPA disappears. What the work uses is some form of latent regularizer more like BYOL but with next frame as the target instead of an augmented view of the same frame.

(w4) The JEPA term is never shown to matter and in Active Matter JEPA-only pretraining equals random initialisation (0.1430 for both, Table 3), and its frozen probes are worse than a random encoder (Table 5). Table 4 credits the rotation and physics losses on decoded fields (rotation alone: 0.1427 -> 0.1306). Also in CFD2D and scalars the UniJEPA and JEPA are not distinguishable. Overall the improvements seem marginal to none compared to JEPA.

(w5) There are numerous number of terminologies used without proper definition and are not standard to the field. This is not an exhaustive list but mostly the ones that stand out and significantly reduce the readability of the paper and weakens the claims in general:

  • Use of “semantics” for physical variables (Secs. 1, 3.1). The term is non-standard and conflates two independent notions. First a field’s tensor rank, which fixes its transformation law, and a physical relation between fields. Stress and strain are rank-2 tensors related through a stiffness tensor that, for an isotropic material, reduces to a scalar Young’s modulus; the three share one relation but not one type.

  • Use of “pretraining” implies a general pretraining stage, but it is benchmark-specific: one checkpoint per family (Sec. 3.1), so the representations are never used across benchmarks or problems.

  • “Labeled trajectory” / “scarce labels” / “limited labeled trajectories must support learning the target dynamics” (Sec. 1). In forecasting every future frame is its own label, so pretraining and fine-tuning use the same supervision; the phrasing hides this. “Target dynamics” (Sec. 1). Undefined; the system forecast downstream is the system pretrained on.

(w6) The losses the main result depends on are not defined. L_eq never appears as a loss: App. A defers to App. D, where Eq. 37 is a test metric, and the model, inputs and group it is applied to during pretraining are unstated. L_phys for Active Matter is implied to be a kinetic residual (Eq. 9), declared non-evaluable in App. B, and reduced to strain consistency plus realizability in App. F.2, with the strain-consistency loss never given. Eq. 15 has five weights, Table 19 reports two, and the downstream fine-tuning objective is never written at all.

(w7) Main pipeline figures are not clear. For instance in figure 1 what is rho, u, or Q how are they different and what is a unified typed encoder? What is the stop gradient acting on and which loss is not backpropagated through which block? Figure 2 contains similar issues and is additionally too busy to understand what are the contributions.

Questions for authors

Following questions are raised to clarify the objective of the work and help to resolve the weaknesses. It would also help to provide any clarifications on the weaknesses raised directly.

Q1. Can authors describe the need for a JEPA like framework in deterministic environments where we can simply do pixels space predictions with masked modelling or next token prediction to extract rich features? The datasets evaluated here never seem to suffer from noisy observations. See w2.

Q2. Can authors provide some description on the stability of the framework? This framework contains and encoder-predictor-decoder all trained end to end. Decoder tries to spread information for reconstruction purposes and the predictor tries to collapse these into lower spectrums.

Q3. Does removing the JEPA from the pipeline actually affect the results provided here? The difference to baselines are not significant as presented, consequently it is expected the sources of gains to be ablated in detail.

Reviewer_aLZKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper claims that learning latent representations describing physical structure and temporal dynamics is better for few-shot forecasting than directly predicting physical states. Their method, UNIJEPA, encodes scalar, vector, and tensor inputs separately (typed encoder) and predicts future states in the representation space, as in JEPA, with physical constraint losses. A key contribution is the typed encoder, which promotes the understanding of each physical quantity. The performance difference between models with and without the typed encoder demonstrates its effectiveness.

Critical strengths
  1. Typed encoder is a sound and simple method to embed on the physics-aware representation.
  2. They provide various empirical tasks to support the performance again of their method.
Critical weaknesses
  1. The paper compares the typed encoder with a baseline encoder using concatenated inputs. However, a more appropriate baseline would use separate encoders for different input types and aggregate their embeddings to obtain a shared representation. This would better isolate the contribution of the typed encoder because simply concatenating inputs is analogous to concatenating language and image inputs without separate tokenization in a multimodal LLM, which would naturally lead to performance degradation. Furthermore, the paper should also compare the proposed method with other physics-aware representation learning methods (e.g., Phys-JEPA and seq-JEPA). It only includes comparisons with Scratch FNO and vanilla JEPA, which do not represent the PDE solvers used in practice. Therefore, the current comparisons do not explain why UNIJEPA should be preferred over other physics-aware representation learning methods.

  2. Although the paper emphasizes performance gains in percentage terms, the actual improvements still appear marginal relative to the complexity of the method. In particular, the reported VRMSE values for Base and Base+UNIJEPA overlap within their measured variance in Table 2. To better justify the performance gains, I would recommend evaluating SOTA PDE foundation models (e.g., Poseidon and MPP) and checking where the proposed method stands relative to these baselines and SOTA models.

Questions for authors

None

Reviewer_W6m1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces UniJEPA, a variant that includes a) type-specific encoding/decoding and b) a physics-preserving loss term. The authors show that this improves one-step rollout fidelity across multiple physical systems over traditional methods (FNO) and vanilla JEPAs.

Critical strengths

S1: Timely contribution with good ideas. Optimizing learning algorithms for physical and world modeling can have lots of important downstream applications, and the extension of JEPAs to this domain is still relatively new and underexplored. The idea of separating out and treating different typed inputs differently is intuitively interesting.

Critical weaknesses

W1: No gain on longer-horizon rollouts. Tables 15 and 16 show that UniJEPA is statistically identical to standard JEPA on long-horizon rollouts. While improved performance on one-step prediction is important, arguably the biggest unsolved problem we face in physical modeling is the instability of long-horizon rollouts and this approach doesn't seem to make gains there at all.

W2: Existing experiments and baselines only weakly support the claims. Only JEPA and Scratch FNO are tested, but there are plenty of SoTA/seminal approaches that were omitted, such as Poseidon [1], MPP [2], Walrus [3], etc. While "Large Scratch" is parameter-matched to the JEPA baselines, the comparison is a bit misleading given that the JEPA models have an extensive pretraining stage. Additionally, the experimental details about the JEPA baseline are not included in the paper, so it is not clear whether it is building on some existing work (which would be ideal for moving the progress of this field forward) or whether the authors did any tuning to make this an appropriate comparison. JEPAs are notoriously sensitive to things like hyperparameters and architectural choices, so a bit of hyperparameter tuning can make a big difference. Finally, the only experiment that includes all the types of fields (scalar, vector, and tensor), which is the main selling point of this work, is Active Matter, and this experiment does not make the comparison to vanilla JEPA (Table 1), only the two Scratch FNO variants.

[1] McCabe et al., 2023, Multiple Physics Pretraining for Physical Surrogate Models [2] McCabe et all, 2025, Walrus: A Cross-Domain Foundation Model for Continuum Dynamics [3] Herde et al., 2024, Poseidon: Efficient Foundation Models for PDEs

Questions for authors

Q1: How is the JEPA baseline being trained? Was there any hyperparameter tuning/training recipe tweaks done on this? Can the authors validate that the JEPA representations haven't collapsed? The paper text is missing any clarity on these questions.

Q2: Related to Q1, Table 3 "Active state" and "Active tensor" results for randomly initialized weights and JEPA baseline are almost (or exactly) identical. Was this a typo/mistake? If not, the fact that pretrained JEPA and random init model achieve basically the same results downstream indicates some serious pathologies with the way the JEPA baseline is being trained.

Echoing my point in W2:

Q3: Could you report the full set of results (Table 1) on the JEPA baseline for the Active Matter task? I think this is very important since Active Matter is the only dataset that incorporates all the major types and type-based training is so central to this work.

Q4: I would really encourage the authors to test against other published important works in the physical modeling space, e.g., Walrus [2], Poseidon [3]. While many of these are large foundation models and can't be FLOP- or data-matched, I believe these comparisons are important to document and these caveats can be noted in the paper.

Minor:

  • I would suggest to report stdevs from the seed runs in Table 1 rather than in the appendix.
  • Presentation could generally be improved. For example, the metrics (e.g., Raw full-state rel-L2, Strain-consistency rel-L2) aren't defined anywhere. Section 4.3 is very bare-bones - it's not clear at all how the authors are using UniJEPA to do transfer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.