acceptodds
Under review as a conference paper at ICLR 2027

PDE-JEPA: Predictive Representation Learning of Latent Dynamics modeling for Parametric PDEs

12 views

Abstract

Physical trajectories contain more than snapshots of a system: they also reveal how its states evolve under governing conditions. However, representation learning for parametric partial differential equations (PDEs) has largely relied on reconstruction-based objectives that emphasize recovering observed physical fields. In this paper, we investigate predictive representation pretraining as an alternative to reconstruction-based learning. We find that predictive representations preserve rich physical information, yet this advantage alone does not ensure accurate field evolution. Based on these observations, we introduce PDE-JEPA for parametric PDE dynamics. Specifically, we first train an encoder using a masked-latent prediction to capture the underlying regularities of PDE dynamics. To explicitly adapt the pretrained representation toward a more dynamics-aligned state space, we then introduce a geometry projector that aligns latent trajectory geometry with the evolution geometry of physical fields. Finally, building on this geometry-aligned latent space, we further develop a physics-structured latent predictor that decomposes the dynamics into parameter-independent evolution and parameter-dependent response components. Extensive experiments on nine widely used PDE benchmarks demonstrate that our framework outperforms existing state-of-the-art methods by an average of 33.4% in-distribution, while achieving an average improvement of 51.4% when extrapolating to unseen governing parameters. Our implementation will be released.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
parametric pde, jepa latent space, geometry aligned, physical structured latent predictor

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_iRK6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PDE-JEPA for parametric PDE forecasting, combining predictive representation pretraining, latent trajectory geometry alignment, and a parameter-structured latent ODE. Experiments on nine PDE benchmarks, including parameter-shift evaluations, report improved forecasting accuracy. Representation probes and ablations investigate the distinction between physical informativeness and temporal predictability.

Critical strengths

1.The distinction between informative representations and representations suitable for accurate rollout is meaningful for scientific machine learning.

2.The reported gains are substantial, supported by module ablations, representation probes, and parameter-response analyses.

Critical weaknesses

1.The empirical comparisons require correction and clarification. Five baseline rows imported from Zebra reverse the Heat and Burgers entries, affecting Tables 1 and 26. Differences in temporal sampling, spatial resolution, and test-time conditioning also leave the comparability of borrowed results unclear.

2.The contribution of the geometry loss is not sufficiently isolated. PAG introduces a projector, auxiliary dynamics supervision, and additional training together. Existing ablations do not compare against the same setup with only the geometry loss removed, leaving the source of the improvement uncertain.

Questions for authors
  1. Could you reconcile the reporting discrepancies in Tables 1 and 26, particularly the reversed Heat/Burgers baseline entries and Zebra’s Wave-2D reconstruction error of 0.0010 versus 0.010 in the original source? Do these corrections change the conclusions?

  2. Are the compared methods evaluated using equivalent spatial resolutions, temporal sampling, forecast durations, and test-time conditioning information? Please clarify how directly cited baseline results are comparable to your setting.

  3. On Vorticity and Wave-2D, how does PAG perform when the geometry loss is removed while keeping the projector, auxiliary dynamics loss, anchor loss, and training budget unchanged?

  4. What do the seven conditioning dimensions for Burgers/Heat encode, and how is time-dependent forcing handled? For HeterNS, why is forcing frequency modeled through the linear term mD(q)mD(q), given its nonlinear effect on the spatial forcing field?

4 comments

We appreciate the reviewer’s thoughtful assessment and constructive feedback. Below, we address the comments in detail and present additional experiments to further validate our approach and strengthen our findings.


CW1.1 Baseline transcription errors

We apologize for the mistakes and thank the reviewer for the detailed suggestion. The revision is made in Table 1 in the main paper and Table 28.

CW1.2: Heterogeneous experimental settings (Resolution, trajectory length, training strategies, model capacity, PDE priors, training budget.)

Clarification: The heterogeneous experimental settings primarily stem from differences in the original baseline configurations. For instance, ENMA uses a spatial resolution of 128 for Advection and Combined Equation, while Zebra conditions on a full trajectory under a given PDE parameter, compared to ENMA's three historical frames. Our initial goal was to preserve each baseline's original configuration for faithful reproduction, which inadvertently led to heterogeneous experimental settings.

To address your concerns regarding resolution and trajectory length, we have conducted additional experiments on major baselines with identical resolution and length settings with PDE-JEPA (as shown in Table 1).

Table 1. Matched resolution and trajectory length experiments. Retrained models are in Bold.

1DZebraGEPSUnisolverDPOTPoseidonEMNAPDE-JEPA
Advect (14 ×\times 256)0.00790.1110.0400.0470.0220.0390.0074
Burgers (25 ×\times 256)0.1150.5530.2390.6570.1220.3340.0428
Heat (25 ×\times 256)0.1540.7990.2410.5360.09640.6170.0274
Wave-B (25 ×\times 256)0.2160.7020.5370.5250.1110.8940.035
Combined (14 ×\times 256)0.00960.0230.00920.0450.1780.3230.0074
2DZebraGEPSUnisolverViT-in-contextLE-PDEMAE-PDEENMAPDE-JEPA
Vorticity (30 ×\times 128$\times$ 128) ID0.1340.09030.09600.350.8670.1330.3010.035
Vorticity (30 ×\times 128$\times$ 128) OOD0.4080.4421.0310.711.010.5100.5130.288

PDE-JEPA still outperforms all baselines. We have also revised the report in Main paper Table 1, 2, Appendix Table 33 and configuration in Appendix Table 10, 11, 13, 14, 15, 17, 18.

To address your concerns on model capacity and PDE prior, we add Table 2 into Appendix C.2, Table 20 from line 2169 to 2191. Regarding PDE priors, we listed below:

  1. Trajectory history: multiple frames as inputs.
  2. Current (latent) state: current single (latent) frame.
  3. Support trajectory: full trajectory as context.
  4. PDE parameters: physics parameter (ξ\xi).

Table 2: Model Details.

FNOGEPSZebraUnisolverLNSMAE-PDE
Capacity50M1M42M55M24M20M
PDE PriorTrajectory historyCurrent state with a support trajectoryTarget initial state with a support trajectoryCurrent state, PDE parametersCurrent latent state and PDE parametersTrajectory history
MPPDPOTPoseidonLE-PDEENMAPDE-JEPA
Capacity115M30M10M20M60M24M
PDE PriorTrajectory historyTrajectory HistoryTrajectory-historyTrajectory history with<br> a support trajectoryTrajectory HistoryCurrent latent state and PDE parameters

To further address your concerns on training strategies and budget, we have conducted controlled ViT comparisons (Table 3), matching architecture, capacity, data settings, conditioning information, downstream training protocols, and training budgets.

Table 3 Strictly matched experiment on resolution, trajectory length, model capacity, PDE prior, training strategy and training budget.

ID RolloutOOD Rollout
ViT encoder+predictorJEPA0.0860.49
JEPA+PAG0.0400.39
JEPA+PAG + PSP0.0340.28
ViT encoder+predictorReconstruct0.0590.47
Reconstruct+PAG0.0630.62
Reconstruct+PAG + PSP0.0570.43
ViT encoder+predictorMAE0.120.51
MAE+PAG0.170.59
MAE+PAG + PSP0.150.54
LNSOriginal0.0590.48
LNSOriginal+PAG0.0620.62
LNSOriginal+PAG + PSP0.0570.45

With PAG and PSP, JEPA achieves best ID/OOD errors, supporting its advantage within the tested framework. We have added this in the main paper in section 5.4.


CW2: Ablations for geo loss

To address the concerns about geo loss, we conduct experiments on Vorticity and Wave-2D on PAG without geo loss (as shown in Table 4).

Vorticity IDVorticity OODWave-2D IDWave-2D OOD
PAG w/o geo loss0.0830.4910.350.50
Table 4: PAG without geo loss experiment.

Removing only this loss, with all other components and training budgets unchanged, increases ID/OOD errors from 0.040/0.397 to 0.083/0.491 on Vorticity and from 0.143/0.321 to 0.350/0.500 on Wave-2D. The geometry loss provides vital efforts on both datasets.


Q1: Writing mistakes

We apologize for the errors, and the conclusion will not be changed. . Since Zebra does not rank among the top two methods on either Burgers or Heat, correcting these values does not affect our conclusions regarding PDE-JEPA. For Wave-2D, correcting the reconstruction error from 0.001 to 0.01 reduces Zebra-R from 207 to 20.7. However, this remains higher than JEPA-R (8.72), and therefore does not change our conclusions. We have revised our manuscript.


Q2: Settings problem

Please refer to CW1.2.


Q3: Ablation on PAG losses

In Appendix D.3.2 Table 36, we find that removing anchor loss degrades representation's information. We further conduct experiments on loss ablations for PAG module (Table 2).

Table 2: PAG loss ablation.

Vorticity IDVorticity OODWave-2D IDWave-2D OOD
Full PAG0.0400.3970.1430.321
Full PAG w/o dyn loss0.0450.4530.1690.349
Full PAG w/o anchor loss0.0470.4230.1660.354
Full PAG w/o geo loss0.0830.4910.3550.498

Removing one loss at a time, with all other settings unchanged, increases ID/OOD errors, we find that geo loss matters the most. We add this results to Appendix D.3.2 paragraph "PAG loss ablations".


Q4.1: conditioning input for Burgers and Heat

We thank the reviewer for bringing this to our attention. This feedback has helped us improve the clarity of the manuscript.

The seven conditioning dimensions encode the viscosity/diffusivity and the forcing averaged over each prediction interval. For both Burgers and Heat, the condition vector is

\left[ \log\beta,\, S_{1,n},\,C_{1,n},\, S_{2,n},\,C_{2,n},\, S_{3,n},\,C_{3,n} \right].$$ For the transition interval $(t_n,t_{n+1})$, define $\Delta t_n=t_{n+1}-t_n$, $t_{\mathrm{mid},n}=(t_n+t_{n+1})/2$, and $\operatorname{sinc}(z)=\sin(z)/z$, with $\operatorname{sinc}(0)=1$. The coefficients are $$\begin{aligned} S_{\ell,n} &= \sum_{\substack{j=1,\ldots,5\\\\ l_j=\ell}} A_j \operatorname{sinc}\left(\frac{\omega_j\Delta t_n}{2}\right) \cos\left(\omega_j t_{\mathrm{mid},n}+\phi_j\right), \\\\ C_{\ell,n} &= \sum_{\substack{j=1,\ldots,5\\\\ l_j=\ell}} A_j \operatorname{sinc}\left(\frac{\omega_j\Delta t_n}{2}\right) \sin\left(\omega_j t_{\mathrm{mid},n}+\phi_j\right), \qquad \ell\in\\{1,2,3\\}. \end{aligned}$$ Thus, the five forcing modes $$A_j,\omega_j,l_j,\phi_j$$ listed in the appendix are aggregated by spatial harmonic into six interval-specific Fourier coefficients. $l_j$ selects the spatial-frequency channel, while temporal information enters through the midpoint phase and interval-averaging factor, without an additional time-conditioning dimension. Here, $$\beta$$ denotes viscosity for Burgers and thermal diffusivity for Heat. **We have added this to Appendix C.1.1 "Burgers and Heat: interval-dependent conditioning."** **Q4.1: PSP for HeterNS** We greatly thank the reviewer for giving us very useful suggestion. **The reviewer is correct: forcing frequency is not a linear amplitude coefficient in the governing equation.** We therefore **replace $mD(q)$ with a nonlinear conditional branch $D(q,m)$ (shown in Table 3).** > Table 3: HeterNS frequency module experiment. | | mD(q) | w/o mD(q) | D(q,m) | | :-----------------: | :---------: | :---------: | :---------: | | **ID** | 0.0089 | 0.0091 | 0.0083 | | **Visc./Force OOD** | 0.011/0.103 | 0.012/0.104 | 0.011/0.043 | Forcing-frequency OOD error decreases from 0.103 to 0.043, approximately 59%, while ID and viscosity OOD errors improve from 0.0089/0.011 to 0.0083/0.011 . Removing the forcing branch instead yields 0.0091/0.012/0.104, indicating limited benefit from the original affine branch. **We have revised Table 1, 2 in main paper and paragraph "HeterNS" in Appendix C.1.1.** ______

Dear Reviewer iRK6,

As the discussion period is approaching its deadline, we would like to kindly follow up and ask whether you have had an opportunity to review our response, and whether there are any remaining questions or concerns that we could further address.

Your feedback, including both constructive suggestions and points of concern, has been invaluable in helping us improve our work. We have made every effort to carefully address each of your comments and clarify any potential misunderstandings. We sincerely value the opportunity for further discussion and would greatly appreciate any additional feedback you may have.

Thank you again for your time and thoughtful consideration.

Best regards,

Authors of Submission 8314

Reviewer_gHGDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a JEPA style learning framework for modeling spatiotemporal PDE systems. The authors argue that a reconstruction based model can be instead replaced with a predictive latent based pretraining. To address rollout stability, they propose a geometry projector to align latent geometry with physical evolution. They introduce a physics structured latent predictor that provides inductive bias and decouples parameter independent evolution from the dependent responses.

Critical strengths

The motivations are strong. It's an interesting direction and addresses a lot of open problems in PDE literature. The breadth of empirical studies is quite satisfactory, with a large number of recent baselines and benchmarks considered for the experiments. The model exhibits good OOD generalization, statistically significant improvements over powerful and recent models across most benchmarks.

Critical weaknesses

The motivations for PAG and PSP are not clearly established. Particularly, with the section on "preserving informativeness", it's not clear to me how or why it would preserve informativeness and how it's different from JEPA objective. Similarly, for PSP, it's not easy to follow how C, r and D are determined. The framework seems to rely on a large number of losses. While the authors provide ablations, intuitively it's hard to follow what each one adds to the mix. What would be helpful, is to show the error field with each term removed, to clearly identify specific PDE structures that each loss term helps preserve.

Overall, the framework seems rather complicated and requires several parts and the paper is predominantly empirical and lacks intuitions on why a collection of losses is organically required

The text is also rather dense and difficult to follow, and required several rounds of reading, with a lot of difficulty stemming from introducing too many variables, not citing sources that motivate some of the losses.

Questions for authors

Please refer to the weaknesses above. The key question I have is whether the authors can provide more intuitions for why the framework requires so many loss functions, and whether there is some organic way that these integrate with each other.

5 comments

We thank the reviewer for the careful evaluation of our work and the valuable suggestions. In the following, we provide detailed responses to each comment, along with supplementary experiments to further substantiate our findings.


CW1.1 The motivation for PAG and PSP

Clarification: PAG improves temporal representation organization, while PSP targets generalization across governing parameters.

Their motivations appear in section 4.2, lines 261–267, and section 4.3, lines 299–304: informative representations may remain difficult to evolve recursively, while unrestricted state–parameter interactions may extrapolate poorly. PAG therefore adjusts trajectory geometry, whereas PSP structures how parameters enter the dynamics.

CW1.2 Why PAG preserves information

We clarify that the anchor loss has a simple motivation: to discourage excessive disruption of the pretrained latent structure during geometric modulation. Its benefit for preserving physical information is supported empirically by Table 36 and Figure 33 in Appendix D.3.2. During training, we observe that the anchor loss initially increases, indicating a temporary deviation from the pretrained latent space, before gradually decreasing and stabilizing, as shown in Fig. 33. Information preservation is empirical, not a lossless guarantee: removing the anchor improves Vorticity angle error from 6.3° to 4.8°, but reduces state-probing R2R^2 from 0.94 to 0.81 and increases param MSE from 0.009 to 0.024 (Appendix D.3.2 Table 36).

CW1.3 How are CC, rr, and DD determined in PSP?

CC and DD are MLP head followed by a shared backbone taking history frame as input (stated in Appendix C.1.2), while rr comprises normalized physical parameter. The intuition is that CC models shared evolution and DD models state-dependent parameter responses. Both are trained through latent-transition prediction. For Vorticity, rr normalizes viscosity using the training range (Appendix C.1.1). PSP is trained only with teacher forcing loss aligned with V-JEPA-AC[1]. CC and DD's output are summed to form Δq\Delta q for further loss computation. We have revised our main text in line 325-328.

CW1.4 Loss functions for PDE-JEPA

The losses serve distinct purposes at separate training stages.

  1. JEPA pretraining: We use the standard masked-latent prediction objective to train the encoder (Eq. 3).
  2. PAG:
    1. Geometry loss guides modulation toward physical trajectory alignment.
    2. Anchor loss limits disruption of pretrained information.
    3. Dynamics loss encourages the modulated state remain predictable.

We further conduct ablation study on each loss (as shown in Table 1).

Table 1: PAG loss ablation

IDOOD
Full PAG0.0400.397
Full PAG w/o dyn loss0.0450.453
Full PAG w/o anchor loss0.0470.423
Full PAG w/o geo loss0.0830.491

Removing one loss at a time, with all other settings unchanged, increases ID/OOD errors, we find that geo loss matters the most. Moreover, removing the anchor loss substantially degrades the readout of both PDE parameters and physical states, as shown in Appendix Table 36.

  1. PSP: PSP is trained only with teacher forcing loss (Eq. 17 in main paper) aligned with V-JEPA-AC[1]. The design contribution is the predictor’s physics-structured architecture.
  2. Decoding: the physical-field reconstruction head is trained separately, with the encoder, projector, and predictor frozen.

CW2: Complexity of losses

Please refer to answer to CW1.4.

Moreover, the training curves in Appendix Figure 33 further illustrate the balance between geometric adaptation and representation preservation.


CW3: Presentation issues

We thank the reviewer for this valuable suggestion. To improve the readability and clarity of the manuscript, we have made several major revisions, as summarized below:

  1. We add the overview of our PDE-JEPA in line 244-250 to make our framework easier to understand.
  2. We add more precise introduction in section 4.1 to distinguish pretraining and downstream forecasting training.
  3. We revise the opening of section 4.2 with more direct motivation for our PAG module.
  4. We have added a hyperlink in the “Preserving Informativeness” paragraph of Section 4.2 to direct readers to the corresponding results in the Appendix for further details.
  5. We have clarified the definitions and roles of CC and DD in the “Structured Latent Vector Field” paragraph of Section 4.3 to improve the understanding of the PSP module.
  6. We have added a hyperlink in section 5.1 paragraph "Baselines" to locate the details including input resolution and length, training strategies and budgets, model capacity and conditioning input information for easier finding the implementation details.
  7. We have moved the detailed experiments from Appendix D.3.2 to Section 5.4 of the main paper. In particular, we have brought the most important controlled experiment, comparing JEPA and reconstruction-based encoders under matched settings, into the main text.
  8. We have revised the pipeline figure by replacing the manifold illustration in the PSP module with a shared-backbone module to make the architecture clearer and easier to understand.

Q1: How losses are organized?

The main additional loss terms are introduced in the PAG module. PAG’s three losses jointly constrain the same residual modulation:

  1. Geometry loss guides trajectory alignment.
  2. Anchor loss limits representation drift.
  3. Dynamics prediction encourages predictable increments. With the encoder frozen, PAG jointly trains the projector and auxiliary predictor. The auxiliary predictor is then discarded, and PSP learns the final dynamics with the projector frozen.

Further details can refer to CW1.2-1.4.


References

[1] V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning.

Dear Reviewer gHGD,

As the discussion period is approaching its deadline, we would like to kindly follow up and ask whether you have had an opportunity to review our response, and whether there are any remaining questions or concerns that we could further address.

Your feedback, including both constructive suggestions and points of concern, has been invaluable in helping us improve our work. We have made every effort to carefully address each of your comments and clarify any potential misunderstandings. We sincerely value the opportunity for further discussion and would greatly appreciate any additional feedback you may have.

Thank you again for your time and thoughtful consideration.

Best regards,

Authors of Submission 8314

Reviewer_zE9oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to learn latent state spaces that support accurate long-horizon prediction and parameter generalization for parametric PDEs. Its central observation is that representations containing readily accessible physical information are not necessarily well suited to autoregressive evolution. The paper evaluates the framework on nine PDE datasets, reporting the best results on eight ID benchmarks and all five evaluated OOD datasets, with average relative improvements of 33.4% and 51.4%, respectively.

Critical strengths
  1. The paper identifies a meaningful problem and connects its observations to a coherent modeling approach.

  2. The empirical investigation covers diverse systems and extends beyond aggregate prediction errors. The evaluation spans transport, diffusion, wave propagation, reaction–diffusion, and fluid dynamics.

Critical weaknesses
  1. Important aspects of the evaluation protocol and conditioning inputs remain insufficiently specified, limiting the reproducibility and interpretation of the main comparisons. Appendix C.2 states that the results for CAPE, CoDA, Zebra, [CLS] ViT, and ViT-in-context are taken directly from Zebra. The assurance of identical evaluation trajectories and rollout protocols applies only to the reimplemented baselines. This distinction matters: the original Zebra paper describes Vorticity trajectories with 10 states on a 64×64 grid, whereas Table 8 of this submission specifies 30 frames on a 128×128 grid. The submission does not explain how the imported scores are made comparable under the final evaluation protocol.

  2. The independent contribution of the physical geometry loss has not been sufficiently isolated.

  3. The PAG stage jointly introduces a trainable projector, an auxiliary dynamics predictor, and the losses. Consequently, the improvement from adding PAG in Table 2 could partly arise from additional dynamics supervision and representation adaptation.

  4. The HeterNS parameterization does not match the physical dependence on forcing frequency. Appendix A.7, Eq. (33), defines the forcing as

fm(x)=0.1[sin⁡(mπ(x1+x2))+cos⁡(mπ(x1+x2))],f_m(x)=0.1\left[\sin\left(m\pi(x_1+x_2)\right)+\cos\left(m\pi(x_1+x_2)\right)\right],

where mm changes the spatial frequency. However, Appendix C.1.1 parameterizes its latent contribution as mDθm(q)mD_\theta^m(q), which imposes affine dependence on mm at a fixed state.

For example, the stated predictor necessarily satisfies

Fθ(q;ν,3)−2Fθ(q;ν,2)+Fθ(q;ν,1)=0,F_\theta(q;\nu,3)-2F_\theta(q;\nu,2)+F_\theta(q;\nu,1)=0,

whereas the corresponding physical forcing generally does not. The decomposition is therefore not directly inherited from the stated HeterNS equation in the same way as the viscosity decomposition is inherited from Vorticity.

Questions for authors
  1. Please specify how the five imported baselines and PDE-JEPA compare in spatial resolution, prediction time points, evaluated channels, and conditioning information, particularly for the Vorticity grid and frame-count differences. Were the errors recomputed under a common evaluation protocol? If not, what are the results for the strongest relevant baselines under a matched protocol? Please also verify and correct the Burgers/Heat column assignments in Table 1.
  2. What exactly are the seven conditioning variables used for Burgers and Heat? How do they represent the forcing coefficients listed in the appendix and the time-dependent forcing?
  3. What are the ID and OOD rollout results on Vorticity and Wave-2D when LgeoL_{\mathrm{geo}} is removed while keeping the projector, auxiliary predictor, LdynL_{\mathrm{dyn}}, LanchorL_{\mathrm{anchor}}, and training budget unchanged?
  4. How does the HeterNS predictor represent changes in forcing frequency?
4 comments

We are grateful to the reviewer for the constructive feedback and thoughtful assessment of our work. We address each of the questions raised below and provide additional experiments to further clarify and support our contributions.


CW1.1 Evaluation protocol and conditioning inputs

Clarification:

  1. Evaluation: we utilized relative L2 stated in section 5.1 'Metric' for evaluation same as Zebra.
  2. Conditioning inputs:
    1. Trajectory history: multiple frames as inputs.
    2. Current (latent) state: current single (latent) frame.
    3. Support trajectory: full trajectory as context.
    4. PDE parameters: physics parameter (ξ\xi).

Table 1: Conditioning inputs for different method.

FNOGEPSZebraUnisolverLNSMAE-PDE
Conditioning inputTrajectory historyCurrent state with a support trajectoryTarget initial state with a support trajectoryCurrent state, PDE parametersCurrent latent state and PDE parametersTrajectory history
MPPDPOTPoseidonLE-PDEENMAPDE-JEPA
Conditioning inputTrajectory historyTrajectory HistoryTrajectory-historyTrajectory history with a support trajectoryTrajectory HistoryCurrent latent state and PDE parameters

To address your concerns regarding resolution and trajectory length, we have conducted additional experiments on major baselines with identical resolution and length settings with PDE-JEPA (as shown in Table 1).

Table 2. Matched resolution and trajectory length experiments.

1DZebraGEPSUnisolverDPOTPoseidonEMNAPDE-JEPA
Advect (14 ×\times 256)0.00790.1110.0400.0470.0220.0390.0074
Burgers (25 ×\times 256)0.1150.5530.2390.6570.1220.3340.0428
Heat (25 ×\times 256)0.1540.7990.2410.5360.09640.6170.0274
Wave-B (25 ×\times 256)0.2160.7020.5370.5250.1110.8940.035
Combined (14 ×\times 256)0.00960.0230.00920.0450.1780.3230.0074
2DZebraGEPSUnisolverViT-in-contextLE-PDEMAE-PDEENMAPDE-JEPA
Vorticity (30 ×\times 128$\times$ 128) ID0.1340.09030.09600.350.8670.1330.3010.035
Vorticity (30 ×\times 128$\times$ 128) OOD0.4080.4421.0310.711.010.5100.5130.288

PDE-JEPA still outperforms all baselines. We have also revised the report in Main paper Table 1, 2, Appendix Table 33 and configuration in Appendix Table 10, 11, 13, 14, 15, 17, 18.

CW1.2 CAPE, CoDA, Zebra, [CLS] ViT, and ViT-in-context experiments

We thank the reviewer for pointing it out. We retrained PDE-JEPA at 1010 \times64 64\times64 64 and the five baselines at default 3030 \times128 128\times128 128 (shown in Table 3). All methods in this default-setting comparison were scored on the same target trajectories, future time points, and physical channels.

Table 3: Vorticity on 10 ×\times 64$\times$ 64 and 30 ×\times 128$\times$ 128 setting.

ZebraCAPECoDA[CLS]ViTViT-in-contextPDE-JEPA
Vorticity (10 ×\times 64$\times$ 64) ID0.119-0.6780.9720.1730.028
Vorticity (10 ×\times 64$\times$ 64) OOD0.32---0.370.246
Default: Vorticity (30 ×\times 128$\times$ 128) ID0.134-0.7901.010.350.035
Default: Vorticity (30 ×\times 128$\times$ 128) OOD0.408-1.261.520.710.288

Under the conditions in Table 1, the ID/OOD rollout errors are 0.035/0.288 for PDE-JEPA and 0.134/0.408 for Zebra. We revise Table 1 in main paper.


CW2+CW3: PAG loss ablations

In Appendix D.3.2 Table 36, we find that removing anchor loss degrades representation's information. We further conduct experiments on loss ablations for PAG module (Table 4).

Table 4: PAG loss ablation.

IDOOD
Full PAG0.0400.397
Full PAG w/o dyn loss0.0450.453
Full PAG w/o anchor loss0.0470.423
Full PAG w/o geo loss0.0830.491

Removing one loss at a time, with all other settings unchanged, increases ID/OOD errors, we find that geo loss matters the most. We add this results to Appendix D.3.2 paragraph "PAG loss ablations".


CW4: PSP for HeterNS

We greatly thank the reviewer for giving us very useful suggestion. The reviewer is correct: forcing frequency is not a linear amplitude coefficient in the governing equation. We therefore replace mD(q)mD(q) with a nonlinear conditional branch D(q,m)D(q,m) (shown in Table 5).

Table 5: HeterNS frequency module experiment.

mD(q)w/o mD(q)D(q,m)
ID0.00890.00910.0083
Visc./Force OOD0.011/0.1030.012/0.1040.011/0.043

Forcing-frequency OOD error decreases from 0.103 to 0.043, approximately 59%, while ID and viscosity OOD errors improve from 0.0089/0.011 to 0.0083/0.011 . Removing the forcing branch instead yields 0.0091/0.012/0.104, indicating limited benefit from the original affine branch. We have revised Table 1, 2 in main paper and paragraph "HeterNS" in Appendix C.1.1.


Q1.1 Dataset settings

We use the same data source from Zebra. For Vorticity, the difference arises from subsampling (Table 6). PDE-JEPA retains 3030 \times128128\times128 128, whereas Zebra downsamples to 1010 \times6464\times64 64, using 2×2 spatial averaging and temporal stride 3. The retained indices are 0,3,0,3,\ldots,27,27; forecasting from the initial frame therefore targets 3,6,3,6,\ldots,27 ,27, versus 1,1,\ldots,29,29 in our default setting. Both evaluate the single vorticity channel ω\omega.

Table 6: Dataset Settings difference on Vorticity

ResolutionTemporal lengthConditioning input
PDE-JEPA128 ×\times 12830 (0,1,...,29)Target initial state with a support trajectory
Zebra64 ×\times 6410 (0,3,...,27)Current latent state and PDE parameters

PDE-JEPA retains 3030 \times128128\times128 128, whereas Zebra downsamples to 1010 \times6464\times64 64. Both evaluate the single vorticity channel ω\omega.

Q1.2 Comparable experiments

Please refer to answer to CW1.1 and CW1.2.

Q1.3 Writing mistakes

We apologize for the mistakes and thank the reviewer for the detailed suggestion. The revision is made in Table 1 in the main paper.


Q2: conditioning input for Burgers and Heat

We thank the reviewer for bringing this to our attention. This feedback has helped us improve the clarity of the manuscript.

The seven conditioning dimensions encode the viscosity/diffusivity and the forcing averaged over each prediction interval. For both Burgers and Heat, the condition vector is

\left[ \log\beta,\, S_{1,n},\,C_{1,n},\, S_{2,n},\,C_{2,n},\, S_{3,n},\,C_{3,n} \right].$$ For the transition interval $(t_n,t_{n+1})$, define $\Delta t_n=t_{n+1}-t_n$, $t_{\mathrm{mid},n}=(t_n+t_{n+1})/2$, and $\operatorname{sinc}(z)=\sin(z)/z$, with $\operatorname{sinc}(0)=1$. The coefficients are $$\begin{aligned} S_{\ell,n} &= \sum_{\substack{j=1,\ldots,5\\\\ l_j=\ell}} A_j \operatorname{sinc}\left(\frac{\omega_j\Delta t_n}{2}\right) \cos\left(\omega_j t_{\mathrm{mid},n}+\phi_j\right), \\\\ C_{\ell,n} &= \sum_{\substack{j=1,\ldots,5\\\\ l_j=\ell}} A_j \operatorname{sinc}\left(\frac{\omega_j\Delta t_n}{2}\right) \sin\left(\omega_j t_{\mathrm{mid},n}+\phi_j\right), \qquad \ell\in\\{1,2,3\\}. \end{aligned}$$ Thus, the five forcing modes $$A_j,\omega_j,l_j,\phi_j$$ listed in the appendix are aggregated by spatial harmonic into six interval-specific Fourier coefficients. $l_j$ selects the spatial-frequency channel, while temporal information enters through the midpoint phase and interval-averaging factor, without an additional time-conditioning dimension. Here, $$\beta$$ denotes viscosity for Burgers and thermal diffusivity for Heat. **We have added this to Appendix C.1.1 "Burgers and Heat: interval-dependent conditioning."** __________________ **Q3: ablations for geo loss** To address the concerns about geo loss, **we conduct experiments on Vorticity and Wave-2D on PAG without geo loss (Table 7).** > Table 7: PAG without geo loss experiment. | | Vorticity ID | Vorticity OOD | Wave-2D ID | Wave-2D OOD | | :------------------: | :----------: | :-----------: | :--------: | :---------: | | **PAG w/o geo loss** | 0.083 | 0.491 | 0.35 | 0.50 | Removing only this loss, with all other components and training budgets unchanged, increases ID/OOD errors from 0.040/0.397 to 0.083/0.491 on Vorticity and from 0.143/0.321 to 0.350/0.500 on Wave-2D. **The geometry loss provides vital efforts on both datasets.** _______________ **Q4: PSP for HeterNS** Please refer to answer to CW4. _____________

Dear Reviewer zE9o,

As the discussion period is approaching its deadline, we would like to kindly follow up and ask whether you have had an opportunity to review our response, and whether there are any remaining questions or concerns that we could further address.

Your feedback, including both constructive suggestions and points of concern, has been invaluable in helping us improve our work. We have made every effort to carefully address each of your comments and clarify any potential misunderstandings. We sincerely value the opportunity for further discussion and would greatly appreciate any additional feedback you may have.

Thank you again for your time and thoughtful consideration.

Best regards,

Authors of Submission 8314

Reviewer_qHzCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper considers representation learning for latent surrogate models of spatio-temporal physical dynamics. The emphasis is put on solving parametric PDEs, which involves generalizing to unseen PDE parameters. Latent surrogates usually follow an encode-process-decode framework where the encoder and the decoder are trained on reconstruction objectives. Here the authors propose instead to optimize a prediction objective directly in the latent space, leveraging a JEPA framework. Probing experiments demonstrate that the learned representations preserve PDE dynamics and state-space informations, but are less effective than reconstruction-based encoders for forecasting. The authors then propose a three-steps framework : JEPA pretraining using masked spatio-temporal prediction, aligning the temporal geometry of latent and physical trajectories, leveraging physical priors to decompose the evolution equation in the latent space into a data-driven and a physics-aware component. A numerical integrator then performs time stepping and unrolling in the latent space and a decoder projects back the latent predictions to the physical space. Experiments are performed on nine 1D and 2D PDE benchmarks and the authors report substantial improvements over a series of baselines.

Critical strengths
  • Topic : The paper addresses an interesting question for the field of physical AI : how to learn physically representative latent representations that will support spatio-temporal forecasting tasks in a parametric PDE setting. This is a fundamental problem that involves extrapolating to multiple physical conditions, an open challenge for data-driven surrogate models. For this problem, the authors explore the potential of latent JEPA-learned representations with supervision defined directly in the latent space. This offers an alternative to the usual reconstruction objectives, with supervision in the physical space, commonly used in latent PDE surrogate models. Note that the authors are among the first to explore this direction.

  • Experiments : The authors performed an impressive amount of experimental work, they compare their method to 10+ baselines on 9 PDE benchmarks and trained or fine-tuned on these datasets many of these baselines which represents a very large experimental setting. They indicate substantial improvements, in the range of 50%-70% over the baselines, for many of these experiments.

Critical weaknesses
  • Experiments: The experimental comparisons involve heteregeneous settings including different spatial resolutions, temporal contexts, training procedures, model capacities, prior PDE knowledge and probably training budgets although this last point is not addressed. This makes it difficult to determine if the improvements come from the proposed methodology or from any other factor like architecture or training procedure. Overall the relevance of these comparisons is questionable. Instead of extensive experimentations that are not conclusive, it would be more relevant to focus on a limited number of controlled comparisons. For the forecasting experiments, you could compare JEPA and reconstruction¬based pretrained encoders with a similar capacity and downstream predictor, with (i) vanilla prediction, (ii) PAG, (iii) PAG + PSP, while matching all the other factors (resolution, input history, etc). This would better reveal if JEPA itself helps and wether your proposed methodology (PAG + PSP) generalizes to other encoders. A final remark concerns the magnitude of the improvements that exceed 50% on several benchmarks. The magnitude and consistency of the improvements are difficult to explain from the proposed methodology and contributions alone and should be justified. Concerning the analysis of the experiments, there is a discrepancy between the impressive experimental effort and the limited insights extracted from the results.

  • Model motivation : There are several aspects of the model that lack justification or motivation.

First, as observed by the authors, vanilla JEPA alone does not outperform SOTA models (as already noted, the comparison itself is questionable). The ablation in Table 2 reveals that a significant part of the improvement comes from the PAG component. The latter matches directional similarities along physical and latent trajectories, but there is no motivation nor intuition for this addition. Non linear encoder would usually alter the trajectory geometry, what is then the justification for this constraint ? Also note that as opposed to the JEPA encoding philosophy, this relies on a direct supervision in physical space.

Second, from my understanding, the encoder is trained on whole trajectories, while it is used to encode single frames at inference. This introduces an important discrepancy between training and inference and questions the pretraining objective of encoding the dynamics. Paper organization and writing : The main paper lacks essential elements that are deferred to the appendix where they are scattered in different tables and appendices. The reader has to do all the work by himself to gather this information in order to understand the method and the experimental conditions. This motivates a significant reorganization and rewriting of the paper. Note that the figures themselves are overloaded with details and do not provide the guidance required to understand the different aspects of the method.

  • Paper organization and writing : The main paper lacks essential elements that are deferred to the appendix where they are scattered in different tables and appendices. The reader has to do all the work by himself to gather this information in order to understand the method and the experimental conditions. This motivates a significant reorganization and rewriting of the paper. Note that the figures themselves are overloaded with details and do not provide the guidance required to understand the different aspects of the method.
Questions for authors
  1. JEPA benefits -The paper does not establish whether JEPA is a better starting point for the complete forecasting framework than reconstruction-based pretraining. Could the authors compare JEPA and reconstruction-based encoders using identical architectures and downstream predictors, with and without PAG and PSP?
  2. Geometry alignment - What is the justification for assuming that latent trajectories should preserve the « turning-angle » characteristics of physical trajectories? Why should this improve forecasting? Could the authors isolate its contribution from the auxiliary losses?
  3. Spatiotemporal pretraining - JEPA is pretrained on complete trajectories, but the encoder is subsequently applied to individual frames. Could the authors clarify the motivation for this choice?
  4. Comparability of evaluations – Given all the differences in the evaluation setting (table 1), how do the authors justify the meaningfulness of the comparison? How do you justify the magnitude of the improvements
  5. Computational efficiency. The proposed forecaster relies on RK4 integration. This is expensive compared to most baselines and probably prevents scaling. Could the authors comment on that?
7 comments

We sincerely appreciate your insightful and professional comments and your recognition of our paper’s main contributions. Below, we respond to your questions point by point and present our supplementary experiments.


CW1.1: Heterogeneous experimental settings (Resolution, trajectory length, training strategies, model capacity, PDE priors, training budget.)

Clarification: The heterogeneous experimental settings primarily stem from differences in the original baseline configurations. For instance, ENMA uses a spatial resolution of 128 for Advection and Combined Equation, while Zebra conditions on a full trajectory under a given PDE parameter, compared to ENMA's three historical frames. Our initial goal was to preserve each baseline's original configuration for faithful reproduction, which inadvertently led to heterogeneous experimental settings.

To address your concerns regarding resolution and trajectory length, we have conducted additional experiments on major baselines with identical resolution and length settings with PDE-JEPA (as shown in Table 1).

Table 1: Matched resolution and trajectory length experiments. Retrained models are in Bold.

1DZebraGEPSUnisolverDPOTPoseidonEMNAPDE-JEPA
Advect (14 ×\times 256)0.00790.1110.0400.0470.0220.0390.0074
Burgers (25 ×\times 256)0.1150.5530.2390.6570.1220.3340.0428
Heat (25 ×\times 256)0.1540.7990.2410.5360.09640.6170.0274
Wave-B (25 ×\times 256)0.2160.7020.5370.5250.1110.8940.035
Combined (14 ×\times 256)0.00960.0230.00920.0450.1780.3230.0074
2DZebraGEPSUnisolverViT-in-contextLE-PDEMAE-PDEENMAPDE-JEPA
Vorticity (30 ×\times 128$\times$ 128) ID0.1340.09030.09600.350.8670.1330.3010.035
Vorticity (30 ×\times 128$\times$ 128) OOD0.4080.4421.0310.711.010.5100.5130.288

PDE-JEPA still outperforms all baselines. We have also revised the report in Main paper Table 1, 2, Appendix Table 33 and configuration in Appendix Table 10, 11, 13, 14, 15, 17, 18.

To address your concerns on model capacity and PDE prior, we add Table 2 into Appendix C.2, Table 20 from line 2169 to 2191. Regarding PDE priors, we listed below:

  1. Trajectory history: multiple frames as inputs.
  2. Current (latent) state: current single (latent) frame.
  3. Support trajectory: full trajectory as context.
  4. PDE parameters: physics parameter (ξ\xi).

Table 2: Model Details.

FNOGEPSZebraUnisolverLNSMAE-PDE
Capacity50M1M42M55M24M20M
PDE PriorTrajectory historyCurrent state with a support trajectoryTarget initial state with a support trajectoryCurrent state, PDE parametersCurrent latent state and PDE parametersTrajectory history
MPPDPOTPoseidonLE-PDEENMAPDE-JEPA
Capacity115M30M10M20M60M24M
PDE PriorTrajectory historyTrajectory HistoryTrajectory-historyTrajectory history with a support trajectoryTrajectory HistoryCurrent latent state and PDE parameters

CW1.2: JEPA vs Reconstruction-based encoder

To further address your concerns on training strategies and budget, we have conducted controlled ViT comparisons (Table 3), matching architecture, capacity, data settings, conditioning information, downstream training protocols, and training budgets.

Table 3: Strictly matched experiment on resolution, trajectory length, model capacity, PDE prior, training strategy and training budget.

ID RolloutOOD Rollout
ViT encoder+predictorJEPA0.0860.49
JEPA+PAG0.0400.39
JEPA+PAG + PSP0.0340.28
ViT encoder+predictorReconstruct0.0590.47
Reconstruct+PAG0.0630.62
Reconstruct+PAG + PSP0.0570.43
ViT encoder+predictorMAE0.120.51
MAE+PAG0.170.59
MAE+PAG + PSP0.150.54
LNSOriginal0.0590.48
LNSOriginal+PAG0.0620.62
LNSOriginal+PAG + PSP0.0570.45

With PAG and PSP, JEPA achieves best ID/OOD errors, supporting its advantage within the tested framework. We have added this in the main paper in section 5.4.

CW1.3 Magnitude of gains and experimental insights

Clarification: Appendices D.1–D.3 jointly explain the sources of the rollout gains and identify their limitations. We have noted it in section 5.5 and 5.6.

Appendix D.1 presents JEPA representation analyses:

  1. What physical information is encoded, including governing parameters and local physical states? (D.1.2, D.1.5)
  2. How is this information organized globally and spatially in the latent space? (D.1.3)
  3. Are the representations easy to evolve, beyond their ability to reconstruct physical states? (D.1.4, D.1.6)

Appendix D.2 reports PAG and PSP mechanism analyses:

  1. Does geometry projection align latent evolution with physical trajectories? (D.2.3, D.2.6)
  2. Does PSP capture parameter-dependent dynamics more faithfully? (D.2.2)
  3. Do these improvements translate into better forecasting under parameter shifts? (D.2.4)
  4. Does improved long-horizon extrapolation arise from smaller one-step errors or reduced error amplification? (D.2.5)

Appendix D.3 complements these analyses with ablations on anchoring, representation dependence, predictor structure/capacity, physical-tangent supervision, and numerical integration (Tables 38–42).


CW2.1: Motivation for PAG

We clarify the motivation of PAG in three fold:

  1. We observe that physical-information accessibility does not guarantee accurate recursive forecasting (Introduction section observation 2(Figure 1(c–d))).
  2. We are inspired by [1], which demonstrates that latent-space geometry influences control performance. [1] straightens latent trajectory to obtain large performance gain.
  3. We observe a discrepancy between the curvature of physical fields and latent trajectory (opening of section 4.2). These observations motivate investigating temporal geometry adaptation while retaining pretrained information.

CW2.2: Why use PAG

Based on the motivation above, we try to modulate our latent space with geometric alignment.

We compare three geometric conditions: geometric alignment, straightening curvature, no geometric alignment (Table 4).

Table 4 Geometric anaylsis.

VorticityID rollout ↓OOD rollout ↓
JEPA+PAG0.0400.397
JEPA+PAG w/o geo loss0.0830.491
JEPA+PAG with Straightening [1]0.1100.582

As shown in the table, physical-geometry alignment performs best. This supports matching physical trajectory-direction relationships, rather than simply reducing latent curvature, without requiring global geometry preservation. We add this to Appendix D.3.2, paragraph "Choice of geometric reference".

CW2.3: Inconsistence with JEPA

We thank the reviewer for pointing it out. To the best of our knowledge, the core principle of JEPA is predictive learning in representation space without reconstruction supervision. In this way, the encoder can learn semantic rich latent space. PDE-JEPA is pretrained through masked representation prediction and remains frozen during PAG. There is no reconstruction based loss during PAG training. Therefore, we believe our method remains consistent with the core principles of JEPA, and we warmly welcome further discussion with the reviewer on this point.

CW2.4: Training and Inference Discrepancy

We appreciate the reviewer for this thoughtful and detailed question. As stated in section 4.1, lines 250–253, we follows the protocol in V-JEPA-AC in [2]. However, we would like to clarify that frame-wise encoding is used consistently during both downstream PAG and PSP training and inference. JEPA pretraining learns the spatial-temporal representation, while the downstream projector and predictor learns its temporal evolution from frozen frame embeddings.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.