acceptodds
Under review as a conference paper at ICLR 2027

Latent Weather Forecasting with Joint-Embedding Predictive Architectures

Abstract

We investigate Joint-Embedding Predictive Architectures (JEPAs), a self-supervised approach that predicts in embedding space rather than pixel space, as a basis for data-driven weather prediction. We train JEPA-WX on 39 years of ERA5 reanalysis at 5.6° resolution for 12 upper-air variables at three pressure levels. The architecture combines a Vision-Transformer context encoder, an exponential-moving-average (EMA) target encoder, a Transformer predictor with free-running rollout training, and a lightweight convolutional decoder. Training uses three terms: a latent prediction loss, a Slicing Univariate Test regularizer against representation collapse, and an auxiliary reconstruction loss that we find essential for downstream decoding. We show that (i) JEPA-WX outperforms persistence at all evaluated lead times on all 12 variables, with a day-1 anomaly correlation (ACC) of 0.94 on Z500; (ii) JEPA-WX is competitive with a pixel-space U-Net at short leads and outperforms it at day 14 on 8 of 12 variables; and (iii) RMSE grows by 3.2× from day 1 to day 14 for JEPA-WX versus 7.6× for the U-Net, suggesting that the latent bottleneck acts as an implicit regularizer against autoregressive error accumulation. Code, checkpoints and evaluation scripts are released.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Joint-Embedding Predictive Architecture, JEPA, weather forecasting, ERA5, latent prediction, self-supervised learning, medium-range forecasting, Vision Transformer, autoregressive rollout, anomaly correlation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_A8kCOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies whether predicting weather in a learned latent space can reduce error accumulation in forecasts. The proposed model, JEPA-WX, encodes weather fields, predicts how the encoded state changes, and decodes it back into weather fields. It combines a JEPA prediction objective with reconstruction and regularization losses, and trains on multiple forecast steps.

The authors use 39 years of ERA5 data at 5.6° resolution and compare forecasts up to 14 days with persistence and a U-Net. They report smaller errors than persistence and slower error growth than U-Net, although U-Net performs better at short lead times. Tests that remove individual training components suggest that reconstruction and regularization are needed, while multi-step training improves forecast accuracy.

Critical strengths

This paper explores a very useful question: can prediction in a learned latent space improve weather forecasts at longer lead times? The reported results show slower error growth than U-Net and improvements on some variables at longer leads. This makes the approach worth further study.

Critical weaknesses
  1. The comparisons are too limited. The paper evaluates only persistence and a U-Net, with no alternative latent dynamics model or controlled comparison using the same architecture. The training also differs substantially. JEPA-WX uses multi-step training, 80 training epochs, and another 20 epochs for decoder tuning. U-Net uses single-step training for 30 epochs. Table A.5 reports 16.5 GPU-hours for JEPA-WX training versus 3 for U-Net. These differences leave a basic question unanswered: does JEPA reduce error growth, or does the model benefit from its architecture and training? Resolving this requires substantial new experiments.

  2. The evaluation uses test-set targets to fit the calibration factor applied to the main RMSE results (§4.4). This violates the basic requirement that test data remain separate from fitting the prediction procedure, including its post-processing. The appendix does not resolve this concern. Table A.1 reports exactly the same RMSE values as the calibrated results in Table 1, although the text says they are 5–20% higher. Thus, neither the effect of calibration nor the reported uncalibrated results can be verified from the paper. The authors need to fit calibration on validation data and report consistent results on an untouched test set.

Questions for authors

I have no further clarification questions. For a revised submission, I suggest adding non-JEPA latent dynamics baselines and doing comparable multi-step training. Calibration should be fitted on validation data, with consistent calibrated and uncalibrated results reported on the test set.

Reviewer_wsjKOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a controlled study of JEPA-style latent prediction for medium-range weather forecasting. JEPA-WX combines a ViT context encoder, an EMA target encoder, a Transformer predictor trained with a free-running rollout curriculum (K ∈ {1,2,4,8}), and a lightweight convolutional decoder, trained with a composite loss (latent MSE + SIGReg anti-collapse regularizer + auxiliary pixel reconstruction). Experiments use ERA5 at 5.6° (32×64, 12 channels: Z/T/U/V at 500/700/850 hPa), trained on 1979–2017, validated on 2018, tested on 2019–2020 with 100 starts. Main findings: (i) beats persistence everywhere; (ii) day-1 Z500 ACC of 0.94, below a pixel-space U-Net at short leads but better at day 14 on 8/12 variables; (iii) RMSE grows 3.2× vs. 7.6× from day 1 to 14, which the authors attribute to the latent bottleneck filtering chaotic high-frequency modes. The paper is explicitly framed as a proof of concept, not a SOTA attempt.

Critical strengths
  1. The paper asks an interesting and well-scoped question: whether latent-space prediction provides a useful inductive bias for autoregressive weather forecasting. The controlled, low-resolution setting makes the proposed architecture relatively inexpensive to reproduce and study.

  2. The paper is reasonably transparent about several limitations and provides detailed hyperparameters, a compute-budget table, and ablations of the regularization, reconstruction, and rollout components. The finding that a reconstruction objective appears necessary for decoding useful atmospheric fields is potentially informative, although the corresponding ablation requires a fairer training-budget comparison.

Critical weaknesses
  1. The headline RMSE evaluation uses test-set information. Section 4.4 states that the amplitude-calibration factor is computed from the Z500 truth over the test starts, after which the calibrated JEPA-WX results are reported in the main tables. This is test leakage, and the same calibration protocol is not applied to the U-Net. Moreover, Table A.1 is described as uncalibrated but contains exactly the same Z500 RMSE values as the calibrated main table, despite the text claiming that the uncalibrated errors are 5–20% higher. This contradiction prevents assessment of the actual uncalibrated performance and affects the validity of the main RMSE comparison.

  2. The experiments do not isolate the claimed effect of the latent bottleneck. JEPA-WX differs from the U-Net in architecture, training budget, EMA supervision, auxiliary losses, decoder fine-tuning, and, most importantly, multi-step free-running rollout training, whereas the U-Net is trained only for one-step prediction. The reported 3.2x versus 7.6x error-growth comparison is also sensitive to the different day-1 errors; persistence grows by only about 1.95x under the same ratio while remaining a poor forecast. Without a pixel-space baseline trained with the same rollout curriculum, matched capacity and compute, the slower error growth cannot be attributed to latent-space prediction.

Questions for authors
  1. Can the authors re-estimate the calibration factor using only the 2018 validation set, apply the same calibration protocol to every learned model, and report both calibrated and uncalibrated test results? Why are all values in the “uncalibrated” Table A.1 identical to the calibrated JEPA-WX values in Table 1?

  2. Does JEPA-WX still show slower error growth than a U-Net trained with the same free-running rollout curriculum and matched parameter count, number of optimization steps, and compute budget?

  3. Can the authors provide spectral or scale-separated diagnostics that directly test the proposed mechanism that the latent bottleneck preferentially filters unpredictable high-frequency modes? Please also report results over multiple training seeds and paired block-bootstrap confidence intervals for the long-lead comparisons.

Reviewer_J9TXOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

JEPA-WX forecasts 12 upper-air ERA5 channels on a 32×64 grid using a ViT context encoder, EMA target encoder, Transformer latent predictor, and convolutional decoder. Training combines latent prediction, distribution regularization, context reconstruction, and an increasing rollout horizon; a second stage fits the decoder to predicted tokens. Evaluation uses 100 starts in 2019–2020, persistence, and a pixel-space U-Net. The reported result is a short-lead accuracy versus long-lead stability tradeoff: Z500 RMSE changes from 0.082 to 0.260 for JEPA-WX and 0.054 to 0.408 for U-Net between days 1 and 14. The paper attributes this to filtering unpredictable information in latent space.

Critical strengths
  • The paper makes a useful, concrete attempt to connect predictive representation learning with decoded atmospheric forecasting. Sections 3.3–3.4 specify how reconstruction, anti-collapse regularization, and autoregressive training interact, while decoder fine-tuning explicitly uses predicted tokens. This exposes a practical tension between representations that predict well internally and representations that retain information needed for physical outputs. The negative reconstruction ablation is therefore informative as a diagnostic, even though its unequal training duration prevents a firm conclusion.
  • Tables 1–2 and A.2–A.3 report multiple variables and lead times, revealing limitations rather than showing only favorable snapshots. The reported day-14 Z500 improvement is an absolute RMSE difference, not merely a growth-ratio improvement, and wind-channel results suggest a useful hypothesis for further study. The explicit acknowledgement of coarse resolution, under-dispersion, a single seed, and very low day-14 ACC appropriately limits operational claims. A small-scale investigation can be valuable without matching expensive operational systems.
Critical weaknesses
  • Section 4.4 fits calibration using test truth, while Table A.1 duplicates calibrated RMSE despite claiming higher uncalibrated errors. Moreover, src/evaluate.py:33–43,106 ignores calendar dates and scales full fields before subtracting climatology, so the claimed ACC invariance does not generally hold; together, these issues leave both reported scores and model rankings uncertain.
  • Contrary to Section 4.1, src/dataset.py:62–86 saves normalization statistics but returns raw fields, changing channel weighting and leaving normalized RMSE unexplained. requirements.txt also omits lejepa, triggering a different penalty in src/train_jepa.py:18–34 when unavailable; the supplied pipeline therefore does not establish the preprocessing or objective that produced the reported checkpoints.
  • Sections 4.2/4.5 and Table A.5 give JEPA multistep training and 16.5 GPU-hours versus single-step training and 3 GPU-hours for U-Net, with different architectures. Table 3 also compares reconstruction ablations trained for 20 versus 80 epochs, so neither the forecast advantage nor reconstruction benefit is isolated from training and capacity differences.
  • Sections 5.3/6 infer physical filtering from smoothness and relative error growth, but JEPA starts with higher error, and Figure 1’s latent has twice as many coordinates as the input. Without climatology RMSE, spectral/balance diagnostics or uncertainty estimates, the day-14 ACC of 0.110 does not establish that the reported improvement reflects useful, physically realistic dynamics.
  • Sections 2–3 omit LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics, the SIGReg source, and SciVid: Cross-Domain Evaluation of Video Models in Scientific Applications, which evaluates V-JEPA for eight-day weather forecasting. SciVid uses frozen video features rather than weather-trained latent rollouts, but these omissions leaves the borrowed components and narrower methodological novelty insufficiently distinguished.
Questions for authors
  • Can you provide validation-calibrated and uncalibrated scores with correct calendar indexing and anomaly scaling, explain the identical Tables 1/A.1, and specify the actual calibration factor and fitting samples?
  • Which preprocessing, regularizer implementation/version, and evaluation code produced the supplied checkpoints and twelve-channel tables, given the differences from the released default pipeline?
  • Does the long-lead advantage persist with equalized architectures, training budgets, and rollout curricula, and with an 80-epoch no-reconstruction run using the same decoder fine-tuning procedure?
  • What are the absolute improvements over climatology, spectral/balance diagnostics, and uncertainty on paired differnces across training seeds and temporally dependent test starts?
  • Can you position the contribution against the pre-existing latent-weather and SIGReg methods above, and provide a comparable reconstruction-trained latent baseline or narrow the novelty/transfer claims?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.