acceptodds
Under review as a conference paper at ICLR 2027

Earth-JEPA:Toward efficient and transferable Earth-system latent representation learning

46 views

Abstract

Deep learning for Earth system forecasting has advanced rapidly, mostly through end-to-end supervision on reanalysis data for individual tasks. This training paradigm tends to be less efficient on spatiotemporally smooth fields, while making cross-task adaptation relatively expensive. To overcome these challenges, we introduce Earth-JEPA (E-JEPA), a reconstruction-free, multi-resolution, modality-wise Joint-Embedding Predictive Architecture for efficient and transferable Earth-system prediction. Modality-specific branches encode satellite observations, atmospheric reanalysis and geographic fields into a multi-resolution latent state. A latent-pyramid predictor with Scale-Specific Attention forecasts future representations, each supervised by its corresponding exponential moving average (EMA) target through a State-Transition Loss. On WeatherBench2, Earth-JEPA achieves 17.8% higher 24-hour U850 anomaly correlation coefficient (ACC) than a leading large-scale supervised weather forecaster using only about 12.8% of their estimated training compute. Experiments on SEVIR and our proposed EarthCastBench benchmark demonstrate transferability across time-intervals and sensors, robustness to missing reanalysis, and strong performance on dense tasks, e.g., precipitation nowcasting, and forecasting of surface and atmospheric variables. These results suggest a promising direction for efficient and transferable Earth-system latent representation learning. Code and data will be released upon publication.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, self-supervised learning, earth system modeling, foundation models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_n4HcOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Earth-JEPA, a multi-resolution Joint-Embedding Predictive Architecture that fuses FY-4B infrared brightness temperatures, ERA5, and static geography into a 4-scale latent pyramid, using full attention at the coarsest scale and axial attention at finer scales to forecast future latent states supervised by an EMA target encoder via a State-Transition Loss.

Critical strengths

The problem is well motivated and the reporting is candid, evaluating across regional nowcasting, global medium-range forecasting, and cross-sensor transfer while honestly showing in Table 3 that Pangu-Weather and GraphCast reach 0.979 and 0.984 in 24-hour U850 ACC compared to 0.888 for E-JEPA, and noting that E-JEPA trails RainPro-2R on 5 of 6 SEVIR metrics. The low-cost training recipe and public benchmark have practical value, providing a backbone pretrained in 29.7 GPU-hours and a Drop-ERA5 setting where RMSE rises by only 4.1%, which is useful when reanalysis data is unavailable in real time.

Critical weaknesses

The headline accuracy and efficiency claim rests on a mischaracterized baseline and mixed compute accounting, as ClimaX's 0.754 ACC comes from an adapted regime of 10,000 rows, 40 epochs, and a 1.5-degree grid while WeatherGFM at 0.508 falls below persistence at 0.524, indicating poorly tuned baselines; moreover, the 12.8% compute figure counts only E-JEPA's 29.7 GPU-hour pretraining cost whereas the 0.888 ACC comes from the fine-tuned model costing 138.2 GPU-hours, which is 59.5% of ClimaX's 232.3 GPU-hours, while the frozen model matching 29.7 GPU-hours achieves 0.829 ACC for a 9.9% gain.

The transfer comparisons lack controls needed to attribute gains to pretraining, omitting a same-architecture supervised-from-scratch baseline on WeatherBench2 and failing to explain how a backbone pretrained on 4 km regional patches is adapted across a 40x scale mismatch to a global 1.5-degree grid.

The ablations do not support the claimed role of the design components, as the Supervised control in Table 5 is more accurate with a TBB RMSE of 5.32 K versus 5.39 K and ERA5 nRMSE of 0.0886 versus 0.0904, while tube masking, FPN, and axial attention each change probe error by less than 0.01%, dropping the State-Transition Loss changes TBB RMSE by 0.06%, and removing EMA raises TBB RMSE by only 0.75% despite a 92% drop in effective rank.

Questions for authors

Please clarify for every baseline in Tables 1 to 3 what data it was pretrained on and whether it is frozen or fine-tuned, and report WeatherBench2 results for E-JEPA's architecture trained from scratch alongside fine-tuned E-JEPA versus baselines under the same total 138.2 GPU-hours.

Given that ERA5 arrives with delay in operations, how do the main results change under realistic latency, and since ERA5 is hourly, how are the +15, +30, and +45 minute future targets constructed and is surface-pressure forecasting on interpolated reanalysis meaningful?

How are the 121x240 global 1.5-degree WeatherBench2 fields fed into encoders pretrained on 128x128 4 km regional patches in terms of patch size, geography conditioning, and positional handling?

6 comments
AuthorsOpenReview ↗

Response to Reviewer n4Hc (Part 1 / 3)

Thank you for the thorough and constructive review. We appreciate your recognition of the motivation, the low-cost training recipe, and the practical value of EarthCastBench.

In the revised manuscript, we corrected the cost–accuracy comparison and the WeatherGFM evaluation, added same-architecture supervised controls on WeatherBench2 and SEVIR, and clarified how we adapt a regionally pretrained backbone to global forecasting. We address each point below.


W1: The headline accuracy and efficiency claim rests on a mischaracterized baseline and mixed compute accounting, as ClimaX's 0.754 ACC comes from an adapted regime of 10,000 rows, 40 epochs, and a 1.5-degree grid while WeatherGFM at 0.508 falls below persistence at 0.524, indicating poorly tuned baselines; moreover, the 12.8% compute figure counts only E-JEPA's 29.7 GPU-hour pretraining cost whereas the 0.888 ACC comes from the fine-tuned model costing 138.2 GPU-hours, which is 59.5% of ClimaX's 232.3 GPU-hours, while the frozen model matching 29.7 GPU-hours achieves 0.829 ACC for a 9.9% gain.

A1: Thank you for this careful audit. We have corrected both the compute pairing and the WeatherGFM evaluation in the Abstract, Introduction, Figure 1, Section 4.2, Table 3, and Table 6:

  1. Consistent frozen-transfer pairing: Our revised headline pairs frozen E-JEPA with its complete pretraining-plus-adaptation cost (29.7 + 29.3 = 59.0 GPU-hours). On WeatherBench2, frozen E-JEPA achieves 0.9119 24-h U850 ACC (about +21% over ClimaX's 0.754) and 0.9192 four-variable mean 24-h ACC (about +17% over ClimaX's 0.7855), using only about 23% of ClimaX's total cost (232.3 + 26.6 = 258.9 GPU-hours).
  2. WeatherGFM & baseline tuning: In our initial submission, WeatherGFM's WeatherBench2 readout had not converged under the preliminary adaptation schedule. Retraining the readout on the same frozen WeatherGFM backbone under our unified 30-epoch (5-epoch early stopping) adaptation protocol raised its 24-h U850 ACC from 0.508 to 0.766 (well above persistence at 0.524), and we updated its GFLOPs (1,818.6) to include the full in-context prompt–query sequence. All adapted baselines share identical splits and optimization budgets (Appendix C.3).

W2: The transfer comparisons lack controls needed to attribute gains to pretraining, omitting a same-architecture supervised-from-scratch baseline on WeatherBench2 and failing to explain how a backbone pretrained on 4 km regional patches is adapted across a 40x scale mismatch to a global 1.5-degree grid.

A2: Thank you for pointing out these missing controls and interface details. We have added same-architecture supervised-from-scratch controls on both WeatherBench2 (Table 3) and SEVIR (Table 4), and detailed the regional-to-global interface in Section 4.2 and Appendix C.3:

  • Pretraining value vs. supervised controls: On WeatherBench2 (Table 3), frozen E-JEPA approaches the supervised-from-scratch control (24-h mean ACC 0.9192 vs. 0.9251) while keeping the backbone frozen and reusable across tasks. Fine-tuning closes the gap further (0.9225). On SEVIR (Table 4), frozen E-JEPA outperforms the specialized RainPro-2R by about 12% on CSI-p16 (0.4494 vs. 0.4012) and surpasses the supervised control on CSI-p16 and LPIPS.
  • Regional-to-global interface: Each 121x240 global state is bilinearly resampled as a full field onto the 128x128 backbone grid, and a two-layer residual adapter (with a zero-initialized final 1x1 projection) maps all 121 fields into the 195 reanalysis channels (Appendix C.3). Achieving 0.9192 24-h mean ACC under this simultaneous 40x spatial-scale jump, 24x temporal-cadence shift, and missing-satellite setting further underscores the strength and transferability of our pretrained latent representations.
AuthorsOpenReview ↗

Response to Reviewer n4Hc (Part 2 / 3)

W3: The ablations do not support the claimed role of the design components, as the Supervised control in Table 5 is more accurate with a TBB RMSE of 5.32 K versus 5.39 K and ERA5 nRMSE of 0.0886 versus 0.0904, while tube masking, FPN, and axial attention each change probe error by less than 0.01%, dropping the State-Transition Loss changes TBB RMSE by 0.06%, and removing EMA raises TBB RMSE by only 0.75% despite a 92% drop in effective rank.

A3: Thank you for this sharp observation. We agree that effective rank and short-horizon regional probes alone do not prove downstream forecasting gains. In Section 4.4, Table 5, and Appendix Tables 15–17, we separated the regional reconstruction control from the WeatherBench2 target-task control and added downstream WeatherBench2 (24-h and 72-h) objective ablations:

Table R3-1: WeatherBench2 objective ablations and target-task supervision.

Variant or control24-h nRMSE ↓72-h nRMSE ↓Mean nRMSE ↓24-h mean ACC ↑72-h mean ACC ↑
w/o EMA0.15500.26690.210950.80740.4016
w/o state-transition loss0.11560.21140.163450.89610.6497
Supervised (from scratch)0.09830.18050.139360.92510.7524
Earth-JEPA (frozen)0.10270.18980.146230.91920.7245
  • Clear downstream gains: While uncollapsed fine scales compensate over 60-min regional horizons, 72-h global rollouts rely on synoptic-scale dynamics: removing State-Transition Loss increases WeatherBench2 mean nRMSE by about 12% (0.14623→0.163450.14623 \to 0.16345), and removing EMA increases it by about 44% (0.14623→0.210950.14623 \to 0.21095).
  • Component & control roles: Masking, FPN, and axial attention provide missing-data robustness, cross-scale fusion, and efficiency at no cost to representation quality (Section 4.4). Meanwhile, both the regional reconstruction control (5.32 K vs. 5.39 K) and the WB2 supervised control confirm that frozen Earth-JEPA approaches end-to-end target supervision while learning an 18.13x more compact, task-reusable latent target.

Q1: Please clarify for every baseline in Tables 1 to 3 what data it was pretrained on and whether it is frozen or fine-tuned, and report WeatherBench2 results for E-JEPA's architecture trained from scratch alongside fine-tuned E-JEPA versus baselines under the same total 138.2 GPU-hours.

A4: Thank you for requesting these provenance and control details. We have clarified every baseline in Appendix C.3 and Appendix D.2: ClimaX (CMIP6/ERA5 pretraining) and WeatherGFM (multimodal weather pretraining) adapt frozen backbones; Earthformer is pretrained on EarthCastBench and adapted frozen; Earthformer Sup. and CasCast Sup. are task-supervised on precipitation; Pangu-Weather and GraphCast are trained end-to-end on global 0.25° ERA5. Across baselines and controls, training follows a unified 30-epoch budget with 5-epoch validation early stopping, and Table 12 reports the actual measured GPU-hours:

Table R3-2: WeatherBench2 controls and baselines with complete GPU-hour costs.

MethodPretrainingAdaptationTotal24-h U850 ACC ↑72-h U850 ACC ↑
ClimaX (frozen)232.326.6258.90.7540.308
Earthformer (frozen)72.231.2103.40.7650.329
WeatherGFM (frozen)192.90.3193.20.7660.346
E-JEPA (supervised, from scratch)--32.632.60.92190.7414
E-JEPA (frozen)29.729.359.00.91190.7064
E-JEPA (fine-tuned)29.740.269.90.9160.719

Frozen E-JEPA uses 59.0 GPU-hours in total (and fine-tuned 69.9 GPU-hours, well below the earlier budget), while the pretrained backbone (29.7 h) can be reused across tasks.

AuthorsOpenReview ↗

Response to Reviewer n4Hc (Part 3 / 3)

Q2: Given that ERA5 arrives with delay in operations, how do the main results change under realistic latency, and since ERA5 is hourly, how are the +15, +30, and +45 minute future targets constructed and is surface-pressure forecasting on interpolated reanalysis meaningful?

A5: Thank you for raising this practical question. We have summarized the Drop-ERA5 comparison in Section 5 and added the full comparison in Appendix B (Table 10), alongside the alignment details in Appendix B.2:

Table R3-3: Effect of ERA5 input availability on EarthCastBench (matching Appendix Table 10).

MethodRain RMSE ↓Rain MAE ↓CSI ↑MSL RMSE ↓MSL MAE ↓SP RMSE ↓SP MAE ↓
ClimaX0.80820.26510.75710.57850.43700.55710.4287
Earthformer0.81690.27290.75150.56720.43830.55780.4365
WeatherGFM0.75340.25150.76930.56870.42980.54490.4203
E-JEPA (Drop-ERA5)0.71790.24270.76760.55020.41570.53650.4121
E-JEPA0.67760.22500.78180.50400.37960.50030.3800

Under Drop-ERA5 (all ERA5 inputs removed at inference), Earth-JEPA stays close to its full-input performance and still achieves lower precipitation and pressure RMSE than every full-input baseline. Sub-hourly targets are linearly interpolated between adjacent hourly ERA5 analyses (Appendix B.2), applied identically across all models to evaluate agreement with aligned reanalysis references.


Q3: How are the 121x240 global 1.5-degree WeatherBench2 fields fed into encoders pretrained on 128x128 4 km regional patches in terms of patch size, geography conditioning, and positional handling?

A6: Thank you for asking for these implementation details. As detailed in Section 4.2 and Appendix C.3, each 121x240 global field is bilinearly resampled to the 128x128 backbone grid. The 4x4 patch embedding, 8x8 Swin windows, and relative-position bias tables remain unchanged. Global orography, latitude, and longitude fill the geographic slots (with unavailable land-cover, solar-zenith, and 9 TBB slots zeroed, while 4 time encodings are retained). The decoder applies circular longitude padding and replicated polar padding back to 121x240.


Additional Clarification & Core Advantages

"...honestly showing in Table 3 that Pangu-Weather and GraphCast reach 0.979 and 0.984 in 24-hour U850 ACC compared to 0.888 for E-JEPA, and noting that E-JEPA trails RainPro-2R on 5 of 6 SEVIR metrics."

We report these candidly to position Earth-JEPA as a resource-efficient, reusable foundation model trained on 8x A800 GPUs (3.40 TB archive) rather than a native 0.25° global supervised model (where Pangu/GraphCast use ~961x/321x our cost). On SEVIR, after unifying frozen transfer, E-JEPA outperforms RainPro-2R by about 12% on CSI-p16 and occupies the top two positions on 5 of 6 deterministic metrics (Table 4).

Table R3-4: Summary of core advantages and supporting evidence.

Core AdvantageKey ResultLocation
EfficiencyWB2 cost: 29.7 + 29.3 GPU-h (E-JEPA pretrain + frozen adaptation), 0.10% of Pangu-Weather’s A800-equivalent costFig. 1, Table 6
TransferabilitySEVIR performance: beats previous supervised SOTA RainPro-2R (ICLR 2026) on 5/6 metrics, with +12.0% CSI-p16Table 4, Sec. 4.3
AccuracyWB2 performance vs. ClimaX: +47% mean ACC, 39% lower mean MAE, 38% lower mean RMSETables 3, 13, 14
Scalability (preliminary)WB2 72-h mean ACC: 0.678 → 0.725 → 0.749 (50M → 113M → 239M, frozen E-JEPA)Table 3, Fig. 5

We have uploaded the anonymous supplementary material with our model definitions and evaluation code, and will release the full training code, checkpoints, and EarthCastBench upon acceptance. Given the unified cost-accuracy pairing, new supervised experiments, and downstream ablations, we respectfully hope our endeavors can help inform your reassessment of our work.

Reviewer_YqLJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Earth-JEPA for Earth-system representation learning, which combines modality-specific encoding, multi-resolution latent prediction, and state-transition supervision with exponential-moving-average targets. Experiments demonstrate the effectiveness of Earth-JEPA.

Critical strengths

1.Predicting future latent states at multiple spatial resolutions is well motivated for dense Earth-system tasks, and the modality-specific encoding design is easy to follow.

Critical weaknesses
  1. The methodological novelty is limited, and the work appears incremental. Most components, including Swin encoding, EMA targets, tube masking, axial attention, and FPN fusion, are established techniques. The authors should clarify the novelty beyond extending JEPA to multi-resolution Earth-system modeling.

  2. The paper has some weaknesses in the experiments: (1) In Table 5, removing the state-transition loss barely affects TBB RMSE (5.3918 vs. 5.3949), while removing masking or FPN has negligible impact. Higher effective rank alone does not demonstrate forecasting benefits. Further downstream validation is needed. (2) The SEVIR experiment transfers only the encoder and fusion modules, not the latent predictor. Without a non-pretrained baseline using the same architecture, the benefits of pretraining remain unclear.

Questions for authors
  1. Can the authors provide accuracy and pretraining-plus-adaptation cost together for every model in Figure 1, distinguishing measured costs from hardware-normalized estimates?
6 comments
AuthorsOpenReview ↗

Response to Reviewer YqLJ (Part 1 / 3)

Thank you for the thoughtful and constructive feedback. We are pleased that you found the multi-resolution prediction task well motivated and the modality-specific encoding clear.

In the revised manuscript, we strengthened the case for our methodological contribution by expanding the discussion of multi-resolution JEPA and adding WeatherBench2 ablations that show how the State-Transition Loss and EMA targets contribute to downstream forecasting. We also added same-architecture supervised controls to assess the value of pretraining. We address each point below.


W1: The methodological novelty is limited, and the work appears incremental. Most components, including Swin encoding, EMA targets, tube masking, axial attention, and FPN fusion, are established techniques. The authors should clarify the novelty beyond extending JEPA to multi-resolution Earth-system modeling.

A1: Thank you for asking us to clarify the contribution. We use established building blocks to give the community a simple, reproducible multi-resolution JEPA design to build on and improve. To our knowledge, Earth-JEPA is the first JEPA to predict future latent states at multiple spatial resolutions for Earth-system forecasting. The contribution is how we define and learn that prediction target.

First, we predict a spatial pyramid at each forecast step. Recent work H-JEPA [1] independently reflects a related intuition about the value of hierarchy in latent prediction. H-JEPA organizes its hierarchy around different time horizons for visual planning. Earth-JEPA organizes its hierarchy around spatial resolution, with each level predicting a future state at the same forecast step. Retaining the fine-scale prediction pathways reduces brightness-temperature RMSE (Table 5). The latent maps in Figure 6 illustrate that the full predictor preserves more of the spatial detail in the target maps across the pyramid. Together, these results show why we predict at multiple resolutions. The coarse levels capture broad weather patterns, and the finer levels retain their local structure.

Second, we supervise changes in state using slowly updated EMA targets. The State-Transition Loss anchors the first future state and then supervises the changes between successive states at every resolution. The EMA encoder updates gradually, giving the predictor a stable reference as it learns those changes. In the additional WeatherBench2 evaluations in Section 4.4 and Appendix E.1, removing the State-Transition Loss increases mean nRMSE by 12%, and removing EMA increases it by 44%. Removing EMA also sharply reduces the effective rank of the coarsest representation. Together, the two designs give the predictor a stable learning target across scales, which we later show is critical for preventing representation collapse without any auxiliary losses or explicit covariance regularizers.

Pangu-Weather helped establish AI-based medium-range global forecasting by showing that a 3D Earth-specific Transformer could outperform ECMWF’s operational numerical weather prediction system. Earth-JEPA advances a different training objective for building efficient, reusable Earth-system foundation models that transfer across tasks and dataset sources. We pretrain the model to predict how a compact latent pyramid evolves. The same frozen backbone can then be adapted across sensors, spatial grids, and forecast intervals, as we show in regional precipitation nowcasting, surface-variable forecasting, global atmospheric prediction, and radar nowcasting.

[1] H-JEPA: End-to-end learning of hierarchical world models for visual planning. Wancong Zhang, Basile Terver, Michael Rabbat, Yann LeCun, and Randall Balestriero. arXiv 2026.

AuthorsOpenReview ↗

Response to Reviewer YqLJ (Part 2 / 3)

W2 (1): In Table 5, removing the state-transition loss barely affects TBB RMSE (5.3918 vs. 5.3949), while removing masking or FPN has negligible impact. Higher effective rank alone does not demonstrate forecasting benefits. Further downstream validation is needed.

A2.1: Thank you for this important point. We agree that effective rank alone does not establish forecasting benefits. In Section 4.4, Table 5, and Appendix Tables 15–17, we have separated the architectural components from the training objectives and added downstream WeatherBench2 (24-h and 72-h) ablations:

Table R2-1: WeatherBench2 learning-objective ablations and target-task supervision.

Variant or control24-h nRMSE ↓72-h nRMSE ↓Mean nRMSE ↓24-h mean ACC ↑72-h mean ACC ↑
w/o EMA0.15500.26690.210950.80740.4016
w/o state-transition loss0.11560.21140.163450.89610.6497
Supervised (from scratch)0.09830.18050.139360.92510.7524
Earth-JEPA (frozen)0.10270.18980.146230.91920.7245
  • Downstream objective gains: While uncollapsed fine scales compensate over 60-min regional horizons, 72-h global rollouts rely on synoptic-scale dynamics: removing State-Transition Loss increases WeatherBench2 mean nRMSE by about 12% (0.14623→0.163450.14623 \to 0.16345), and removing EMA increases it by about 44% (0.14623→0.210950.14623 \to 0.21095), proving clear downstream forecasting benefits.
  • Role of components & controls: Masking, FPN, and axial attention provide missing-data robustness, cross-scale fusion, and computational efficiency at no cost to representation quality (Section 4.4). Moreover, both the regional reconstruction control (5.32 K vs. 5.39 K) and the WB2 supervised control confirm that frozen Earth-JEPA approaches end-to-end target supervision while learning an 18.13x more compact, task-reusable latent target.

W2 (2): The SEVIR experiment transfers only the encoder and fusion modules, not the latent predictor. Without a non-pretrained baseline using the same architecture, the benefits of pretraining remain unclear.

A2.2: Thank you for pointing out this inconsistency in our initial draft. While we originally explored different adaptation scopes, we have now unified all frozen transfer experiments—including SEVIR—so that both the encoder and the Latent-Pyramid Predictor remain frozen, with only the radar adapter and task decoder (48.64M trainable parameters) trained (Section 4.3, Appendix C.3). We also added a same-architecture supervised-from-scratch control on SEVIR:

Table R2-2: SEVIR radar VIL nowcasting (5 observed to 20 future frames).

MethodCSI ↑CSI-p4 ↑CSI-p16 ↑HSS ↑LPIPS ↓SSIM ↑
RainPro-2R (ICLR 2026)0.35000.37620.40120.44740.24230.5912
E-JEPA (supervised)0.35420.39500.44460.45430.24050.5920
E-JEPA (frozen)0.35010.39220.44940.44840.23630.5911

SEVIR tests a demanding triple transfer (sensor: satellite TBB → radar VIL; cadence: 15-min → 5-min; sequence: 8→4 to 5→20). Frozen E-JEPA outperforms the specialized RainPro-2R by about 12% on CSI-p16 and about 4% on CSI-p4 with lower LPIPS, and surpasses the supervised control on CSI-p16 and LPIPS without updating the backbone.

AuthorsOpenReview ↗

Response to Reviewer YqLJ (Part 3 / 3)

Q1: Can the authors provide accuracy and pretraining-plus-adaptation cost together for every model in Figure 1, distinguishing measured costs from hardware-normalized estimates?

A3: Thank you for requesting the complete breakdown. Figure 1, Table 6, and Table 12 now pair each model's accuracy with its complete pretraining-plus-adaptation cost in GPU-hours. The totals for Pangu-Weather and GraphCast and the pretraining costs for ClimaX and WeatherGFM are hardware-normalized estimates. All other listed pretraining and adaptation costs were measured on A800 GPUs.

Table R2-3: Accuracy and complete training cost for all models in Figure 1 (GPU-h).

MethodPretrainingAdaptationTotal24-h U850 ACC ↑24-h MSLP ACC ↑
Pangu-Weather----56,713.80.9790.997
GraphCast----18,953.80.9840.998
ClimaX232.326.6258.90.7540.843
Earthformer72.231.2103.40.7650.850
WeatherGFM192.90.3193.20.7660.868
E-JEPA-50M (frozen)17.317.034.30.89330.9554
E-JEPA (113M, frozen)29.729.359.00.91190.9655
E-JEPA-239M (frozen)43.740.283.90.92060.9704

For Pangu-Weather, the budget covers both the 6-h and 24-h rollout networks (147,456 V100 GPU-h), converted via the V100-to-A800 ratio (Appendix D.2). Frozen E-JEPA uses 59.0 GPU-hours in total (about 23% of ClimaX's 258.9 h), and the pretrained backbone can be reused across tasks without retraining.


Summary of Core Advantages

Table R2-4: Summary of core advantages and supporting evidence.

Core AdvantageKey ResultLocation
EfficiencyWB2 cost: 29.7 + 29.3 GPU-h (E-JEPA pretrain + frozen adaptation), 0.10% of Pangu-Weather’s A800-equivalent costFig. 1, Table 6
TransferabilitySEVIR performance: beats previous supervised SOTA RainPro-2R (ICLR 2026) on 5/6 metrics, with +12.0% CSI-p16Table 4, Sec. 4.3
AccuracyWB2 performance vs. ClimaX: +47% mean ACC, 39% lower mean MAE, 38% lower mean RMSETables 3, 13, 14
Scalability (preliminary)WB2 72-h mean ACC: 0.678 → 0.725 → 0.749 (50M → 113M → 239M, frozen E-JEPA)Table 3, Fig. 5

We have uploaded the anonymous supplementary material with our model definitions and evaluation code, and will release the full training code, checkpoints, and EarthCastBench upon acceptance. We believe our new downstream ablations, same-architecture controls, and complete cost-accuracy tables address all your concerns, and we kindly invite you to reconsider your rating in light of these additions.

Reviewer_AvYTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The article is about foundation models for remote sensing time series. Most models are trained to reconstruct physical fields from reanalysis data, which allocates most of its resources to smooth, highly predictable content and limits the learned representation to a single task.

The authors propose Earth-JEPA, an adaptation of JEPA. Modality-specific encoders convert satellite observations, reanalysis, and static geography into a multi-resolution latent pyramid. A masked latent pyramid predictor, which uses full attention at the coarsest scale and axial attention at finer scales, along with pyramid-style cross-scale fusion, forecasts future latent states as cumulative increments. Each scale is supervised by the EMA teacher through a reconstruction loss on the temporal transition of the embeddings, not the embeddings themselves. Then, task-specific heads adapt the frozen or fine-tuned backbone to downstream tasks. The authors also introduce EarthCastBench, a benchmark that aligns FY-4B satellite data, ERA5, IMERG precipitation, and geography.

Experiments cover forecasting and nowcasting tasks. Earth-JEPA outperforms some baseline methods on EarthCastBench and WeatherBench2 with lower computing power, remains robust when ERA5 inputs are removed, and is competitive. However, it remains well behind native-resolution supervised models.

Critical strengths
  1. Using JEPA-style training on Earth-system data with a multi-resolution latent pyramid is a reasonable approach. Predicting in representation space avoids the reconstruction of redundant, smooth fields. Supervising every scale preserves the fine-scale information necessary for dense tasks.

  2. Supervising temporal transitions rather than full states is an elegant design reminiscent of video coding. Only the initial state requires precise matching; then, the model learns the simpler-to-represent transitions. When combined with cumulative-increment parameterization, this gives the predictor a natural inductive bias toward learning state evolution.

Critical weaknesses
  1. The main text does not adequately explain several parts of the model, so the reader must constantly refer to the appendix for notation (e.g., Eq. 1 and the predictor).

  2. Figures 4 and 9 use red boxes to highlight regions, but they do not provide zoomed-in views. This makes it very difficult to assess the qualitative comparison at the given resolution.

  3. The 17.8% increase in accuracy with 12.8% of computing power pairs fine-tuned accuracy with the cost of pretraining. Including 108 hours of fine-tuning brings the total to approximately 59% of ClimaX computing power, and the gain of the frozen model is about 9.9%.

Questions for authors

Could you add more explanation and notation to the Methods section to improve readability?

3 comments
AuthorsOpenReview ↗

Response to Reviewer AvYT (Part 1 / 2)

Thank you for the detailed and constructive feedback. We are pleased that you found the multi-resolution JEPA approach well motivated and the State-Transition Loss a natural way to learn how the state evolves.

In the revised manuscript, we added same-architecture supervised controls on WeatherBench2 and SEVIR and now report frozen-transfer accuracy alongside the full pretraining and adaptation cost. These additions strengthen the evidence for Earth-JEPA’s efficiency and transferability. Following your suggestions, we also clarified the method and notation and added zoomed-in views to Figures 4 and 9. We address each point below.


W1 & Q1: The main text does not adequately explain several parts of the model, so the reader must constantly refer to the appendix for notation (e.g., Eq. 1 and the predictor). Could you add more explanation and notation to the Methods section to improve readability?

A1: Thank you for pointing out where the Methods section needed clearer self-contained notation. We have revised Sections 3.2–3.4 and reorganized Table 7 (Appendix A) so readers can follow the full pipeline directly in the main text:

  • Section 3.2 explicitly defines the historical index tt, future step kk, pyramid level ss, modality-specific encoders, fused latent state, and EMA target.
  • Section 3.3 adds Eqs. (2)–(3) to formally define the concatenated history/future-query sequence (qk(s)=m(s)+c(s)+lk(s)q_k^{(s)} = m^{(s)} + c^{(s)} + l_k^{(s)}) and the spatiotemporal tube masking operator before introducing scale-specific attention and cross-scale fusion.
  • Section 3.4 explicitly defines the initial-state residual, subsequent transition residuals, and validity weights in the loss.

W2: Figures 4 and 9 use red boxes to highlight regions, but they do not provide zoomed-in views. This makes it very difficult to assess the qualitative comparison at the given resolution.

A2: Thank you for the helpful suggestion. We have updated Figure 4 with zoomed-in views of the red-boxed regions so the localized rainband structure can be compared directly. Figure 9 has been updated in the same way in the revised manuscript.


W3: The 17.8% increase in accuracy with 12.8% of computing power pairs fine-tuned accuracy with the cost of pretraining. Including 108 hours of fine-tuning brings the total to approximately 59% of ClimaX computing power, and the gain of the frozen model is about 9.9%.

A3: Thank you for catching this mismatch. We agree that accuracy must be paired with the complete training cost of the exact same configuration. We have updated the headline in the Abstract, Introduction, Figure 1, Table 6, and Table 12 to use frozen transfer, pairing its accuracy with its complete pretraining-plus-adaptation cost:

Table R1-1: WeatherBench2 accuracy and complete training cost (GPU-h).

MethodPretrainingAdaptationTotal24-h U850 ACC ↑
ClimaX232.326.6258.90.754
E-JEPA-50M (frozen)17.317.034.30.8933
E-JEPA (113M, frozen)29.729.359.00.9119
E-JEPA-239M (frozen)43.740.283.90.9206

Using 59.0 GPU-hours in total (29.7 for pretraining + 29.3 for adaptation), frozen E-JEPA improves 24-h U850 ACC by about 21% over ClimaX (0.9119 vs. 0.754) and improves the four-variable mean 24-h ACC (U850, T850, Q700, MSLP) by about 17% (0.9192 vs. 0.7855), while using only about 23% of ClimaX's total cost (59.0 vs. 258.9 GPU-hours).

AuthorsOpenReview ↗

Response to Reviewer AvYT (Part 2 / 2)

Additional Clarification

However, it remains well behind native-resolution supervised models.

Thank you for placing our results in this broader context. We candidly acknowledge that absolute accuracy against native-resolution global models like Pangu-Weather and GraphCast is not our primary claim. Rather than avoiding this comparison, we address it directly in Figure 1, Table 3 and Section 5:

  1. Challenging transfer setting & compute constraints: Our experiments were conducted on 8x NVIDIA A800 GPUs with a 3.40 TB global dataset archive. Under these constraints, Earth-JEPA tests whether a backbone pretrained on 15-minute regional satellite and atmospheric data can transfer to a global 6-hourly grid without satellite inputs at inference. After hardware normalization (Section 4.2, Appendix D.2), the training budgets of Pangu-Weather and GraphCast correspond to roughly 961x and 321x the total cost of frozen E-JEPA (59.0 GPU-hours).
  2. A starting point for reusable Earth-system models: We view Earth-JEPA as a practical starting point for resource-efficient, multimodal representation learning. We are also collaborating with operational partners on pilot deployments with real-time satellite streams, and scaling to larger archives and native global grids is an exciting direction for future work.

Summary of Core Advantages

To summarize our revised evidence across efficiency, transferability, accuracy, and preliminary scalability:

Table R1-2: Summary of core advantages and supporting evidence.

Core AdvantageKey ResultLocation
EfficiencyWB2 cost: 29.7 + 29.3 GPU-h (E-JEPA pretrain + frozen adaptation), 0.10% of Pangu-Weather’s A800-equivalent costFig. 1, Table 6
TransferabilitySEVIR performance: beats previous supervised SOTA RainPro-2R (ICLR 2026) on 5/6 metrics, with +12.0% CSI-p16Table 4, Sec. 4.3
AccuracyWB2 performance vs. ClimaX: +47% mean ACC, 39% lower mean MAE, 38% lower mean RMSETables 3, 13, 14
Scalability (preliminary)WB2 72-h mean ACC: 0.678 → 0.725 → 0.749 (50M → 113M → 239M, frozen E-JEPA)Table 3, Fig. 5

We have uploaded the anonymous supplementary material with our model definitions and evaluation code, and will release the full training code, checkpoints, and EarthCastBench upon acceptance. We believe our response and revised manuscript have addressed your concerns, and we respectfully ask if you would consider raising your assessment in light of our revisions and additional evidence.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper!

Comments on the paper

3 comments

Dear Reviewers,

As the discussion period is coming to an end, I kindly ask all reviewers who have not yet responded to the authors' rebuttal to carefully review their responses and provide feedback or update their reviews accordingly.

Thank you for your time and efforts!

AuthorsOpenReview ↗

Thank You and a Summary of Revisions

Dear Area Chair,

Thank you for taking the time to follow up with the reviewers as the discussion draws to a close. We appreciate your help.

In the revised manuscript, we have made three main updates:

  • Additional experiments. We added same-architecture supervised controls on WeatherBench2 and SEVIR, and evaluated how the State-Transition Loss and EMA targets affect downstream forecasting.
  • Clearer compute comparisons. We paired frozen-transfer accuracy with the full pretraining and adaptation cost and updated the WeatherGFM evaluation.
  • Clearer presentation. We clarified the method and transfer protocols and added zoomed-in views to the qualitative comparisons.

Our main contribution is a JEPA that predicts future latent states at multiple spatial resolutions, providing an efficient, accurate, transferable, and scalable approach to Earth-system forecasting. The revised experiments show how the same frozen model can be adapted across tasks, sensors, and forecast intervals.

We look forward to discussing the revision with the reviewers and would be happy to clarify any remaining points. Thank you again for facilitating the discussion.

Best regards,
The authors of Submission #61459

AuthorsOpenReview ↗

Final Response

Dear Area Chair,

We sincerely thank the Area Chair for following up with the reviewers, and all the reviewers for their constructive feedback. We appreciate that Review n4Hc found that the revised comparisons and ablations addressed their main concerns and raised their rating. We also appreciate that Review YqLJ found the new experiments helpful and clarified their remaining concerns about novelty and the cost of training.


Core Contribution of Our Work

Earth-JEPA uses a simple, elegant pretraining design for efficient and transferable Earth-system modeling. It predicts future latent states at multiple spatial resolutions to capture both large-scale weather patterns and local detail. The State-Transition Loss uses slowly updated EMA representations to define stable learning targets across the pyramid. The same pretrained model can be reused across forecasting tasks, sensors, and forecast intervals.


Updates of Experimental Results During Rebuttal

  1. Complete training cost. We clarified the cost comparison throughout the manuscript. Figure 1 and Tables 6 and 12 now pair accuracy with the full pretraining and adaptation cost and distinguish measured GPU-hours from hardware-normalized estimates. Frozen E-JEPA achieves 17% higher mean 24-h ACC than ClimaX using about 23% of its total training cost.

  2. Same-architecture supervised controls. We added these controls on WeatherBench2 and SEVIR. Frozen E-JEPA remains competitive with training from scratch on WeatherBench2 while allowing the same pretrained model to be reused across tasks. On SEVIR, frozen E-JEPA also improves CSI-p16 and LPIPS over the same-architecture supervised control.

  3. Downstream ablations. The initial submission included lightweight downstream probes under a limited compute budget. During rebuttal, we added downstream WeatherBench2 evaluations. Removing the State-Transition Loss or EMA targets raises mean nRMSE by about 12% or 44%, respectively (Table R2-1). Together with the effective-rank analysis, the new results show how stable learning targets and transition supervision support downstream forecasting and help prevent coarse-scale representation collapse without auxiliary losses or explicit covariance regularizers.

  4. Potential for operational use. In response to Reviewer n4Hc, we added the Drop-ERA5 comparison in Section 5 and Appendix B (Table 10). With all ERA5 inputs withheld at inference and non-ERA5 task history retained, Earth-JEPA stays close to its full-input performance and achieves lower precipitation and pressure RMSE than the full-input baselines in the table. This supports its potential for nowcasting when reanalysis is delayed or unavailable.


Additional Analysis and Presentation

We refined the presentation throughout the manuscript, expanded the theoretical analysis and discussed the potential for real-time operation in the new Discussion and Future Work section (Section 5), and included zoomed-in views in Figures 4 and 9.


Core Advantages

Core AdvantageKey ResultLocation
EfficiencyWB2 cost: 29.7 + 29.3 GPU-h (E-JEPA pretrain + frozen adaptation), 0.10% of Pangu-Weather’s A800-equivalent costFig. 1, Table 6
TransferabilitySEVIR performance: beats previous supervised SOTA RainPro-2R (ICLR 2026) on 5/6 metrics, with +12.0% CSI-p16Table 4, Sec. 4.3
AccuracyWB2 performance vs. ClimaX: +47% mean ACC, 39% lower mean MAE, 38% lower mean RMSETables 3, 13, 14
Scalability (preliminary)WB2 72-h mean ACC: 0.678 → 0.725 → 0.749 (50M → 113M → 239M, frozen E-JEPA)Table 3, Fig. 5

We value the discussion and remain happy to clarify any remaining questions. We do believe that multi-resolution latent pretraining is a unique and promising idea that could change how the community approaches Earth-system modeling. Earth-JEPA provides an efficient, transferable model that researchers can reuse, adapt, and build on as a starting point for the next generation of Earth-system foundation models.

We would be grateful if you would weigh the originality and longer-term potential of this approach alongside the revised evidence in your assessment. Thank you again for your time and for helping move the discussion forward.

Best regards,
The authors of Submission #61459

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.