acceptodds
Under review as a conference paper at ICLR 2027

JEPACast: Precipitation Nowcasting in a Decodable and Predictable Latent Space

10 views

Abstract

Generative models have become a dominant approach to probabilistic radar-based precipitation nowcasting, yet the latent space where they generate has barely been examined. Most models rely on reconstruction-trained autoencoders, while advances focus on increasingly specialized generators without changing the representation. A simple decomposition of forecast error motivates two fundamental requirements for a forecasting latent space: *decodability* and *predictability*. We find that representations from a V-JEPA encoder trained on radar data remain strongly decodable while achieving higher predictability of future dynamics than widely used reconstruction-based representations. Building on this, we introduce **JEPACast**, which adapts the representation autoencoder paradigm to generative nowcasting by combining the V-JEPA encoder with a latent flow transformer and a ViT decoder. On SEVIR and MeteoNet, JEPACast outperforms various baselines on key forecast metrics, with larger gains for intense precipitation and longer lead times. These results highlight latent representation design as an important direction for generative precipitation nowcasting.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Precipitation Nowcasting, Joint-Embedding Predictive Architecture, Generative Modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JeuaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents JEPACast, a generative precipitation-nowcasting framework which combines a V-JEPA-style encoder that has been trained from scratch on radar sequences, a latent flow-matching Transformer, and a ViT decoder that was trained independently. It uses two complementary criteria to characterise the forecasting representations: decodability, defined as the capacity to faithfully reconstruct radar observations, and predictability, which is assessed by how effectively lightweight probes can extract future-relevant information from frozen past representations. The key contributions are a representation-centric diagnostic framework for nowcasting latents and a specific model that carries out generative forecasting in a representation space that has been learned for predictability. The evaluation involves round-trip reconstruction diagnostics; the use of linear and MLP probes to identify future motion, high-intensity precipitation, and source–sink dynamics; controls regarding latent dimensionality; architectural ablations; and benchmark comparisons on SEVIR and MeteoNet. The results show that future motion is considerably more accessible to the probes examined—especially the linear readouts—and that JEPACast generally leads to an improvement in threshold-based forecast skill, with more evident gains in the case of intense precipitation and at longer lead times. Nevertheless, the method does not outperform every metric or evaluation scale, and its SEVIR CRPS is still worse than that of FlowCast.

Critical strengths
  1. The study goes beyond comparing end-to-end forecast predictions by looking separately at reconstruction fidelity and the accessibility of future-relevant information in the frozen representations. Evidence is provided by means of round-trip reconstruction diagnostics, linear and MLP probes, controls involving latent dimensionality, and lead-time analyses. The most convincing of these are the results from the linear motion probes, which indicate that, with the proposed probe protocol, the JEPA representation makes future motion considerably more accessible while still maintaining competitive—although not always the best—reconstruction quality. Furthermore, the equal-width analysis implies that latent dimensionality by itself is not sufficient to account for this advantage in motion, although the corresponding advantages regarding intensity and source–sink dynamics diminish under strong compression.

  2. JEPACast shows excellent threshold-based forecasting results on both SEVIR and MeteoNet, especially when it comes to predicting intense precipitation and at longer lead times for SEVIR. The benchmark results, the component ablations, and the qualitative comparisons all support the idea that the entire pipeline is effective. The fact that these downstream improvements are in line with the representation-level findings offers encouraging evidence that predictive representation learning is a useful approach for generative nowcasting.

Critical weaknesses
  1. The JEPA training objective's contribution is not entirely separate from other factors. As shown in Table 4, the latent-representation ablation affects the representation-learning objective, the encoder architecture, the native latent dimensionality, and the decoder at the same time. Even though the different variants in this ablation employ a matched but shorter forecaster-training schedule, it is still not clear whether the observed advantage would continue when full training is carried out or at convergence. The equal-width VideoVAE and PCA analyses do provide some useful partial controls, but these are mainly carried out at the diagnostic-probe level and not through matched end-to-end forecasting experiments. A comparison in which the encoder architecture, the latent size, the decoder capacity and the training procedure, the forecaster configuration, and the optimization budget are kept constant while only the representation-learning objective is varied would reinforce the claim that the improvements in the downstream task are due to predictive representation learning rather than to the wider JEPA-based representation pipeline.
  2. The connection between diagnostic predictability and downstream forecasting performance is still indirect. Instead of directly assessing the difficulty of modelling the full conditional distribution of the future latents, the probes look at how easily constrained readouts are able to obtain the selected future quantities—specifically, motion, intensity, and source–sink dynamics. Following the equal-width compression shown in Table 5, the JEPA representation still has a clear advantage when it comes to motion, but its advantages in terms of intensity and source–sink either decrease or reverse compared to some of the representations learned through reconstruction. The evidence thus provides stronger support for improved accessibility of future motion than for a general improvement in the predictability of the future. In order to more convincingly show that the diagnostic is an indicator of downstream nowcasting skill, the paper should set up a systematic relationship between the probe scores and the corresponding end-to-end forecasting performance—for instance, across different representation-learning objectives, latent widths, and forecaster capacities.
Questions for authors
  1. Beyond the reported controls, is there existing end-to-end evidence comparing predictive and reconstruction-based pretraining under matched encoder architecture, latent dimensionality, decoder architecture and training, and forecaster settings? Both variants in Table 4 use the same shortened training schedule, preserving within-group fairness; are full-schedule results available to determine whether the advantage persists? These results would affect how strongly I attribute the downstream gains to the JEPA objective rather than the overall representation pipeline.

  2. Do existing results show whether representations with stronger probe scores also achieve better downstream forecasting performance under matched forecaster settings? In particular, do the motion advantages retained at equal latent width correspond to forecasting gains despite the weaker intensity and source–sink results? This would affect my assessment of the diagnostic’s utility for evaluating forecasting representations beyond the specific JEPACast configuration.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Jeua

We thank Reviewer Jeua for the careful and constructive review. We are encouraged that the reviewer found (i) the representation-level analysis valuable, (ii) the linear motion probes convincing, and (iii) the downstream gains consistent with the representation-level findings.

Following the reviewer's suggestions, we added a matched end-to-end comparison of latents (Table C1) and full-schedule learning curves (Table C2). The end-to-end experiment with equal latent width (Q2) is in progress, and we will report the results here as soon as it is complete.

[W1, Q1] Isolating the pre-training objective, and the full schedule

As the reviewer requested, Table C1 compares latents end-to-end with the same forecaster (20k steps) and sampler. V-JEPA 2.1, temporal masking, and VideoMAE additionally share our encoder architecture and size, training data, pre-training budget, and decoder recipe, so that for them only the pre-training recipe changes, and VideoVAE (128-ch) matches our latent size exactly with a reconstruction objective. With these factors fixed, masked latent prediction forecasts better than reconstruction. JEPACast reaches a CSI-M of 0.4646, compared with 0.4475 for masked pixel reconstruction (VideoMAE) and 0.4302 for autoencoding at the same latent size (VideoVAE (128-ch)), and it also improves on the original V-JEPA 2.1 recipe and on the causal temporal-masking variant.

Table C1. Latent comparison on SEVIR. Probes report R² (linear / MLP) as in Table 1.

LatentMotion ↑Intensity ↑Source–sink ↑CSI-M ↑CSI-219 ↑CRPS ↓
JEPACast+0.757 / +0.806+0.299 / +0.472+0.250 / +0.4330.46460.19960.0199
V-JEPA 2.1 (original recipe)+0.721 / +0.768+0.277 / +0.446+0.249 / +0.4140.45580.19040.0196
Temporal masking+0.738 / +0.781+0.311 / +0.471+0.322 / +0.4380.43870.16360.0208
VideoMAE+0.677 / +0.748+0.314 / +0.406+0.235 / +0.3600.44750.18270.0215
VideoVAE (128-ch)+0.263 / +0.590+0.299 / +0.391+0.204 / +0.3460.43020.18020.0256

Full schedule (Q1). As the reviewer asked, we continued training to the full 45k steps (Table C2). The advantage of the JEPA latent remains over the whole schedule, and it leads at every evaluated step. On the JEPA latent, the forecaster is still improving at 45k (0.4715). With the FlowCast VAE latent, which is used in the latent ablation of Table 4 in our paper, the same forecaster peaks at 15k (0.4551, close to FlowCast's own model) and then overfits, and on VideoVAE (128-ch) it saturates at 0.4401, a level that the JEPA latent reaches at 5k steps.

Table C2. Test CSI-M of the same forecaster during training.

StepJEPACastFlowCast VAEVideoVAE (128-ch)
5k0.44060.43000.3707
10k0.45780.44530.4203
15k0.46090.45510.4302
20k0.46460.45480.4302
30k0.46880.42990.4377
40k0.47080.40290.4401
45k0.47150.39330.4401

Revision. We will add the matched comparison and the learning curves to the revised paper.

[W2, Q2] Probe scores and forecasting performance

Thank you for this important question. With the same forecaster, probe scores and forecasting performance follow the same order for four of the five latents at the native width of 512. Among the latents of Table C1, JEPACast, V-JEPA 2.1, VideoMAE and VideoVAE (128-ch) are ranked in the same order by the MLP probes of all three targets, by the linear motion probe (R² +0.757, +0.721, +0.677 and +0.263) and by CSI-M (0.4646, 0.4558, 0.4475 and 0.4302). The one exception is the causal temporal-masking variant, whose probes are comparable to ours but whose forecasts are lower. It is the only encoder trained directly to predict the future from the past frames, which is what the probes test, so its probe scores follow from its objective rather than from the overall quality of the latent. Its latent is less decodable (round-trip CSI-M 0.854 vs. 0.877), and the gap is already present at the first lead (+5 min, 0.660 vs. 0.694), where little has to be predicted. This is why our diagnostic measures both predictability and decodability.

What the probes measure. The direct measure of how hard the future latents are to model is the forecaster's own training loss, but each latent has its own dimensionality and scale, so this loss cannot be compared across latents. The probes instead predict the same physical quantities from every latent, which makes them comparable. They can thus serve as an early, complementary indicator of how well a forecaster will learn on a latent, which the learning curves then show directly (Table C2).

Revision. We will add the comparison of probe scores with forecasting performance to the revised paper and narrow the statement at L72 to future motion.

AuthorsOpenReview ↗

Follow-up to Reviewer Jeua: equal latent width

As promised in our first response, we report the end-to-end experiment at equal latent width (Q2).

[W2, Q2] Equal latent width

We trained our encoder at a latent width of 16, the width of the FlowCast VAE latent, and paired it with the same forecaster (20k steps) as in Table C1. At equal width of 16, the motion advantage of the JEPA latent carries over to forecasting (Table C3). The JEPA latent has the higher motion probe and the higher CSI-M. Compared with our 512-wide latent, reducing the width to 16 lowers CSI-M by 0.0055.

Table C3. End-to-end comparison at equal latent width on SEVIR, with JEPACast (width 512) for reference. Width is the number of channels per 16 km cell, and the motion probe reports R² (linear / MLP) from Tables 1 and 5 of our paper.

LatentWidthMotion probe ↑CSI-M ↑
JEPACast512+0.757 / +0.8060.4646
V-JEPA (16-dim)16+0.231 / +0.6660.4591
FlowCast VAE16+0.183 / +0.6190.4548

Revision. We will add the equal-width comparison at width 16 to the revised paper, together with the comparison at width 512 (Table C1).


[Update] We have uploaded the revised paper, which includes all the revisions promised in our responses, with the changes marked in blue. We also corrected a typo in Table C2. We would greatly appreciate it if you could take a look at the revised paper and our responses.

AuthorsOpenReview ↗

Gentle reminder

Dear Reviewer Jeua,

Thank you again for the time and care you have put into reviewing our paper. We understand that this is a very busy period, and we truly appreciate your effort.

We would like to gently follow up on our responses and the revised paper. As the discussion period will close soon, we would be grateful if you could let us know before then whether they address your concerns, and we would be glad to answer any further questions in the remaining time.

Reviewer_o7GuOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks which properties actually make latent spaces suitable for forecasting and propose to measure two quantities: decodability and predictability. The authors propose to use a V-JEPA encoder on radar data, as it was found to model physical dynamics and motion well. Their method, JEPACast, then combines this encoder with a normal deterministic decoder and a latent space flow model. The main claim follows the notion of current image and video generation works, which find that representation encoders actually yield better latent representations for downstream generative modeling, and this work successfully applies this to precipitation nowcasting.

Critical strengths

Overall, the paper is clearly written and follows a very nice common thread. Most importantly, it asks a very interesting and valuable question about what the properties that make a latent space suitable for forecasting. It nicely formalizes this in the predictability and decodability properties, and analyses it using the probinbg diagnostic protocol. The diagnostic analysis is carefully controlled, e.g. with width-matched VideoVAEs and equal-width projections to separate the training objective from latent capacity, and the protocol could be reused to evaluate other nowcasting representations. Furthermore, the method achieves very strong results with a simple pipeline and thoroughly ablates individual design choices.

Critical weaknesses

W1: Is predictability really neccessary for latent forecasting? While I see that predictability is a nice property, I am asking myself whether it is really required for latent space models? The main motvation from latent space models is efficiency and abstracting away imperceptible details (in image generation). Hence, if we model a latent space without predictability, then forecasting can still be accomplished with the latent space flow model. Hence, what is then the benefit of this analysis or in general the method? Does JEPA bring any efficiency gains, because it shifts heavy-lifting of forecasting away from the flow model into the encoder-decoder? In particular, for V-JEPA the goal in the first place was never to decode back to data space, but use the latent space, e.g. for planning. The decoder was merely an inspection tool. However, for radar precipitation we actually care more about the final radar maps. So does the latent space really need predictability? Or is it fine to use a decoder like e.g. FlowCast or FREUD, which leaves forecasting completely to the latent space model. To be specific, is the decomposition actually informative or neccessary?

W2: "Predictive" motivation of the encoder only partially applies The main motivation of JEPAs training paradigm is that in natural images or video the model can infer or predict the missing regions from the context. However, since radar maps often have very localized intensity peaks, this motivation might only partially hold. More importantly, the masking blocks span all 25 frames. Hence, the encoder is therefore never trained to predict future frames from past ones, and rather learns inpainting of spatial tubelets. For me it is unclear, how this objective specifically encourages representations to encode motion.

W3: Predictability seems partially tied to latent capacity The paper states that other approaches “expose substantially less future-relevant structure” (L72). This clearly holds for motion, but at equal width (d=16, table 5) with MLP the advantages on intensity and source-sink mostly vanish, and JEPACast even falls behind the VideoVAE. This indicates that the gain might at least partly come from the increased latent capacity rather than the specific training paradigm. While the 128-channel VideoVAE control suggests that the probing advantage at native width is not purely due to capacity, the forecasting comparison is not width-controlled, as table 4 only compares against the 16-dimensional FlowCast VAE. Hence, the “expose substantially less future-relevant structure” might be a bit overclaimed. This is particularly relevant as the encoder-decoder architecture seems to be overcomplete. The architecture does not include temporal downsampling (L1127), so each frame has 24x24 tokens with dimensionality 512 (~294k) vs the input only being 384x384 (hence ~147k values). Furthermore, despite all integrated advancements in architecture (e.g. DDT head), the method falls short in CRPS, indicating that while the latent space is overcomplete, the model might still lack sufficient information to model the full forecasting distribution.

Questions for authors
  • regarding W1: does the latent flow model learn faster, because it has better predictability in the latent space? Reporting forecast metrics along training (e.g., CSI-M over training steps for the JEPA latent vs. the FlowCast VAE from table 4, and ideally the 128-channel VideoVAE) would help isolate whether predictability really is a favorable property for the forecaster, independently of the final performance.
  • I was wondering why results are not blurry, given that the decoder is trained with an L1 loss only. Is this because the latent space is overcomplete? Does decoding become blurrier when the latent is compressed via PCA to the width of the baselines (figure 7 indicates this)? Are the qualitative forecasts in figure 4 single ensemble members or the ensemble mean?
  • Given that the latent space is overcomplete, a natural baseline would be to train the flow model directly on concatenated (patchified) pixel values or a simple overcomplete autoencoder. This would further justify the training of the two stages (JEPA-encoder / decoder), probably because it will show that it naively fails and we need the encoder to be trained with the JEPA representation loss.
  • The inference hyperparameters seem to be selected on test data: the sweep over Euler steps and guidance weights in Appendix H uses the first 2k test windows. In addition, the chosen settings differ considerably between datasets. Were these settings selected on a validation split? If not, how sensitive are the results to these choices when they are selected on held-out data?
  • Would it be possible to also report calibration metrics for the model? For example, rank histograms or reliability index. The reported CRPS indicates calibration, but does not fully reflect it. Furthermore, both CFG and the deterministic decoder could shrink the ensemble spread, so it would be nice to see whether the decoded ensemble is under-dispersive.
6 comments
AuthorsOpenReview ↗

Response to Reviewer o7Gu (1/3)

[Update] We have uploaded the revised paper, which includes all the revisions promised in our responses, with the changes marked in blue. We also corrected a typo in Table B2. We would greatly appreciate it if you could take a look at the revised paper and our responses.


We thank Reviewer o7Gu for the thoughtful and constructive review. We are encouraged that the reviewer found (i) the central question valuable, (ii) the diagnostic carefully controlled, and (iii) the results strong for a simple pipeline.

Following the reviewer's suggestions, we added a matched comparison of latents (Table B1), learning curves (Table B2), a width-controlled comparison with a pixel baseline (Table B3), a CRPS comparison of latents (Table B4) and calibration diagnostics (Table B5).

[W1, Q1] Is predictability necessary, and does the forecaster learn faster?

Thank you for this fundamental question. Although a forecasting model can be built on a decodable latent, as FlowCast and FREUD show, a latent that is both decodable and predictable forecasts better. Higher predictability can give higher forecasting performance and faster learning. As mentioned in L68–69, the latent determines how much of the temporal structure the forecaster has to learn by itself. To test this fairly, we fixed the forecaster, its training budget and the sampler, and changed only the latent.

Higher forecasting performance (Table B1). The JEPA latent gives the highest CSI-M and CSI-219 of the five latents, which all share the same ViT decoder recipe. Forecasting performance follows the order of the motion probes, with one exception (causal temporal masking) that we discuss in [W2].

Table B1. Latents compared with the same forecaster (20k steps) and sampler on SEVIR. Probes report R² (linear / MLP) as in Table 1.

LatentMotion ↑Intensity ↑Source–sink ↑CSI-M ↑CSI-219 ↑CRPS ↓
JEPACast+0.757 / +0.806+0.299 / +0.472+0.250 / +0.4330.46460.19960.0199
V-JEPA 2.1 (original recipe)+0.721 / +0.768+0.277 / +0.446+0.249 / +0.4140.45580.19040.0196
Temporal masking+0.738 / +0.781+0.311 / +0.471+0.322 / +0.4380.43870.16360.0208
VideoMAE+0.677 / +0.748+0.314 / +0.406+0.235 / +0.3600.44750.18270.0215
VideoVAE (128-ch)+0.263 / +0.590+0.299 / +0.391+0.204 / +0.3460.43020.18020.0256

Faster learning (Q1, Table B2). As the reviewer suggested, we report CSI-M during training up to the full 45k steps. The forecaster leads on the JEPA latent at every evaluated step. At 5k steps, it already matches VideoVAE (128-ch) at 45k (0.4406 vs. 0.4401), and at 10k it exceeds the best step of the FlowCast VAE latent (0.4578 vs. 0.4551 at 15k), so the JEPA latent reaches the same performance with far less forecaster training. On the JEPA latent, the forecaster also keeps improving up to 45k, whereas it saturates on VideoVAE (128-ch) at 40k and overfits on the FlowCast VAE latent after 15k.

Table B2. Test CSI-M of the same forecaster during training.

StepJEPACastFlowCast VAEVideoVAE (128-ch)
5k0.44060.43000.3707
10k0.45780.44530.4203
15k0.46090.45510.4302
20k0.46460.45480.4302
30k0.46880.42990.4377
40k0.47080.40290.4401
45k0.47150.39330.4401

Decoding back to radar. As the reviewer notes, nowcasting must return radar maps, and this is exactly why decodability is our second criterion. The frozen JEPA latent with a separately trained ViT decoder reconstructs on par with the VAE latents (round-trip CSI-M 0.877 vs. 0.876 for FlowCast and 0.881 for FREUD, Table 1).

Revision. We will make this argument explicit in the revised paper and add the comparisons above.

[W2] In what sense the encoder is predictive

We thank the reviewer for this important question. We answer its two parts in turn: the masks that span all frames and the localized intensity peaks.

Masks that span all frames. Every masked block spans all 25 frames, as in V-JEPA 2.1, so pre-training is not causal, but the objective is predictive in time as well as in space. Since a masked region is not visible in any frame, its content cannot simply be copied from a neighboring frame. To predict it, the encoder has to infer from the visible surroundings how precipitation moves through the region over time, so the objective rewards representations that track the motion of the field. The probes confirm this, since a linear readout of the context frames alone predicts future motion with R² +0.757, compared with +0.263 for VideoVAE (128-ch), which is trained by reconstruction (Table B1).

AuthorsOpenReview ↗

Response to Reviewer o7Gu (2/3)

[W2] (continued)

We also trained the encoder with causal temporal masking alone. Its probes are comparable to ours, but its forecasts are lower (CSI-M 0.4387 vs. 0.4646, Table B1). This encoder is trained directly to predict the future from the past frames, which is what the probes test, so its high probe scores follow from its objective. What this objective does not reward is how faithfully each frame itself is encoded. Its latent is less decodable than ours (round-trip CSI-M 0.854 vs. 0.877), and the forecast gap is already present at the first lead (+5 min, 0.660 vs. 0.694), where the field has hardly changed and little has to be predicted. Based on this analysis, combining temporal masking with tube masking is a promising direction, in line with the radar-specific masking schemes we list as future work (Appendix A).

Localized intensity peaks. As the reviewer notes, masked prediction alone rarely teaches the value of an isolated peak, because the surrounding context usually does not reveal it. However, the peaks can be learned through the visible-context loss of V-JEPA 2.1, where the predictor must reproduce the EMA targets of the visible tokens, including features from shallow encoder blocks that stay close to the local content of each token. In our training statistics, heavy rain occupies only 0.85% of the tokens, yet a heavy-rain token is visible in about 20% of training steps. As a result, the JEPA latent decodes heavy rain almost as well as the FlowCast VAE (round-trip CSI-219 0.781 vs. 0.794, Table 1) and gives the highest CSI-219 in the forecasts (0.1996, Table B1).

Revision. We will describe the pre-training objective and the role of the visible-context loss more precisely in the revised paper.

[W3, Q3] Latent capacity and over-completeness

At the same latent size, the JEPA latent forecasts better than a reconstruction latent, so over-completeness does not explain the gain. Table B3 compares JEPACast with a reconstruction latent of the same size, together with the pixel baseline of Q3 (the same forecaster trained directly on 16 × 16 pixel patches).

Table B3. Comparison with a reconstruction latent of the same size and a pixel baseline on SEVIR. Forecasters are equally trained for 20k steps.

LatentCSI-M ↑CSI-219 ↑CRPS ↓
JEPACast0.46460.19960.0199
VideoVAE (128-ch)0.43020.18020.0256
Pixel patches0.42380.18040.0252

VideoVAE (128-ch), which corresponds to the over-complete autoencoder of Q3, has exactly the latent size of JEPACast but forecasts lower, both at 20k steps and with the full schedule (0.4401 vs. 0.4715, Table B2), and the pixel baseline is lower still.

We will narrow the statement at L72 to future motion, where the advantage of the JEPA latent holds at every width. For intensity and source–sink, the JEPA latent matches or exceeds the reconstruction latents at native width (Table 1), and Table 5 shows that this information is spread over more latent dimensions than motion. Table 5 projects every wider representation to 16 principal components, which keep 96% of the variance of VideoVAE (128-ch) but only 38% of that of JEPACast (Table 6), so it measures how much survives strong compression rather than predictability at equal native width.

The CRPS gap does not indicate missing information in the latent. With the forecaster and sampler fixed, the JEPA latent has a lower CRPS than both reconstruction latents (Table B4), and the gap at our default setting comes from the sampler, as discussed in [Q5].

Table B4. Latents compared with the same forecaster (20k steps) and sampler on SEVIR.

LatentCSI-M ↑CRPS ↓
JEPACast0.46460.0199
FlowCast VAE0.45480.0220
VideoVAE (128-ch)0.43020.0256

Revision. We will narrow the claim at L72, clarify what Table 5 measures, and add the width-controlled comparison and the pixel baseline to the revised paper.

[Q2] Sharpness of the L1 decoder and Figure 4

An L1 decoder returns the conditional median of the frame given its latent, so it blurs only when the latent leaves the frame uncertain. In JEPACast, the uncertainty about the future is handled by the flow forecaster, which samples one future latent, and the decoder only has to turn that latent back into a radar frame. Each latent determines its frame almost exactly (round-trip CSI-219 0.781 and SSIM 0.981 in Table 1), so the decoded frames stay sharp as long as the forecaster generates accurate latents. As the reviewer suggests, the wide latent contributes to this, since compressing it to 16 principal components does blur the decoded fields (Figure 7). The qualitative forecasts in Figure 4 show a single ensemble member (member 0), and we will clarify this in the caption of the figure.

AuthorsOpenReview ↗

Response to Reviewer o7Gu (3/3)

[Q4] Sampler selection

The samplers for both datasets were selected on validation data, not on the test set. Appendix H is a separate analysis, run on the first 2,048 test windows, that only shows how the forecasts change with the sampling hyperparameters. On SEVIR, we searched Euler steps {1, 2, 3, 4, 6, 10} × guidance {none, w = 1.5, 2.0, 2.5, 3.0, ramp 1.25→2.0} at α = 9 (36 settings), and α ∈ {6, 9, 13} at two steps with the ramp (38 distinct settings in total). On MeteoNet, we searched Euler steps {1, 2, 4} × guidance {none, w = 1.5, 2, 3, ramps 1→3, 1→6, 1→8} × α ∈ {1, 2, 3, 5} (63 distinct settings, since α has no effect at one step). In each grid, we chose the setting that best balances the CSI metrics, and the choice is not sensitive, since the three best SEVIR settings lie within 0.003 CSI-M of each other.

The settings differ because the suitable guidance scale depends on the dataset. SEVIR and MeteoNet have different precipitation characteristics, and they are also scored differently, with a single sample under the DiffCast protocol on MeteoNet and the ensemble mean on SEVIR.

Revision. We will replace the figures in Appendix H with their validation versions. We will also state the validation splits and the sampler selection procedure precisely in the main text and the appendix.

[Q5] Calibration and under-dispersion

As the reviewer suggested, we computed the reliability index of the rank histogram and the spread–skill ratio under several sampler settings (Table B5).

Table B5. Sampler settings on SEVIR (10 members). JEPACast uses the guidance ramp unless marked "no CFG". Spread/RMSE is 1 for a calibrated ensemble, and RI is the reliability index of the rank histogram (lower is better).

ModelSamplerCSI-M ↑CSI-219 ↑CRPS ↓Spread/RMSERI ↓
JEPACast2 steps (default)0.47150.20710.01950.440.81
JEPACast3 steps0.46850.20040.01870.560.60
JEPACast4 steps0.46500.19340.01840.620.49
JEPACast4 steps, no CFG0.45140.16180.01770.740.34
FlowCastdefault0.45730.18060.01790.530.52
CasCastdefault0.44010.18510.02020.550.61

The ensembles are under-dispersive, and this holds for all three models. The spread–skill ratio, which is 1 for a calibrated ensemble, is 0.53 for FlowCast and 0.55 for CasCast, and ranges from 0.44 to 0.74 for JEPACast depending on the sampler. The sampler contributes to this through both the number of steps and the guidance. With the guidance on, going from 2 to 4 Euler steps raises the spread–skill ratio from 0.44 to 0.62, and removing the guidance at 4 steps raises it further to 0.74 (reliability index 0.81, 0.49 and 0.34). Fewer steps and guidance therefore trade calibration for CSI.

With calibration comparable to FlowCast, JEPACast still has a higher CSI. With 4 steps and the ramp, it is calibrated at least as well as FlowCast (reliability index 0.49 vs. 0.52, spread–skill 0.62 vs. 0.53) and has higher CSI-M (+0.0077) and CSI-219 (+0.0128), while FlowCast keeps a slightly lower CRPS (0.0179 vs. 0.0184). Without guidance, JEPACast has a lower CRPS than FlowCast (0.0177 vs. 0.0179) and better calibration (reliability index 0.34 vs. 0.52) but a lower CSI-M. Since the number of steps and the guidance weight are set at inference, the trade-off between CSI and calibration can be chosen per application without retraining.

Revision. We will report these sampler settings and calibration diagnostics in the revised paper and make it clear that the default favors CSI over calibration.

Reviewer_CJp6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks what makes a latent space suitable for precipitation nowcasting. It proposes two criteria: decodability and predictability. A diagnostic on SEVIR shows that reconstruction-trained latents are decodable but poorly predictable, while a V-JEPA-style radar encoder is both. The authors then build JEPACast: a frozen JEPA encoder, a flow-matching forecaster, and a ViT decoder. It improves threshold metrics on SEVIR and MeteoNet, especially for high-intensity and long-lead forecasts on SEVIR.

Critical strengths
  1. The decodability/predictability diagnostic is a useful contribution. It is better designed than CSI-only evaluation: multiple probes, three physical targets, and proper references against climatology and Lagrangian persistence.
  2. The SEVIR results are strong on threshold metrics. A frozen JEPA encoder can also be decoded well without a reconstruction objective, which is a genuinely interesting finding.
Critical weaknesses
  1. Encoder experiments are insufficient. The encoder is the paper's central claim, yet only one architecture and objective are tested, with no scaling, cross-dataset transfer, or comparison to V-JEPA 2.1, MAE/VideoMAE, DINO, or temporal masking. The JEPA recipe itself is not ablated. The only latent swap gives +0.0125 CSI-M, less than the wide head (+0.0257), and at equal width (Table 5) JEPACast is worse than VideoVAE-128 on intensity and source–sink. The claim that the latent is the main lever is not supported.
  2. The results are mixed. CRPS is 0.0195, worse than FlowCast (0.0179) and FREUD (0.0182); MeteoNet P16 CSI-M is 0.6971, below the DiffCast variants (0.7001–0.7200). CSI is computed on the ensemble mean, with no calibration analysis and no seeds or error bars.
  3. The delta over JEPA is limited. The method is V-JEPA 2.1 plus a visible-context loss, multi-layer targets, and a DDT/RAE forecaster/decoder; the core JEPA loss is unchanged. Pretraining is spatial masked inpainting with non-causal attention, not causal future prediction. No controlled comparison quantifies progress over V-JEPA 2.1, so the contribution is an application and diagnostic rather than a new JEPA method.
  4. Other concerns. Decodability is decoder-dependent; the comparison is not compute-matched; only two datasets are used; the CFG ramp appears to be selected on test windows and should be clarified.
Questions for authors
  1. Can you provide encoder ablations: standard V-JEPA 2.1 vs. your modifications, MAE/VideoMAE, a DINO-style encoder, and a temporal-masking variant? Which component drives the gain?
  2. Can you provide a capacity-matched comparison and a scaling study? How much of the gain survives when latent width and compute are controlled?
  3. Why is CRPS worse than FlowCast and FREUD despite better CSI? Please report per-lead CRPS, spread–skill, rank histograms, and member-wise vs. mean CSI.
  4. Why is MeteoNet P16 worse than DiffCast, and does the high-intensity/long-lead advantage hold on lighter rain?
  5. Were the CFG ramp and sampler selected on validation or test data? Please clarify and re-select on validation if needed.
  6. Please report multiple seeds/error bars and significance tests for the main results and ablations.
7 comments
AuthorsOpenReview ↗

Response to Reviewer CJp6 (1/3)

[Update] We have uploaded the revised paper, which includes all the revisions promised in our responses, with the changes marked in blue. We would greatly appreciate it if you could take a look at the revised paper and our responses.


We thank Reviewer CJp6 for the detailed and constructive review. We are encouraged that the reviewer found (i) the diagnostic a useful contribution, (ii) the decodability of a frozen JEPA encoder an interesting finding, and (iii) the SEVIR results strong on threshold metrics.

Following the reviewer's suggestions, we added encoder ablations (Table A1), a CRPS comparison of latents (Table A2), calibration diagnostics (Table A3), confidence intervals (Table A4), and a third dataset (Table A5). Seed repeats of the forecaster are still training, and we will post them in this thread when they finish.

[W1, Q1, Q2] Encoder ablations and latent width

As the reviewer suggested, we trained V-JEPA 2.1, VideoMAE, and a temporal-masking encoder with our encoder size, training data, and budget. For a fair comparison, every row of Table A1 uses the same decoder recipe, forecaster (20k steps), and sampler, so that only the encoder changes. JEPACast forecasts best among all encoder variants, and all four differences are significant.

Table A1. Encoder comparison on SEVIR at the same latent width (512 channels). Δ is the difference from JEPACast (paired bootstrap, 95% CI).

EncoderCSI-M ↑Δ CSI-M [95% CI]CSI-219 ↑CRPS ↓
JEPACast0.4646–0.19960.0199
V-JEPA 2.1 (original recipe)0.4558−0.0088 [−0.0098, −0.0079]0.19040.0196
VideoMAE (pixel reconstruction)0.4475−0.0171 [−0.0182, −0.0160]0.18270.0215
Temporal masking (predicts future frames)0.4387−0.0259 [−0.0269, −0.0249]0.16360.0208
VideoVAE (128-ch)0.4302−0.0344 [−0.0379, −0.0308]0.18020.0256

Where the gain comes from (Q1, Q2). Table A1 separates the components of the pre-training recipe, and two of them drive most of the gain (Q1). The first is the prediction target, since predicting the latents of the masked regions, as JEPACast does, is 0.0171 CSI-M better than reconstructing their pixels, as VideoMAE does. The second is the masking pattern, since masking space-time tubes that span all frames is 0.0259 CSI-M better than masking only the future frames, as in our temporal-masking variant. Our own changes to the V-JEPA 2.1 recipe, which we list in [W3], add a further 0.0088. The gain also survives when capacity is controlled (Q2). Every encoder in Table A1 has the same latent width as JEPACast and is paired with a forecaster of the same architecture and training budget, so none of these differences can come from a wider latent or a larger forecaster. The reconstruction autoencoder VideoVAE (128-ch), which has exactly this width, is the lowest of all (−0.0344 CSI-M), so width alone does not explain the gain.

Latent swap vs. wide head. The wide head and the latent are not competing levers. Following RAE, the denoiser has to be wider than the latent tokens, so removing the head handicaps generation in any wide latent, whereas the latent swap changes the latent with the forecaster fixed. The effect of the latent is of comparable size, with 0.0344 CSI-M against the same-width reconstruction latent at 20k steps (Table A1), compared with 0.0257 for the head at 10k steps (Table 4).

Revision. We will add these encoder comparisons with their confidence intervals to the revised paper.

[W2, Q3] CRPS and calibration

The CRPS gap comes from the sampler, not from the representation. With the same forecaster and sampler, the JEPA latent has a lower CRPS than both reconstruction latents (Table A2).

Table A2. Latents compared with the same forecaster (20k steps) and sampler on SEVIR.

LatentCSI-M ↑CRPS ↓
JEPACast0.46460.0199
FlowCast VAE0.45480.0220
VideoVAE (128-ch)0.43020.0256
AuthorsOpenReview ↗

Response to Reviewer CJp6 (2/3)

[W2, Q3] (continued)

Our default sampler uses two Euler steps with guidance, which favors the CSI of the ensemble mean but narrows the ensemble and therefore raises CRPS. As the reviewer requested, Table A3 reports calibration and member-wise CSI under several sampler settings.

Table A3. Sampler settings on SEVIR (10 members). JEPACast uses the guidance ramp unless marked "no CFG". Spread/RMSE is 1 for a calibrated ensemble, and RI is the reliability index of the rank histogram (lower is better).

ModelSamplerCSI-M ↑CSI-219 ↑CRPS ↓Spread/RMSERI ↓Member-wise CSI-M ↑
JEPACast2 steps (default)0.47150.20710.01950.440.810.4551
JEPACast3 steps0.46850.20040.01870.560.600.4438
JEPACast4 steps0.46500.19340.01840.620.490.4361
JEPACast4 steps, no CFG0.45140.16180.01770.740.340.4214
FlowCastdefault0.45730.18060.01790.530.520.4366
CasCastdefault0.44010.18510.02020.550.610.4177

At comparable calibration, JEPACast keeps its advantage in CSI. With 4 steps it is calibrated at least as well as FlowCast (reliability index 0.49 vs. 0.52, spread–skill 0.62 vs. 0.53) and still has higher CSI-M (+0.0077) and CSI-219 (+0.0128), while FlowCast keeps a slightly lower CRPS (0.0179 vs. 0.0184). Without guidance, JEPACast also has a lower CRPS than FlowCast (0.0177 vs. 0.0179) and better calibration, at the cost of a lower CSI-M. Since the number of steps and the guidance weight are set at inference, the trade-off between CSI and calibration can be chosen per application without retraining. As for FREUD, the 0.0182 reported in our Table 2 is the empirical CRPS from its paper, whereas all other entries use the Gaussian form. When we re-evaluate FREUD with the Gaussian form, its CRPS is 0.0187, so JEPACast with 4 steps is better than FREUD in both CSI-M and CRPS (0.4650 and 0.0184 vs. 0.3831 and 0.0187).

For the other requested diagnostics, all three models are under-dispersive (spread–skill ratio 0.53 for FlowCast, 0.55 for CasCast, and 0.44 to 0.74 for JEPACast depending on the sampler), and member-wise CSI-M preserves the ranking at the default samplers (0.4551 vs. 0.4366 for FlowCast). Per-lead CRPS of the default sampler rises from 0.0086 at +5 min to 0.0197 at +30 min and 0.0272 at +60 min.

Revision. We will add Table A3 and the calibration figures to the revised paper and correct FREUD's CRPS in Table 2 by using the same CRPS definition for all models.

[W2, Q4] MeteoNet P16 and lighter rain

On MeteoNet, the P16 gap comes from isolated light echoes at long lead times, while heavy rain is not lost. At 12 dBZ the forecast has more rain pixels than observed but 11% fewer rainy 16 × 16 blocks, and the block POD falls from 0.94 at +5 min to 0.72 at +100 min, so light rain is kept inside the main rain areas while isolated light echoes are dropped. At 32 dBZ, the highest threshold, most observed blocks are captured (block POD 0.77) and heavy rain is over-forecast rather than lost (block bias 1.15). The decoder is not the cause, since decoding the ground-truth latents gives a P16 of 0.948.

DiffCast reports only aggregate scores on MeteoNet and releases neither a MeteoNet checkpoint nor its diffusion training code, so a comparison by threshold and lead time is not possible. In aggregate, JEPACast trails only at P16 (0.6971 vs. 0.7001 to 0.7200) and leads every DiffCast variant at the pixel and 4 × 4 scales (CSI-M 0.4263 vs. at most 0.3512).

Revision. We will discuss the P16 gap in the revised paper and correct the MeteoNet frame interval in Appendix D to 5 minutes.

[W2, Q6] Error bars and significance

As the reviewer requested, we report 95% confidence intervals from a paired bootstrap that resamples whole test days (153 days, 10,000 resamples), because windows from the same day are correlated. The gains of JEPACast over FlowCast are significant on all three threshold scores (Table A4), and so are all encoder differences in Table A1. The CRPS difference is discussed in [W2, Q3].

Table A4. JEPACast vs. FlowCast on SEVIR (default samplers, paired bootstrap 95% CI).

MetricJEPACastFlowCastΔ [95% CI]
CSI-M ↑0.47150.4573+0.0142 [+0.0131, +0.0157]
CSI-219 ↑0.20710.1806+0.0265 [+0.0225, +0.0312]
HSS-M ↑0.59420.5753+0.0189 [+0.0173, +0.0209]
CRPS ↓0.01950.0179+0.0016 [+0.0015, +0.0017]

Revision. We will add this comparison with confidence intervals to the revised paper.

AuthorsOpenReview ↗

Response to Reviewer CJp6 (3/3)

[W3] Contribution relative to V-JEPA 2.1

Our contribution is to show what a latent space needs for generative nowcasting and that a predictive latent provides it. We propose decodability and predictability as criteria and a diagnostic to measure them. With this diagnostic we find that the VAE latents of current nowcasters are decodable but less predictable, and with JEPACast we show that forecasting in a predictive latent improves forecasting performance without nowcasting-specific modules. The JEPA objective itself is not our contribution, so JEPACast builds on V-JEPA 2.1 by design, and we will state explicitly that the visible-context loss and the multi-level targets belong to its recipe. Our encoder differs from that recipe in three places: (1) the predictor reads only the last encoder layer, which is the layer that the decoder and the forecaster use, (2) the masked-prediction term is weighted twice as heavily as the visible-context term, and (3) the loss is smooth-L1 instead of L1. As the reviewer requested, Table A1 quantifies their effect, and with the same data, budget, decoder, and forecaster, these changes improve CSI-M by 0.0088.

Pre-training masks space-time tubes with non-causal attention, but the objective is predictive in time as well as in space. A block that is masked in every frame cannot be copied from a neighboring frame, so the encoder has to predict how the masked region evolves from what is visible around it. As a result, a linear readout of the context frames alone predicts future motion with R² +0.757, compared with +0.263 for VideoVAE (128-ch) (Table 1). Our causal temporal-masking variant forecasts lower (Table A1), and combining it with tube masking is a promising direction.

Revision. We will make the attribution to V-JEPA 2.1 explicit and list the differences from its recipe in the revised paper.

[W4, Q5] Sampler selection and other concerns

Sampler selection (Q5). The samplers for both datasets were selected on validation data, not on the test set. Appendix H is a separate analysis, run on the first 2,048 test windows, that only shows how the forecasts change with the sampling hyperparameters. We searched 38 settings on the SEVIR validation split and 63 settings on the MeteoNet validation split, and chose the setting that best balances the CSI metrics. Detailed grids are in our response to Reviewer o7Gu, [Q4].

Additional dataset. To address the limited number of datasets, we added the KMA radar composite over South Korea under the exPreCast protocol (six 10-minute frames predicted from seven, test year 2023), with the sampler again selected on validation data (the year 2022). JEPACast leads exPreCast and CasCast on all six reported metrics (Table A5).

Table A5. Results on KMA (10-member ensemble mean, baselines from exPreCast's Table 1).

ModelCSI-M ↑P4 ↑P16 ↑CSI-20 (P4) ↑CSI-80 (P4) ↑HSS ↑
SimVP0.27620.25580.25980.14440.03810.3958
CasCast0.25730.37420.48370.30660.09260.3806
exPreCast0.27940.37210.48410.30130.08350.4042
JEPACast0.29830.40270.50940.34490.12460.4302

Compute. In Table A1, every row uses the same forecaster, the same 20k training steps, and the same sampler, so no row benefits from a larger forecaster or a longer schedule. Against the published baselines, JEPACast has 337M parameters, compared with 160M for FlowCast, 402M for CasCast, and 643M for FREUD, and its inference cost is lower than that of FlowCast and FREUD (47.5 TFLOP per ensemble member with 10 members, vs. 81.4 and 392.7) and higher than that of CasCast (34.6).

Decoder dependence. By definition, decodability is a property of an encoder–decoder pair (Sec. 3.1, Appendix C.5). In our own experiments, the decoder is controlled, since every latent we trained is decoded by a ViT decoder with the same architecture and training recipe (Table A1). For baselines released with their own decoder, such as the FlowCast VAE, we keep that decoder, because a VAE's encoder and decoder are trained jointly.

Revision. We will describe the sampler selection precisely, replace the figures of Appendix H with their validation versions, and add the KMA results and a cost comparison to the revised paper.

Comments on the paper

1 comment

Dear AC and Reviewers,

We sincerely thank the Area Chair and all reviewers for the time and care they devoted to our paper. The reviews were detailed and constructive, and the experiments they suggested have greatly strengthened our paper. We are pleased that the reviewers recognized the following strengths of our work:

  • Decodability and predictability as criteria for nowcasting latents. We ask what makes a latent space suitable for generative nowcasting and propose a carefully controlled diagnostic for both properties (Reviewers CJp6, o7Gu, Jeua).
  • A latent that is both decodable and predictable. A JEPA encoder trained on radar data remains decodable while making future motion much easier to predict with simple readouts (Reviewers CJp6, Jeua).
  • Strong results with a simple pipeline. JEPACast forecasts in this latent without nowcasting-specific modules, and its gains are consistent with the representation-level findings (Reviewers CJp6, o7Gu, Jeua).

We appreciate the opportunity to address the reviewers' concerns with additional experiments. In our responses, we added the following:

  • Encoder ablations with the forecaster fixed. With the same data, budget, decoder, and forecaster, JEPACast forecasts better than the original V-JEPA 2.1 recipe, masked pixel reconstruction (VideoMAE), a causal temporal-masking variant, and a reconstruction autoencoder of the same width (Table A1 for Reviewer CJp6, Table B1 for o7Gu, Table C1 for Jeua).
  • Control of latent width. To address the concern about over-completeness, we compared it with a reconstruction autoencoder of exactly the same size and with a pixel baseline, and JEPACast forecasts better than both (Table B3 for o7Gu). At an equal width of 16, the V-JEPA latent also has a higher CSI-M than the FlowCast VAE latent (Table C3 for Jeua).
  • Faster learning over the full schedule. The forecaster learns faster on the JEPA latent, matching at 5k steps what it reaches at 45k steps on the reconstruction latent of the same width, and it leads at every evaluated step up to 45k (Table B2 for o7Gu, Table C2 for Jeua).
  • Significance, seeds, and calibration. The gains over FlowCast are significant under a paired bootstrap and hold across three seeds, and at comparable calibration JEPACast keeps its advantage in CSI (Tables A3, A4, and A6 for CJp6, Table B5 for o7Gu).
  • A third dataset. On the KMA radar composite, JEPACast leads the strongest baselines on all six reported metrics (Table A5 for CJp6).
  • Sampler selection. We clarified that the sampler of each dataset was selected on its validation split and not on the test set, and we listed all the settings that were searched ([W4, Q5] for CJp6, [Q4] for o7Gu).

We hope that these results and clarifications resolve the reviewers’ concerns, and we have incorporated all of them into the revised paper. We would greatly appreciate it if the reviewers could take a look at our responses, and we would be glad to answer any remaining questions. We sincerely thank all the reviewers and the Area Chair.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.