acceptodds
Under review as a conference paper at ICLR 2027

Stabilizing Probabilistic Forecasts via Stochastic Interpolation Cascades

Abstract

We study whether generative forecasters sample the correct distribution of future states given coarse observations of a physical system. We build a benchmark pairing observed coarse-grained states with many future states generated by varying the unobserved variables, which enables direct evaluation of conditional distributions. The benchmark reveals that commonly used diffusion models and stochastic interpolants trained directly over long forecast horizons can generate samples with plausible spatial statistics but substantially mismatched conditional distribution. We then propose and analyze a stylized mathematical model of dynamics and prove the existence of a regime where learning and composing shorter transitions outperforms learning the full transition directly. Shorter transitions reduce approximation error, while repeated conditioning can discard hidden information, leading to an optimal intermediate number of stages. This analysis motivates Cascaded SI, which implements staged forecasting with stochastic interpolants. Experiments across several physical systems show substantial improvements in conditional sampling, with gains persisting when the number of supervised transitions is held fixed.

Primary area:
generative models
Keywords:
Conditional generative modeling, Probabilistic forecasting, Stochastic interpolants, Diffusion models, Dynamical systems, Posterior sampling

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_Ka2JOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates probabilistic forecasting under partial observation, comparing direct stochastic interpolants (Direct SI), which learn the conditional distribution between initial and final states, with cascaded stochastic interpolants (Cascaded SI), which learn shorter transitions and compose them to obtain the final forecast.

The authors introduce a benchmark involving five physical systems, where multiple trajectories are generated from the same observed initial condition by varying the unobserved components. This allows the quality of the learned conditional distributions to be evaluated against reference ensembles obtained from numerical simulations.

The experimental results show that Cascaded SI significantly improves conditional forecasting accuracy over Direct SI and EDM in four of the five systems. However, the improvement depends non-monotonically on the number of cascade stages.

To explain this behavior, the authors introduce a theoretical framework describing the trade-off between the approximation error associated with learning long transitions and the loss of information caused by repeatedly conditioning on partially observed states. The analysis establishes conditions under which an intermediate cascade depth is optimal.

Critical strengths

1. Relevant research question and well-designed conditional evaluation benchmark.

The paper addresses an important problem in probabilistic forecasting: whether learning and composing shorter transitions can provide more accurate conditional distributions than directly learning the full forecasting horizon. I particularly appreciate that the work focuses on the quality of conditional distributions rather than simply generating plausible individual forecasts.

The construction of multiple reference trajectories sharing the same observed initial condition is a valuable contribution, as it provides a meaningful way to evaluate whether generative models recover the correct conditional distribution. The inclusion of several physical systems strengthens the experimental evaluation. I also appreciate the presentation of both the conditional mean and pointwise standard deviation in Figure 2, which provides valuable information about the quality of the learned distributions beyond individual generated samples.

2. Interesting theoretical interpretation of the cascading trade-off.

The distinction between approximation error and memory loss provides an interesting explanation for why increasing cascade depth does not necessarily lead to better performance. The theoretical results establishing conditions under which an intermediate cascade depth is optimal are particularly relevant.

Combined with the experimental improvements obtained by Cascaded SI across different physical systems, this provides a compelling motivation for investigating the trade-off between learning longer transitions directly and composing shorter transitions.

Critical weaknesses

1. General theoretical analysis is not sufficiently emphasized in the main paper (Section 4 and Appendix C).

While I appreciate the theoretical analysis, I find that some of the most interesting and general results are left in Appendix C. In particular, Section 4 mainly focuses on the rotation–OU model, while Appendix C develops a more general framework explaining why cascading can reduce approximation error and why repeated conditioning on partially observed states can introduce a memory error.

Theorem 11, for example, provides an interesting characterization of the limiting behavior of increasingly fine cascades under specific assumptions. I believe these results represent an important part of the scientific contribution and deserve greater emphasis in the main paper.

I would suggest including at least a concise presentation of the general theoretical results in Section 4, while keeping the detailed proofs in the appendix. This would help readers understand how the conclusions drawn from the stylized rotation–OU model relate to the broader class of dynamical systems considered in the paper.

2. Limited characterization of when cascading is preferable to direct forecasting.

In Section 3.3 and Figure 3, the authors study how forecasting performance changes with the cascade depth S, while keeping the final forecasting horizon T fixed for each physical system.

However, an important question remains: how does the relative performance of Direct SI and Cascaded SI change as the forecasting horizon itself increases? For relatively short horizons, directly approximating the conditional endpoint distribution might be sufficiently simple, while for longer horizons the advantage of composing shorter transitions could become more significant. On the other hand, repeated conditioning on incomplete observations could also lead to increasing memory loss.

Furthermore, the experiments do not directly isolate the contribution of memory loss to the performance degradation observed at larger cascade depths. Appendix C.6 introduces a theoretically motivated alternative that retains the initial observation throughout the cascade, but this variant is not evaluated experimentally.

These limitations make it difficult to establish precisely under which conditions cascading is preferable and to what extent the observed non-monotonic behavior is explained by the theoretical mechanisms identified in the paper.

Questions for authors

1. How does the optimal cascade depth change with the forecasting horizon T? (Section 3.3)

Would it be possible to extend the experimental evaluation by varying the forecasting horizon T and comparing Direct SI and Cascaded SI across different horizons and cascade depths?

In particular, a plot showing the forecasting error as a function of T for the different approaches, together with the evolution of the optimal cascade depth S*(T), would provide valuable insight into when cascading becomes advantageous and how its benefits change with the forecasting horizon.

I believe this would substantially improve our understanding of the applicability of the proposed method.

2. Can the effect of memory loss be isolated experimentally? (Sections 4.1–4.3 and Appendix C.6)

In lines 369–373, the authors explain that repeatedly conditioning on the current observed state can lead to information loss, even when the individual short-time conditional transitions are known exactly.

I find this distinction interesting, but I would appreciate a clearer discussion of how this mechanism relates to exposure bias and error accumulation in autoregressive forecasting.

In particular, exposure bias arises from the discrepancy between conditioning on ground-truth states during training and generated states during inference, while the memory error discussed here can occur even with exact transition kernels. Although these mechanisms are different, they may interact in practical learned forecasting systems.

Moreover, Appendix C.6 introduces an interesting alternative in which the initial observation X₀ is retained as additional conditioning information throughout the cascade. Proposition 12 shows that this construction recovers the correct endpoint conditional distribution when the transition kernels are exact.

Would it be possible to compare the proposed Cascaded SI with a variant that additionally conditions each transition on X₀? Such an ablation could help determine to what extent the observed degradation at large cascade depths is associated with memory loss, and whether retaining the initial condition provides practical benefits despite the additional conditioning complexity.

3. How sensitive are the conclusions to the distribution of hidden initial components? (Section 2)

The hierarchical sampling procedure is an interesting contribution, as it allows the authors to construct reference conditional ensembles by fixing the observed initial state and varying the hidden components.

However, I would be interested in understanding how sensitive the conclusions are to the distributions used to generate the hidden initial states.

Appendix B.1 describes the construction of these components for the different systems. In some cases, the observed and hidden Fourier components are generated independently.

Since the conditional forecasting problem is determined by the distribution of hidden states compatible with a given observation, could the authors discuss whether the results would change under different hidden-state distributions, particularly when observed and unobserved components are statistically dependent?

This would help establish how broadly the conclusions extend beyond the specific synthetic benchmark construction.

Additional minor presentation suggestions

  • Introduction, lines 30–32: I would suggest framing the problem more explicitly in terms of partially observed dynamical systems rather than primarily describing observations as coarse. Coarse-graining is one particular form of partial observation, while the broader problem also includes missing variables and incomplete measurements.

  • Section 2, line 79: The term "hierarchical sampling design" could be introduced more explicitly. Although the construction is subsequently described, a brief explanation of the two levels of sampling would improve readability.

  • Section 2, lines 95–100: When introducing the hidden initial components Z₀, I would suggest explicitly directing the reader to Appendix B.1, where their distributions and construction are described. A brief summary in the main text would also help clarify how the reference conditional ensembles are generated.

  • Section 3.1, lines 186–194: The statement that Direct SI should recover the correct conditional distribution in the ideal setting, whereas Cascaded SI need not do so under partial observation, is important and could benefit from a more intuitive explanation. In particular, briefly discussing the non-Markovian nature of the observed process would help readers understand the motivation for the subsequent theoretical analysis.

5 comments

We thank the reviewer for their thoughtful and constructive feedback, particularly for highlighting the importance of our theoretical results. We have uploaded a revised manuscript alongside this response, incorporating additional requested experiments, theoretical analysis, and clarifications addressing the reviewer’s suggestions. Below, we respond to each point.

1. General theoretical results in the main paper

“While I appreciate the theoretical analysis, I find that some of the most interesting and general results are left in Appendix C. In particular, Section 4 mainly focuses on the rotation–OU model, while Appendix C develops a more general framework explaining why cascading can reduce approximation error and why repeated conditioning on partially observed states can introduce a memory error. Theorem 11, for example, provides an interesting characterization of the limiting behavior of increasingly fine cascades under specific assumptions. I believe these results represent an important part of the scientific contribution and deserve greater emphasis in the main paper. I would suggest including at least a concise presentation of the general theoretical results in Section 4, while keeping the detailed proofs in the appendix. This would help readers understand how the conclusions drawn from the stylized rotation–OU model relate to the broader class of dynamical systems considered in the paper.”

We thank the reviewer for this suggestion. We agree that the general theoretical results deserve greater emphasis in the main paper, and we have therefore expanded the introduction to Section 4 with two paragraphs summarizing the results on approximation error and memory loss developed in Appendix C.

The first paragraph explains how shorter transitions can reduce approximation error and how composing simple transition maps can capture dependencies that a single map cannot represent. The second explains why partial observation can make the observed process non-Markovian, so that even exact conditional transitions need not compose into the correct conditional endpoint distribution. We also provide a concise statement of Theorem 11, including its assumptions: for the deterministic dynamics considered there, increasingly fine cascades converge to a deterministic conditional forecast while preserving the correct one-time marginals, leaving a nonzero memory error whenever the true conditional forecast has nonzero spread.

These additions clarify the general mechanisms before the rotation–OU example, whose analysis remains unchanged. Detailed assumptions and proofs remain in Appendix C.

2. Performance and optimal cascade depth across forecast horizons

“[…] how does the relative performance of Direct SI and Cascaded SI change as the forecasting horizon itself increases? For relatively short horizons, directly approximating the conditional endpoint distribution might be sufficiently simple, while for longer horizons the advantage of composing shorter transitions could become more significant. […] How does the optimal cascade depth change with the forecasting horizon TT? (Section 3.3) Would it be possible to extend the experimental evaluation by varying the forecasting horizon TT and comparing Direct SI and Cascaded SI across different horizons and cascade depths? In particular, a plot showing the forecasting error as a function of TT for the different approaches, together with the evolution of the optimal cascade depth S∗(T)S^\ast(T), would provide valuable insight into when cascading becomes advantageous and how its benefits change with the forecasting horizon.”

We have included the requested horizon–depth experiment and plot (Figure 6) in Appendix B.2.3. We studied five forecasting horizons on Euler, T∈{0.2,0.4,0.6,0.8,1.0}T \in \lbrace 0.2,0.4,0.6,0.8,1.0 \rbrace. We followed the benchmark protocol (same data, architecture, selection and metric) with three training seeds per configuration, and a fixed training-pair budget of 15,840 instead of 16,000 since 15,840 is divisible by every depth. At T=1T=1 it reproduces the benchmark (S∗=10S^\ast=10; 16.6 vs. 16.3 for Cascaded SI).

Test csFID-10k (lower is better):

Horizon TTDirect SICascaded SISelected S∗(T)S^\ast(T)
0.241.319.24
0.466.620.08
0.672.317.98
0.865.618.18
1.059.816.610

Scores are averaged over three training seeds and ten test conditions. At each horizon, both methods are evaluated against the same horizon-specific reference ensembles.

Cascading improves performance at every horizon, by roughly a factor of two at T=0.2T=0.2 and three to four at longer horizons. The selected depth generally increases with TT, from S∗=4S^\ast=4 to S∗=10S^\ast=10. This is consistent with the preferred segment duration T/S∗T/S^\ast remaining between 0.05 and 0.1. The full depth sweeps in Figure 6 (Appendix B.2.3) also show that too deep cascading increases the error.

Absolute csFID values need not increase monotonically with TT, since each horizon has its own reference ensembles, so csFID-10k compares methods at a fixed TT, not one method across TT.

3. Memory loss versus error accumulation: the anchored cascade

“[…] the experiments do not directly isolate the contribution of memory loss to the performance degradation observed at larger cascade depths. Appendix C.6 introduces a theoretically motivated alternative that retains the initial observation throughout the cascade, but this variant is not evaluated experimentally. […] Can the effect of memory loss be isolated experimentally? (Sections 4.1–4.3 and Appendix C.6) […] Moreover, Appendix C.6 introduces an interesting alternative in which the initial observation X0X_0 is retained as additional conditioning information throughout the cascade. Proposition 12 shows that this construction recovers the correct endpoint conditional distribution when the transition kernels are exact. Would it be possible to compare the proposed Cascaded SI with a variant that additionally conditions each transition on X0X_0? Such an ablation could help determine to what extent the observed degradation at large cascade depths is associated with memory loss, and whether retaining the initial condition provides practical benefits despite the additional conditioning complexity.”

We ran this experiment, which Reviewer NnpT also requested, and extended the theory to learned anchored transitions.

With exact kernels, the anchored cascade recovers the correct distribution (Proposition 12). With a restricted function class, the approximation problem returns: each transition must now also represent its dependence on X0X_0. This is the additional conditioning complexity that the reviewer mentions. In the rotation–OU model, this dependence is a rotation that oscillates faster as ω\omega grows, and the quadratic anchored cascade keeps a positive error for every ω≠0\omega \neq 0 (Theorem 15). A neural-network test agrees: raw anchoring cuts the error at ω=0\omega=0 (0.1504→0.00500.1504 \to 0.0050) but not at ω=12\omega=12 (0.0945→0.09340.0945 \to 0.0934), unless we rotate the anchor beforehand (0.0116).

The analysis implies a gain where the dependence on X0X_0 is simple and possibly none where it is complicated. And indeed:

  • Passive scalar (affine conditional mean, Proposition 5): anchoring lowers the error at every depth S≥2S \geq 2 (csFID-10k 41.14→26.0041.14 \to 26.00 at S=10S=10, 75.21→50.1575.21 \to 50.15 at S=40S=40).
  • Euler (nonlinear): anchoring does not help (16.32 against 17.39 at S=10S=10, 23.29 against 26.49 at S=40S=40).

So retaining X0X_0 gives a practical benefit on passive scalar and none on Euler. In both systems, errors increase with depth under anchoring, so memory loss is not the sole cause of degradation. Errors in learned transitions can also contribute to the degradation and accumulate during rollout. Full results are in Appendix B.2.1 (Figure 5, Table 9) and Appendices C.8–C.9 (Theorem 15, Table 14).

4. Relation to exposure bias and error accumulation in autoregressive rollouts

“In lines 369–373, the authors explain that repeatedly conditioning on the current observed state can lead to information loss, even when the individual short-time conditional transitions are known exactly. I find this distinction interesting, but I would appreciate a clearer discussion of how this mechanism relates to exposure bias and error accumulation in autoregressive forecasting. In particular, exposure bias arises from the discrepancy between conditioning on ground-truth states during training and generated states during inference, while the memory error discussed here can occur even with exact transition kernels. Although these mechanisms are different, they may interact in practical learned forecasting systems.”

Super interesting question! A learned cascade has three separate sources of error:

  1. Memory error. The observed process is not Markov, so transitions that condition only on the current observed state lose information. This happens even with exact transition kernels (Appendix C.6).
  2. Transition error. Each learned kernel is imperfect, and these errors add up along the rollout.
  3. Exposure bias. A learned kernel is trained on true states and applied to generated ones, which can lie outside its training distribution.

The second and third vanish when the kernels are exact. The first does not. With exact kernels, the generated intermediate states even have the correct marginal distribution, so the model never leaves its training distribution, and the memory error is still there.

In learned cascades all three are present and can interact, as the reviewer says. The anchored experiment above shows this on passive scalar: conditioning on X0X_0 lowers the error at every depth, yet the error still increases with depth.

Reviewer_Ga6pOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors are considering the problem of forecasting, especially in the context of partial observations, and unobserved components affecting the system dynamics. The authors discern two approaches in the relevant literature: direct stochastic interpolation (DSI) and cascaded stochastic interpolation (CSI). In the former, forecasting is trained on pairs of initial conditions and end states (X0,XTX_0, X_T), whereas in the second the interval [0,T][0, T] is split into SS intervals (Xti,Xti+1)(X_{t_i}, X_{t_{i+1}}) and training is performed on these pairs.

The authors' interest is pragmatic; they investigate how short the transitions ti+1−tit_{i+1} - t_i should be for optimal performance, while ensuring that both approaches have equal training budget. The main contributions are:

  1. A benchmark for forecasting under partial observations.

  2. Interpretations of the experimental results, including an extensive analysis on a relatively simple model.

Critical strengths
  1. I appreciate the depth at which the authors investigate the effect of the different number of cascades.
  2. The stylized example is a nice way of presenting the main points (although see my concern below).
Critical weaknesses
  1. While this is interesting work, it seems mostly an incremental addition to the work by Albergo et al., who introduce the CSI approach. The addition of a benchmark and illustrations/potential explanations of the experimental findings do not constitute a very clearly defined innovation.
  2. To a non-expert like me, it is unclear how much the rotation Orstein-Uhlenbeck process is representative of the impact of cascading stochastic interpolators in general.
Questions for authors
  1. Why did you pick this specific example and how do its results generalize to e.g. forecasts in weather, finance, healthcare?
  2. You specifically pick systems that evolve periodically. Why? Isn't forecasting specifically challenging (and interesting) for systems that are not, or at least not predominantly, periodic?
5 comments

We thank the reviewer for the direct and incisive review. Below, we provide detailed answers.

1. Novelty relative to prior work on Cascaded SI

“While this is interesting work, it seems mostly an incremental addition to the work by Albergo et al., who introduce the CSI approach. The addition of a benchmark and illustrations/potential explanations of the experimental findings do not constitute a very clearly defined innovation.”

Indeed, cascading stochastic interpolants has been used in prior work. We acknowledge this throughout the paper. But it was done primarily as a matter of convenience. What has remained poorly understood is how, when, and why to cascade. Our starting point was the discovery of a previously unreported failure of direct state-of-the-art generative forecasters: their samples can look realistic while coming from the wrong conditional distribution. This discovery was made by what we think is a creative experiment design, not just an iid benchmark. (We elaborate below.) We therefore investigated under which conditions learning and composing shorter transitions could improve the forecast over the same prediction horizon. We found that the physical time interval separating the states in each training pair can dramatically change conditional forecasting accuracy, but shorter transitions are not always better. Several effects compete: shorter transitions can be easier to approximate, while errors can accumulate during the rollout and information can be lost through repeated conditioning on an incomplete state. How much there is to gain also depends on how complicated the target conditional distribution is as a function of the initial observation. In some cases, direct forecasting is best and cascading makes the forecast worse.

To study this properly, we needed multiple futures for the same observed initial state. Existing benchmarks typically contain only one trajectory per initial condition, which prevents this direct comparison of conditional distributions. We therefore designed a hierarchical sampling procedure and used it to construct reference ensembles for five physical systems. These ensembles allow us to compare the learned and true conditional distributions for each observation, and to compare cascades of different depths against direct one-step SI at a fixed prediction horizon—to the best of our knowledge, the first systematic comparison of this kind. We have made the benchmark openly available (see our global comment) so that others can test whether their forecasters recover the conditional distributions they are intended to learn.

Alongside this empirical study, we develop a theoretical analysis of the competing effects behind the observed behavior. As elaborated in our answer to Reviewer NnpT, the solvable rotation–OU model proposed in Section 4 lets us separate approximation error from memory loss under partial observation and establish how their interaction can produce a finite optimal cascade depth. It explains both the gains from shorter transitions and the deterioration when the cascade becomes too deep. The model brings together these two effects, which we study separately under more general assumptions in Appendix C.

Together, experiments and theory turn cascade depth into a design choice whose benefits and limitations can finally be understood, providing guidance for the design of future generative forecasters.

2. Representativeness of the rotation–OU model

“To a non-expert like me, it is unclear how much the rotation Orstein-Uhlenbeck process is representative of the impact of cascading stochastic interpolators in general.”

First, we emphasize that our theoretical analysis is not limited to the rotation–OU model: Appendix C establishes more general results on how the conditional mean can leave a fixed approximation class as the horizon grows and how, under partial observation, composing exact short transitions can lose the conditional spread of the forecast (Theorems 2 and 11). As suggested by Reviewer Ka2J, who found these results interesting, we have added two paragraphs to Section 4 and extended Appendix C of the revised paper to highlight these more general results and clarify their connection to the rotation–OU model.

In the general analysis, we treat approximation error and memory loss separately; with the rotation–OU model we bring them together in a setting where the forecast error can be computed at every cascade depth. It allows us to control two features relevant to our physical forecasting problems: the complexity of the forecast’s dependence on the observation and the information carried across transitions by unresolved dynamics. The rotation makes the conditional mean increasingly oscillatory in the conditioning variable as the horizon grows, while the hidden OU process plays the role of unresolved fine scales whose influence persists across transitions. For a stochastic interpolant that uses polynomials of degree at most two as its approximation class, shorter transitions can reduce the error in approximating the rotation, while repeated conditioning on the observed state discards information carried by the hidden process. Crucially, we can vary the hidden-process contribution without changing the target endpoint conditional distribution, isolating the memory effect without changing the direct forecasting task. The model thus establishes how these competing effects can produce a finite optimal cascade depth, providing a mechanism that can be explicitly analyzed and is consistent with that observed in our PDE experiments.

3. Choice of the rotation–OU example

“Why did you pick this specific example and how do its results generalize to e.g. forecasts in weather, finance, healthcare?”

The rotation–OU model was chosen because its target conditional distribution, learned forecasts, and forecasting errors can all be calculated explicitly, while approximation difficulty and hidden-state memory can be controlled separately. An analysis for general partially observed systems is carried out in Appendix C; we will update the text to make the distinction clearer, and include the main general results as well. In particular, increasingly fine cascades of exact observed-state transitions converge to a deterministic flow driven by the conditional mean velocity. Such cascades therefore lose the conditional variability induced by unresolved components, even while preserving the correct one-time marginal distributions.

The broader concern is forecasting problems in which the current observation omits information from the past that remains predictive of the future. This can arise in the suggested scenarios (weather, finance, and healthcare). The rotation–OU model captures the most relevant aspects of these complex systems, which are nonlinear, only partially observed, probabilistic in nature, and whose forecasts depend sensitively on the initial observed state in some regions of the parameter space. We also note that such reduced models are often common in modeling complex systems, including those highlighted by the reviewer (e.g. the Lorenz system, shell models for Navier-Stokes, or the Black-Scholes equation).

We appreciate these comments and revised the presentation to make our contributions and the role of the analytical example clearer.

4. Clarification about periodicity

“You specifically pick systems that evolve periodically. Why? Isn’t forecasting specifically challenging (and interesting) for systems that are not, or at least not predominantly, periodic?”

Our systems are not periodic in time, but they have periodic boundary conditions in space. This is standard to isolate dynamics from boundary effects. (And it is similar, with a different topology, to (for example) the weather on Earth, which is periodic along great circles.)

If the question also refers to the analytical example, its rotation component is periodic, but the full rotation–OU process includes stochastic forcing and does not have periodic sample paths. The rotation provides a tractable way to control the dependence of the forecast on the observation, and temporal periodicity is not an assumption required by the general results in Appendix C.

Thank you again for your review! As the discussion deadline approaches, we would greatly appreciate hearing your thoughts on our responses and revisions (which include several new experiments and ablations requested by the other reviewers), particularly whether they have adequately addressed your concerns. If questions or concerns remain, a response would give us a chance to provide further clarification before the discussion closes.

Reviewer_NnpTOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether generative forecasters recover the conditional distribution of future observed states when only part of the state, here the coarse Fourier modes, is observed. It introduces a benchmark of five two-dimensional PDE systems (Euler, surface quasi-geostrophic flow, heterogeneous waves, nonlinear Klein–Gordon, passive scalar). For each coarse initial condition, many fine-scale completions are drawn and evolved with a high-resolution solver, which yields a reference ensemble for the conditional law at the forecast horizon. On this benchmark, stochastic interpolants and conditional diffusion models trained directly on endpoint pairs produce realistic-looking samples with clearly wrong conditional means and spreads. Composing shorter learned transitions (Cascaded SI) improves conditional accuracy by large factors on the four non-trivial systems, with an optimal intermediate number of stages. The authors explain this with two competing effects: shorter transitions are easier to approximate, while repeatedly conditioning on the current observed state discards information carried by the unresolved scales. They support this with a general analysis and an explicit rotation–Ornstein–Uhlenbeck model in which the optimal depth is finite.

Critical strengths

The benchmark is the main contribution. Generative forecasters are usually evaluated against a single reference trajectory or through marginal statistics such as spectra, which cannot reveal whether the conditional distribution is correct. Reference ensembles per conditioning state allow a direct check of the conditional mean, spread and full law, and Figure 2 makes the point convincingly: plausible samples can hide substantial conditional errors. The systems are well chosen to span nonlinear transport, dispersive and scattering waves, and a linear Gaussian control case, and the numerical solvers are carefully documented and checked for convergence.

The theoretical analysis of the memory effect is also valuable. Theorem 11 shows that an increasingly fine cascade of exact observed-state transitions converges to the deterministic flow of the conditional mean velocity: it keeps the correct one-time marginals but loses the dependence between initial and final states. This is a clean statement of why too-deep cascades must eventually fail under partial observation. The paper is clearly written, and its limitations are stated honestly.

Critical weaknesses

The failure of direct learning needs a much more careful diagnosis. Direct stochastic interpolants are exact in the population limit, as the authors acknowledge in Section 3.1, so the reported failure must come from estimation, optimisation or sampling, not from the framework itself. The abstract should say so. Several aspects of the setup could produce it. First, the training data have the same ensemble structure as the test data: 80 coarse initial conditions with 200 completions each. The "fixed budget" equalizes the number of training pairs but not the number of distinct conditioning states. Direct models must learn the conditional law from only 80 distinct inputs and generalise to new ones, whereas cascaded models condition on intermediate states, which differ across trajectories, and so see thousands of distinct inputs. The very early checkpoints selected for the direct models are consistent with overfitting. Second, the selected noise levels for the direct models are very large (σ=32\sigma = 32 on waves and σ=128\sigma = 128 on Klein–Gordon, the latter at the boundary of the search) compared with σ=1–2\sigma = 1–2 for the cascades. Combined with 40-step Euler–Maruyama sampling, this raises the possibility of under-resolved sampling. Several direct and diffusion baselines also retain unresolved hyperparameter-search boundaries. Without these issues resolved, the comparison may reflect poorly conditioned or poorly tuned baselines rather than an intrinsic limitation of direct endpoint learning.

The framing of the comparison also deserves discussion. To reach a long horizon, the common practice is to train a forecaster on a shorter horizon and roll it out, which is exactly what Cascaded SI does, as in GenCast and in autoregressive diffusion solvers for PDEs. That such rollouts outperform direct long-horizon learning is therefore expected; the novelty lies in the controlled depth study rather than in the method. It is equally well known that long rollouts accumulate errors and can become unstable. This offers an alternative explanation for the deterioration at large depth, which the paper attributes to memory loss under partial observation. The anchored cascade the authors analyse, conditioning every stage on the initial observation as well (Proposition 12), would separate the two explanations: if deterioration is due to memory loss, it should disappear, whereas error accumulation would persist. This experiment is not performed. On the approximation side, the argument that compositions of short quadratic fits outperform a single quadratic fit concerns a fixed function class and says little about neural networks, whose limitation here is more plausibly statistical.

The benchmark itself is synthetic, which limits its scope. Its notion of the conditional law rests on an independent prior on the unresolved scales, an idealisation, since in real systems the unresolved scales are dynamically tied to the resolved ones. In practice only one realisation per condition is available, so the benchmark is useful for comparing methods on synthetic problems but is not a validation protocol for real forecasts, whose conditional calibration can only be assessed in aggregate through proper scoring rules. The ensemble-based training protocol is likewise unrealistic. Finally, the comparison is not matched in compute (200 to 6,400 network evaluations for the cascades versus 40 for the direct models), and results rest on a single training seed and ten test conditions.

Questions for authors

Do the direct models reproduce the conditional law on the training conditions? Comparing training- and test-condition errors would distinguish a generalization failure, due to too few distinct conditioning states, from an optimisation, sampling or implementation issue.

How do the direct models perform with more sampler steps and with moderate noise levels comparable to those of the cascades, and can the boundary-flagged hyperparameter searches be resolved?

How do the methods compare under a realistic training protocol with many distinct initial conditions and a single trajectory each, or with lag-TT pairs taken from long trajectories?

Can you evaluate the anchored cascade conditioned on the initial observation, to determine whether the deterioration at large depth comes from memory loss or from error accumulation along the rollout?

Relatedly, how does performance behave for horizons much longer than T, where rollout instabilities typically appear?

How do the methods compare at matched inference cost, and how variable are the results across training seeds?

Finally, how sensitive are the conclusions to the choice of prior on the unresolved scales, for instance a fine-scale distribution correlated with the coarse state?

7 comments

We thank the reviewer for their positive, constructive feedback! Below, we address their remaining concerns. We have uploaded a revised manuscript alongside this response, incorporating additional experiments, theoretical results, and clarifications addressing the reviewers’ concerns. We hope these clarifications together with the already positive evaluation of the paper by the reviewer will lead to an increased score.

1. Abstract change

“Direct stochastic interpolants are exact in the population limit, as the authors acknowledge in Section 3.1, so the reported failure must come from estimation, optimisation or sampling, not from the framework itself. The abstract should say so.”

Indeed, our findings concern models of finite capacity trained on finite data. We made this explicit in the abstract:

The benchmark reveals that commonly used diffusion models and stochastic interpolants, while capable in principle of sampling the correct conditional distribution, can generate samples with plausible spatial statistics but wrong conditional distributions when trained directly over long forecast horizons with finite data and model capacity.

2. Number of distinct conditioning states

“The ‘fixed budget’ equalizes the number of training pairs but not the number of distinct conditioning states. Direct models must learn the conditional law from only 80 distinct inputs and generalise to new ones, whereas cascaded models condition on intermediate states, which differ across trajectories, and so see thousands of distinct inputs”

To check whether increasing the number of initial conditions used to train Direct SI brings its performance closer to that of Cascaded SI, we ran an additional experiment on the Euler dataset. We increased the number of initial conditions from 80 to 800 while keeping the total number of training pairs fixed at 16,000. We compared three configurations:

  • Direct SI: 80 initial conditions × 200 trajectories
  • Direct SI: 800 initial conditions × 20 trajectories
  • Cascaded SI (S=10S=10): 80 initial conditions × 20 trajectories × 10 transitions

All models use the same U-Net architecture as in the main benchmark, 30,000 training updates, and independently tuned hyperparameters. All models were evaluated using the same csFID-10k metric and test reference ensembles as in Table 2. Results are averaged over two training seeds.

Euler csFID-10k (lower is better; mean ± SEM across ten test conditions):

Direct SI (80 × 200)Direct SI (800 × 20)Cascaded SI (S=10S=10)
59.9±3.459.9 \pm 3.449.1±3.549.1 \pm 3.516.6±0.8\boldsymbol{16.6 \pm 0.8}

Training Direct SI on ten times as many initial conditions improves its performance by 18%. However, Cascaded SI still performs approximately three times better, despite using one-tenth as many initial conditions and physical trajectories as the 800-condition Direct SI baseline.

This was achieved with the cascade depth fixed at S=10S=10; optimizing the depth might further improve performance. We included the experimental details and results in Appendix B.2, under the paragraph “Effect of training-condition diversity.”

3. Training- versus test-condition error of the direct models

“The very early checkpoints selected for the direct models are consistent with overfitting. […] Do the direct models reproduce the conditional law on the training conditions? Comparing training- and test-condition errors would distinguish a generalization failure […] from an optimisation, sampling or implementation issue.”

We evaluated Direct SI, EDM, and Cascaded SI on all 80 training conditions and 10 test conditions on all five systems. We compared the validation-selected checkpoints with the final checkpoints after 30,000 updates, using the original samplers and unbiased energy distance (EDU\mathrm{ED}_U; 100 generated and 100 reference samples per condition).

Training and test EDU\mathrm{ED}_U (lower is better):

SystemModelTrain (30k)Test (30k)Test (val-selected)
EulerDirect SI−0.022.011.58
Cascaded SI0.130.220.19
Passive scalarDirect/Cascaded SI (S∗=1S^\ast=1)−0.020.230.16

Small negative values are possible with the unbiased estimator.

On Euler, Direct SI fits the training conditions very well, reaching near-zero energy distance after 30,000 updates. However, its test error increases from 1.58 at the validation-selected checkpoint to 2.01 at 30,000 updates. Cascaded SI does well throughout. We see the same on SQG, waves, and NLKG. On passive scalar, Direct SI generalizes well without cascading.

This is also why the selected Direct SI checkpoints are early. With more training, Direct SI gets better on the training conditions and worse on unseen ones. Since we select checkpoints on validation data, the early ones are chosen.

EDM has the same generalization problem on Euler, SQG, and waves, and generalizes well on passive scalar. On NLKG, EDM does not fit its training conditions well (see Rebuttal Section 10). The full comparison for all three methods and all five systems is in Appendix B.2.2.

4. Sampler resolution and matched inference cost

“Combined with 40-step Euler–Maruyama sampling, this raises the possibility of under-resolved sampling. […] How do the direct models perform with more sampler steps […]? […] How do the methods compare at matched inference cost?”

Our benchmark protocol already includes a sampler calibration and resolution test (Appendix B.2): for each method and depth, the number of sampler steps is calibrated by increasing it until doubling no longer changes the validation score by more than 5%, which gives 40 steps for Direct SI and EDM in every system. The model selected for reporting is then re-checked at twice its step count, and none needed more steps.

To further address the reviewer’s concern, we ran an additional experiment, in which we reevaluate the selected Direct SI and EDM checkpoints using the same number of network evaluations (NFE) as Cascaded SI:

csFID-10k (lower is better):

SystemDirect SI (40 → matched NFE)EDM (40 → matched NFE)Cascaded SI
Euler57.02→57.1157.02 \to 57.1156.21→56.2456.21 \to 56.2416.32\boldsymbol{16.32} (800 NFE)
SQG138.48→140.45138.48 \to 140.45118.50→118.56118.50 \to 118.5651.24\boldsymbol{51.24} (1,600 NFE)
Waves115.83→114.56115.83 \to 114.56123.02→123.05123.02 \to 123.0536.66\boldsymbol{36.66} (200 NFE)
NLKG98.27→96.1098.27 \to 96.1092.32→92.2692.32 \to 92.2616.24\boldsymbol{16.24} (6,400 NFE)

The 40-step values were re-generated for this comparison from the Table 2 checkpoints and agree with Table 2 to within 0.24 csFID (different GPU hardware). Cascaded SI values are those of Table 2.

Additional sampling steps have essentially no effect on EDM. Direct SI improves slightly on waves and NLKG (1–2%) but remains far behind Cascaded SI. On Euler and SQG, more steps do not help. We added this table and a short explanation to our revised paper in Appendix section B.2 under paragraph “Sampler resolution.”

The performance gap between Direct SI/EDM and Cascaded SI persists when inference NFE is matched.

5. Training with one trajectory per initial condition

“The ensemble-based training protocol is […] unrealistic. […] How do the methods compare under a realistic training protocol with many distinct initial conditions and a single trajectory each […]?”

Great point. Our use of multiple trajectories per initial condition was simply motivated by the fact that we had the data generated this way. In fact, the hierarchical training protocol is not necessary to observe a benefit from Cascaded SI, although it allows us to use the simulated data more effectively.

To show this and address the reviewer’s concern, we ran an additional experiment on Euler with 16,000 distinct training initial conditions and exactly one trajectory each. Both Direct SI and Cascaded SI use the same 16,000 trajectories and receive only 16,000 supervised pairs, matching the training-data budget of our main benchmark. Direct SI uses one full-horizon endpoint pair per trajectory, while Cascaded SI uses one short transition per trajectory, balanced across the SS segments.

We followed the main benchmark protocol for architecture, optimization, hyperparameter and checkpoint selection, and sampler calibration. Validation and test conditions remain unchanged. We evaluated depths S∈{1,2,5,10}S \in \lbrace 1,2,5,10 \rbrace, selecting S∗=10S^\ast=10 on validation. Depths S=20S=20 and S=40S=40 were omitted due to the rebuttal time budget.

Euler test csFID-10k (mean ± SEM across 10 test conditions; one training seed; lower is better):

ModelcsFID-10k
Direct SI (S=1S=1)44.20±3.2944.20 \pm 3.29
Cascaded SI (S=2S=2)33.77±2.7633.77 \pm 2.76
Cascaded SI (S=5S=5)17.29±1.2517.29 \pm 1.25
Cascaded SI (S=10S=10)14.70±0.48\boldsymbol{14.70 \pm 0.48}

Cascaded SI achieves roughly three times lower csFID-10k. Notably, it achieves a score comparable to the main benchmark (14.70 versus 16.32). These results show that the advantage of cascading does not depend on having multiple training trajectories per initial condition.

6. Memory loss versus error accumulation: the anchored cascade

“The anchored cascade the authors analyse, conditioning every stage on the initial observation as well (Proposition 12), would separate the two explanations: if deterioration is due to memory loss, it should disappear, whereas error accumulation would persist. This experiment is not performed. […] Can you evaluate the anchored cascade conditioned on the initial observation, to determine whether the deterioration at large depth comes from memory loss or from error accumulation along the rollout?”

Following the reviewer’s suggestion, we evaluate anchoring in both the forecasting experiments of the paper (Appendix B.2.1, Figure 5 and Table 9) and the solvable rotation–OU models (Appendices C.8–C.9).

To begin with, it is worth emphasizing that conditioning every cascade segment on the initial observation X0X_0 guarantees that Cascaded SI reproduces the correct conditional distribution only when the transition kernels are exact (Proposition 12, Appendix C.6). Indeed, in our newly added analysis, we show that, with finite representational capacity, retaining X0X_0 does not necessarily improve the forecast: in our rotation–OU model, X0X_0 still enters the forecasting map in a highly oscillatory way and can therefore be difficult to represent with a restricted function class (Theorem 15, Appendix C.8). We also trained neural networks on this model, comparing unanchored SI, anchoring with the original initial observation, and anchoring with an analytically pre-rotated observation. Raw anchoring substantially reduces the error at ω=0\omega=0 (0.1504→0.00500.1504 \to 0.0050), but barely helps at ω=12\omega=12 (0.0945→0.09340.0945 \to 0.0934), unless we rotate the anchor beforehand (0.0116).

The analysis suggests that anchoring can help when the dependence on X0X_0 is simple, but may not help when it is complicated. And indeed:

  • Euler: Anchoring does not improve forecasting accuracy (csFID-10k 16.32 versus 17.39 at S=10S=10, and 23.29 versus 26.49 at S=40S=40). Both variants select S∗=10S^\ast=10.
  • Passive scalar: Its conditional mean is affine at every horizon (Proposition 5), and anchoring improves accuracy at every depth S≥2S \geq 2 (41.14→26.0041.14 \to 26.00 at S=10S=10, and 75.21→50.1575.21 \to 50.15 at S=40S=40). Both variants select S∗=1S^\ast=1.

In both systems the error still grows with depth under anchoring, so memory loss is not the only cause of the degradation: as we write throughout the paper, errors in learned transitions can also contribute and accumulate during rollout.

Thank you very much for these suggestions! We think that all of them further corroborate that our conclusions from the analysis are sound.

Comments on the paper

2 comments

We thank the reviewers for their detailed and constructive reviews! We respond to every point individually below and have uploaded a revised manuscript in which all changes are marked in red.

For reproducibility and to encourage further research, the benchmark datasets, our code, and the trained checkpoints are publicly available through anonymous repositories:

We would like to thank the reviewers for their time and input, which helped us strengthen our paper!

Following our initial revision, we have uploaded a further revised manuscript. In response to the reviewers’ comments, we have changed the title to “Generative Forecasting from Coarse Observations: Does Cascading Help?” to better reflect the scope of our work and emphasize that cascading is not always beneficial. The revised manuscript improves the presentation and organization of the paper, places greater emphasis on the general theoretical results and their connection to our experimental findings, and integrates the additional analyses and experiments conducted during the rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.