acceptodds
Under review as a conference paper at ICLR 2027

The Geometry of Rollout Error

Abstract

Long-horizon rollout remains a central challenge in learned physical dynamics. Models with similar one-step accuracy can produce radically different trajectories, so local prediction error alone does not determine behavior under repeated composition. We analyze this separation through the finite-time transport of local residuals: errors introduced at different times and in different directions can have very different consequences depending on the subsequent learned dynamics through which they propagate. This perspective motivates separating three choices in autoregressive learned dynamics: temporal representation, which determines whether the model learns an instantaneous generator or the finite-time transition that is actually composed at deployment; forward exposure, which determines how many self-fed predicted states are encountered during training; and temporal credit, which determines how far learning signals propagate backward through that rollout. Across ten evaluation conditions spanning nine physical systems from The Well, DISCO-FLO, a direct finite-time-transition formulation of DISCO, achieves lower final-horizon median error than generator-based DISCO in all ten conditions despite lower one-step median error in only four. A matched-credit experiment shows that the representation effect persists when generator and transition models receive the same self-fed exposure and backward credit, while credit-depth ablations show that increasing temporal credit is not uniformly beneficial across systems. Together, these results suggest that long-horizon learned dynamics should be understood and optimized through the geometry of their deployed composition rather than inferred from local prediction accuracy alone.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
neural operators, dynamical systems, physics-informed machine learning, rollout stability, tangent dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_fFg6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies long-horizon prediction in learned physical dynamics, focusing on why models with similar one-step errors can exhibit substantially different rollout behavior. The authors analyze this through the finite-time transport of local residuals and distinguish three factors in autoregressive dynamics: temporal representation, forward exposure, and temporal credit. They compare a direct finite-time transition model with generator-based dynamics across multiple physical systems and show that the direct transition formulation achieves better long-horizon prediction despite not consistently improving one-step accuracy. Additional matched-credit and credit-depth experiments suggest that the advantage is primarily due to the temporal representation rather than differences in rollout exposure or backpropagation depth. Overall, the paper argues that long-horizon performance should be evaluated through the geometry of repeated composition rather than inferred from local prediction error alone.

Critical strengths
  • The core idea is novel and conceptually well motivated.
  • The paper is generally easy to read and follow, and the main intuition is communicated clearly.
Critical weaknesses
  • The overall writing and presentation structure has several issues, which makes the contribution feel less sharply organized than it could be.
  • The core contribution currently feels somewhat limited in scope. The paper identifies one main phenomenon and builds the learning framework around that observation, but the contribution beyond this central finding is relatively modest.
  • The related-work discussion also appears too limited. The number of references is relatively small, and several relevant lines of literature seem to be missing or only briefly acknowledged. In particular, the introduction should provide a broader and more systematic discussion of prior work on long-horizon prediction, autoregressive rollout error, learned physical dynamics, and finite-time versus infinitesimal dynamics. Strengthening the literature review would help clarify the novelty and positioning of the paper.
Questions for authors

Could you improve the presentation of the technical material? Important notation, assumptions, and theoretical details should be stated clearly in the main text and then supported by appropriate references to the appendix. Simply giving a brief explanation in the main text and deferring most of the substance to the appendix does not work well here. The core idea and overall construction are interesting, but the writing occasionally feels fragmented and overly template-like.

  • I suggest reorganizing Section 3 into a more theorem-like structure, with clearly stated propositions/lemmas and proofs moved to the appendix or placed after the corresponding statements. In its current form, the section is difficult to parse.
  • Equation (10) does not seem to be used later in the paper. If it is not essential to the subsequent development, I do not see why it needs to be stated in the main text.
  • I also found the transition from Section 3 to Section 4 insufficiently motivated. In particular, the decomposition into the three components introduced there does not feel natural from the preceding analysis. The paper should explain more clearly why these three factors are the appropriate axes of comparison and how they arise from the theory developed in Section 3.
Reviewer_rEXtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks why autoregressive PDE emulators with similar one-step error can diverge so much over long rollouts. It writes the rollout error as a sum of local residuals transported by products of Jacobians, stacks these into a block-triangular operator THT_H, and observes that a weighted rollout loss pulled back to residual space is the quadratic form THTWTHT_H^T W T_H. From this it argues for treating three training choices separately: temporal representation (learn a generator and integrate it, or learn the Δ\Delta-step map directly), forward exposure (how many self-fed steps HH the model sees during training) and temporal credit (how many steps kk gradients flow through). The practical proposal is DISCO-FLO, which replaces DISCO's ODE-integrated generator with a single residual step ut+1=ut+Δθ(ut)u_{t+1} = u_t + \Delta_\theta(u_t) and trains it with an H=8H=8, k=4k=4 rollout loss. On ten conditions from The Well it beats one-step-trained DISCO at the final horizon in every case. A matched-credit experiment on periodic Euler and a sweep over kk on four systems are meant to separate the effect of representation from that of credit.

Critical strengths
  1. The split between forward exposure and temporal credit is genuinely useful, and the way it is implemented (H fixed, only the backward graph truncated, Eq. 17) is a clean way to vary credit without changing the distribution of training states.
  2. Ten conditions from The Well, covering 2D and 3D, periodic and open boundaries, is a broader test bed than most generator-vs-transition studies use.
  3. The derivations in the appendix are correct and easy to follow, and the limitations section is honest (stable is not the same as physically faithful; the matched experiment covers one system).
Critical weaknesses
  1. Most of the headline gap comes from rollout training, not from the representation. In Table 1 vanilla DISCO is trained with a one-step loss while DISCO-FLO is trained with an H=8 rollout loss, and it is well established that rollout/pushforward training removes much of the blow-up (Brandstetter et al., 2022; List et al., 2025; Koehler et al., 2024). Roughly 2.5 orders of magnitude of NRMSE come from the training objective and a factor of about 1.5 from representation, and that factor rests on one system, one seed and medians close to NRMSE = 1. The 10/10 result and the claim that DISCO-FLO "substantially improves long-horizon prediction" should not be read as evidence about representation.
  2. In the matched experiment, "generator vs transition" is really four Euler substeps vs one. DISCO-FLO is exactly one forward-Euler step of size 1 applied to the network output (Eq. 62); the matched generator is four Euler substeps of size 1/4 with the same network (l. 996–999). Both are discrete maps built from the same network and differ only in how many times it is composed per step, which changes effective depth and per-step compute (4 evaluations vs 1). A neural ODE trained under a fixed discretisation behaves like that discrete map rather than a vector field (Ott et al., ICLR 2021), so the "instantaneous generator vs finite-time transition" framing of Sec. 4.1 is not what is actually being tested. A sweep over substeps (1, 2, 4, 8) would show whether there is a smooth trend or a genuine qualitative split.
  3. The evidence is statistically thin and the protocol has gaps. Every primary result is a single checkpoint per arm, with no seeds or confidence intervals. The system-dependent ordering of k=4 vs k=8 in Fig. 3 (right) rests on differences of about 0.1 on medians near 1, which cannot be separated from noise with one seed. Rollouts are evaluated on the validation split that was also used for checkpoint selection. Batches that produce non-finite states are skipped but the counts are not reported, so the two arms may be scored on different trajectory sets, and dropping the generator's worst failures biases its median downward.
  4. The paper builds directly on DISCO (Morel et al., ICML 2025), The Well (Ohana et al., 2024) and FNO (Li et al., 2021) but cites none of them.
Questions for authors
  1. What does DISCO-FLO achieve when trained with the one-step loss only, on all ten conditions?
  2. In the matched experiment, how does the result change with the number of Euler substeps (1, 2, 4, 8)? With one substep, isn't the generator identical to DISCO-FLO up to initialisation?
  3. How many batches were skipped for non-finite states, per system and per arm, and are both arms evaluated on the same trajectories? What happens to Table 1 if non-finite rollouts are counted as failures on a common paired set?
  4. Can you show the seed 1–3 periodic-Euler results, and seed variability for the k-sweep in Fig. 3 (right)?
  5. Section 3 builds THT_H and MH,WM_{H,W}, but no experiment measures anything from it: no Jacobian spectra, no finite-time growth rates, no transported-residual contributions. The paper itself says the design choices "are empirical design choices rather than consequences of the rollout geometry". Can you design some experiment that connects ina stronger way the theory with the practice?
Reviewer_Wxe3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a modification to the DISCO architecture for neural emulation of PDEs via hypernetwork operator learning. They call their architecture DISCO-FLO. As I understand, the key difference is directly learning the Euler step at a fixed time interval, compared to compared to DISCO's training through an adaptive integration scheme like neural differential equations. They motivate this based on what I consider a fairly simple observation that small initial differences may lead to quite different trajectories depending on the model (this is standard knowledge), and use on a linearized description of rollout error to discuss the differences. The authors of this paper demonstrate their technique on some datasets from the Well dataset and claim improved performance over DISCO.

Critical strengths

1/ I thought the results of the comparable periodic-Euler experiment were interesting to see. Generally it's well-known in this community that training to predict delta(state) at a fixed dt is always going to beat a model that learns the temporal gradient, because said model can implicitly learn its own integration scheme, and ingest less of its own noise by virtue of fewer recursive calls. But it is interesting to see a head-to-head test in this way.

Critical weaknesses

1/ So I have some concerns about whether the experiments run in this paper actually establish whether their DISCO-FLO architecture is better than DISCO, because I felt that their DISCO baseline might not adequately represent the original method.

In the main comparison, there are significant differences in both what the prediction is doing and what the training is doing. For example, DISCO is trained through an adaptive integrator to predict the next state, whereas FLO is trained on a fixed-time-step rollout. It's trained eight steps in advance, feeding its own predictions back through those eight steps. So I think this is not necessarily a fair comparison. And I don't think the large claimed improvements in Table 1 can be attributed to just learning a fixed time step rather than learning to predict the temporal gradient. I think the additional rollout training itself, training through multiple steps, probably accounts for a substantial part of the difference.

I should say that the authors do acknowledge this, and they thankfully provide a fairer comparison on periodic Euler from The Well. The problem is that this is only one physical system. So it's the one physical system where there is a reasonable matchup. And in this setting, they only get about a third lower median error than DISCO. So it's still an improvement, but it's much less significant than their headline claim. I would like to see a similar comparison on more than just one physical system, together with checks that the result is not sensitive to the integration step size.

In the released DISCO code, the context gets updated after a prediction, and it regenerates its operator from that new context. But in this paper, they don't do that. They actually freeze the context. I don't know if that explains some of the poorer results, but that probably has an effect, and it makes me skeptical of the claims.

2/ The other main issue that I have with the paper is that I'm not sure if the mathematical analysis that they do in this paper actually establishes a sufficiently novel or significant contribution beyond what is already known in the literature, and also I don't think this analysis explains the observed empirical improvements in the paper.

On this, I would say that learning a fixed-time state prediction is standard, and the authors do acknowledge this. They also acknowledge that the basis of their mathematics, the Jacobian product analysis, builds on previous works. From what I can tell, the remaining mathematical contribution is mainly to just collect the derivations into this expression for the rollout loss.

In general I wouldn't say that I object to using a linearized description for this kind of error analysis, but I guess my main problem with this approach is that they present this mathematical analysis, but they don't actually make predictions from their proposed geometry. They don't actually measure this proposed geometry on the trained models.

I think there are many explanations for improved long-horizon performance. So I think this mathematical analysis lacks demonstrated predictive power, and they don't really do anything to convince me that this mathematical analysis describes what's actually going on.

So there is a significant gap I see between the claimed analytical explanation and the actual empirical results. From my perspective, the experiments show that a particular training recipe performs better than the baseline, DISCO, but they don't show that the proposed geometry actually explains it. Given that this underlying modeling choice of predicting state transitions with a fixed time interval into the future is standard in PDE emulation, I don't think the present analysis here is enough to justify this being a substantial contribution.

Questions for authors

1/ Could the authors provide more of the training configuration details? For example, there seem to be some missing details like weight decay, warm-up, and channel-level error aggregation. It also wasn't completely clear to me if there were results on a held-out test partition, or if all the results were just validation results.

2/ Can you show loss curves that show whether the models have converged? My interpretation is that 15 epochs is much shorter than in the original DISCO paper, and I don't know if this has converged yet, or if this reported comparison would change with more training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.