acceptodds
Under review as a conference paper at ICLR 2027

Same Target, Different Origins: Aligning Latent Rollout for Consistent PDE Emulation

Abstract

Neural PDE solvers commonly generate multi-step predictions through autoregressive rollout, recursively applying a learned evolution operator and feeding each prediction back as input for the next step. Approximation errors can therefore propagate through the rollout, causing predictions at a shared target time to depend on their temporal origins and rollout depths. For a well-posed deterministic PDE flow, however, states at different times along the same physical trajectory should, under exact evolution, reach the same target state. To this end, we propose **A**lign **L**atent **R**ollout (ALR), a regularization approach that promotes consistency across rollout origins through the neural PDE solver’s internal representations. A second-order Taylor analysis of latent discrepancy propagation motivates two objectives: zeroth-order latent state consistency, which aligns representations at a shared target time, and first-order latent dynamics consistency, which aligns local evolution responses to shared perturbations. ALR can be integrated into different neural PDE solvers by regularizing suitable internal representations, without modifying their backbone architectures. Experiments across multiple PDE benchmarks and neural-operator backbones show that ALR improves prediction accuracy while reducing shared-target discrepancies across rollout origins.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
neural operators, autoregressive rollout, latent consistency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UZ84OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Align Latent Rollout (ALR) is introduced in this paper, which imposes training-time regularization to AR-based NeuralPDEs. For a deterministic PDE, the paper observes this cross-origin consistency phenomenon, where rollouts with different temporal origins would eventually reach identical states. This motivates two loss terms that are derived from the induced latent convolution map. The losses, when added to the standard one-step training objective function, are able to bring consistent relative error reductions.

Critical strengths
  1. This is a light-weight training time regularizer that costs nothing extra during inference. It is agnostic to the underlying model backbone. The comparison is clean and transparent with a uniform gain across several public benchmarks over .

  2. The paper bases itself on the "cross-origin consistency" principle, the semigroup property of deterministic flows. This makes the approach physically well grounded.

Critical weaknesses
  1. Current comparisons are only limited to models with naive unrolling, while other strong baselines, like PDE-Refiner, scheduled sampling, SGNO, and DySlim, are discussed as related work but never actually compared against the proposed method.

  2. The benchmarks are narrow in scope, everything is 2D single channel on regular grids, and no 3D or irregular meshes are included at all. This is concerning because ALR has been reported to perform well on every single dataset, and we want to understand where the boundaries of its limitations are, especially for hard systems with high Reynolds numbers, strongly stiff systems, or systems with bifurcations.

Questions for authors
  1. Why are baselines like PDE-Refiner, scheduled sampling, SGNO, and DySlim not reported? Can the authors provide a comparison of those baselines with the proposed approach?

  2. What is the number of seeds used for reporting the results in Table 1? This is needed to determine the statistical significance of the std bar reported in the table.

  3. Can the authors report ALR's gains on harder systems and datasets, and maybe include a failure mode analysis? To name a few, systems with high Reynolds numbers, strongly stiff systems, or systems with bifurcations.

2 comments

Thank you for your rebuttal, which has addressed most of my concerns. I am happy to raise my score.

We sincerely thank you for your timely reply and positive assessment of our work. Your thoughtful suggestions have helped us strengthen our experiments and improve the paper. We greatly appreciate your time and support.

Reviewer_p6KZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present a method for increasing the stability of neural PDE surrogates by introducing a new regularization objective. This objective aims to align the state and local gradient that is predicted by a surrogate, when queried from different starting points. The method seems to work well, and works across different backbones

Critical strengths
  • The jvp implementation is nice and reduces the compute of needing to materialize a full Jacobian
  • The motivation for the method seems to be sound
  • The method does improve on a baseline surrogate
Critical weaknesses
  • The main issue is the lack of comparison to the cited methods for autoregressive error mitigation (such as PDE-Refiner), and the method is only compared to unrolled training. Adding ALR regularization helps with respect to the base surrogate, but almost any method proposed for autoregressive stability will improve the baseline performance.

  • It is also somewhat surprising that the unrolled training harms rollout accuracy on 2/4 cases it was implemented on. This suggests to me that the implementation may not be ideal? In practice, unrolled training can be unstable (due to error accumulation during training), therefore the gradient can have high variance. Therefore, most implementations use a curriculum (i.e, train w/o unrolled training for 10 epochs, then unroll 1-step for 5 epochs, then unroll 2-steps for 2 epochs … etc.). When training to a large unrolling horizon from scratch, this optimization is usually much more difficult.

  • In general, the method is limited to deterministic/fully-observed settings

Questions for authors
  • Could you provide some more details on the unrolled training? (number of unrolled steps, full backpropagation through time or just backprop through the last step?) It may also be helpful to try a few different unrolling horizons (similar to testing different window sizes).
Reviewer_NJx6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes Align Latent Rollout (ALR), a regularization method for autoregressive neural PDE solvers based on the principle that rollouts originating at different times along the same trajectory should reach a common target state. Motivated by a second-order Taylor analysis of latent discrepancy propagation, ALR penalizes differences between shared-target latent representations and their local evolution responses using Jacobian–vector products. The method introduces auxiliary autoregressive rollouts during training while preserving the original inference procedure. Experiments across multiple PDE benchmarks and neural-operator architectures investigate prediction accuracy, cross-origin consistency, and long-horizon rollout behavior.

Critical strengths
  • Clear and intuitive motivation based on cross-origin consistency. The paper identifies a natural property of deterministic PDE evolution: states originating from different points along the same trajectory should reach the same future state. This provides a meaningful perspective on error accumulation in autoregressive neural PDE solvers.

  • A mathematically motivated regularization framework. The second-order Taylor analysis motivates two complementary objectives: latent state consistency and local dynamics consistency. The latter explicitly encourages similar responses to common perturbations, extending beyond direct state alignment.

  • Architecture-independent and computationally practical implementation. ALR can be incorporated into different neural PDE solvers without modifying their backbone architectures. Stochastic Jacobian–vector products avoid constructing full Jacobians, and the method introduces no additional inference-time overhead.

  • Consistent improvements across multiple PDE benchmarks and architectures. ALR improves prediction accuracy on all four PDE benchmarks and demonstrates benefits across FNO, F-FNO, and LSM, supporting its applicability beyond a particular architecture.

  • Useful ablations and qualitative evidence. The experiments investigate the individual contributions of state and dynamics consistency, the effect of consistency-window length, and cross-origin discrepancies in latent states and local dynamics. These results provide empirical support for the proposed design.

Critical weaknesses
  • Unclear distinctive advantage of the proposed Jacobian regularization. Although the Taylor analysis motivates penalizing Jacobian differences, it does not establish why this objective should be preferable to directly penalizing next-step latent discrepancies or other simpler consistency objectives. Theoretically, exact state alignment already implies Jacobian alignment, and the potential advantage of matching local responses to perturbations is not independently demonstrated.

  • Insufficient comparisons with alternative consistency regularization objectives. The state-only ablation does not establish whether better-tuned state consistency could achieve comparable performance. Important missing alternatives include, for example, direct next-step latent consistency, physical-space cross-origin consistency, and additional supervision of auxiliary rollouts. These comparisons are essential for identifying which components of ALR account for its improvements.

  • Limited competitive baselines and questionable unrolled-training results. The evaluation primarily compares ALR with standard neural operators and a limited unrolled-training baseline, which performs unexpectedly poorly. Comparisons with established rollout-aware training strategies, pushforward training, and existing temporal-consistency or stabilization methods are largely absent. This makes it difficult to assess ALR's competitiveness.

  • Insufficient theoretical and empirical evidence for improved rollout stability. The Taylor analysis describes local discrepancy propagation but does not guarantee stability or error suppression. In particular, discrepancies may be amplified even when the Jacobian consistency loss vanishes. Furthermore, the longer-rollout experiments use separately trained models for different horizons, rather than directly demonstrating generalization from a fixed training horizon to longer rollouts.

  • Substantial training overhead without compute-matched comparisons. ALR increases training time by approximately 2.66× under its default setting. The experiments do not determine whether its accuracy improvements justify this cost relative to simpler regularization methods or established training strategies operating under comparable computational budgets.

Questions for authors

I will consider responses to the weaknesses as answers to my questions.

1 comment

Thank you for the response. The additional experiments substantially strengthen the empirical justification for ALR, particularly the comparisons against alternative consistency objectives and established rollout-aware training methods. I also consider its applicability across different neural-operator architectures an important strength. Rather than requiring a specialized backbone, ALR offers a reusable training-time regularization approach, with benefits demonstrated across several architectures and no additional inference overhead. While some (minor) concerns remain regarding the isolation of auxiliary rollout supervision and the comparative training cost of alternative objectives, the combination of broader applicability and strengthened experimental evidence makes the contribution more convincing. Therefore I will revise my assessment and raise my score.

Comments on the paper

2 comments

Hi,

Thank you for the additional baselines and details on unrolled training. It seems to me that smaller unrolling horizons work better, so it is a bit odd that horizons of just 1 or 2 steps is not tested (given that this is the cheapest + most simple implementation of unrolled training and is the first thing to try). Perhaps I was a bit unclear originally, but my intention was just to try a very simple unrolled training baseline to minimize the amount of extra experiments to run, for example: (0, 300) -> (1, 400) -> (2, 500). (i.e., no unrolling, then unroll 1 step, etc.), or even (0, 400) -> (1, 500).

The chosen curriculum is also quite odd, and does not follow standard practice of reducing the number of training steps as the horizon increases (this is generally done since back-propagating through longer horizons is more costly and less stable, and therefore done for fewer epochs).

For a fair comparison the updated unrolled training settings should also be run for the other datasets as well (KM flow, NS 1e-5), so that the main table is not misleading (currently Table 1 is quite misleading as the only baseline provided (unroll) is the worst implementation of this strategy, and this is not made clear). In my experience even a very simple unrolled training can do quite well, and more complicated curriculum or longer horizons make the training loss much more difficult to optimize.

Also, I am curious why the shallow-water results are different than the ones provided in the main table (Table 1)? It seems to be evaluated in a different setting 40 steps (here) vs. 20 steps (in paper), where the results provided here highlight your method better.

I apologize in advance for this late reply, as I know there is not much time. If it is not possible to run these, then I understand as well. I only bring this up because it is a bit puzzling (and potentially misleading) that the provided experiments are done in a very complex manner and sometimes with different settings than the original results.

AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank the Area Chair and all three reviewers for their constructive feedback, continued engagement, and positive assessments of our work.

In this work, we propose ALR, a training-time regularization approach that promotes cross-origin consistency through latent state consistency and latent dynamics consistency to mitigate error accumulation in autoregressive PDE prediction. ALR is architecture-agnostic, requires no backbone modifications, and introduces no additional inference overhead.

Following the additional experiments and clarifications during the rebuttal, all three reviewers expressed positive assessments of our work:

  • Reviewer NJx6 recognized the strengthened empirical evidence and ALR’s architecture-agnostic applicability without additional inference overhead.
  • Reviewer p6KZ found the updated results rigorous, confirmed that their concerns had been addressed, and explicitly recommended acceptance.
  • Reviewer UZ84 acknowledged that most of their concerns had been addressed.

We greatly appreciate the reviewers’ recognition and suggestions, which have helped strengthen the paper. We also sincerely thank the Area Chair for their time and care throughout the review process and for considering our submission, responses, and additional evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.