acceptodds
Under review as a conference paper at ICLR 2027

Virtual-Transition Temporal-Difference Learning for Autoregressive PDE Dynamics Models

Abstract

Autoregressive PDE models are typically trained to predict each observed next state from its input window. We study a complementary training choice: which examples should exchange prediction-error information? Building on the Markov reward process formulation for supervised learning, we use virtual-transition temporal-difference (TD) learning to couple residuals. These transitions need not follow the physical dynamics; our default pairs different trajectories at the same time index and leaves the prediction architecture and inference procedure unchanged. For linear models, we quantify the cost of imperfect inverse-covariance weighting: the bound on excess covariance relative to oracle generalized least squares (GLS) is quadratic near exact matching, and the same bound controls leading-order finite-horizon rollout mean squared error (MSE). Strict leading-order improvements over supervised least squares persist for exact rollouts at sufficiently small target noise. A local nonlinear extension characterizes when TD retains GLS efficiency and how weighting mismatch affects rollout error. Controlled experiments verify these theoretical results. Using the same pairing rule across PDE domains, TD improves one-step test accuracy on domains and short-horizon rollout accuracy on , with median error reductions of and ; the median reduction over steps 21–60 is across the eligible domains. Original-scale experiments and pairing, backbone, and multistep studies establish the method's utility across several model and training configurations.

Primary area:
learning on time series and dynamical systems
Keywords:
dynamics learning, temporal difference learning, markov process, PDE dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RUftOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training approach for improving autoregressive neural PDE surrogates. For a given training sample, it considers samples at the same time from different trajectories and uses residuals from an EMA target network to construct a coupled residual. The analysis shows that, under the stated conditions, when the induced weighting approximates the inverse covariance of the target errors, the covariance of the TD estimator approaches that of GLS, yielding better statistical efficiency. The paper also presents experiments demonstrating gains over ordinary supervised training (SL) across multiple PDE domains.

Critical strengths
  1. The paper presents a relatively novel approach that uses residuals from different trajectories at the same time to correct training-sample errors, incorporating TD into the training of neural PDE surrogates. This has the potential to improve predictive accuracy and statistical efficiency.
  2. The experiments have broad coverage and are reasonably convincing. The paper supports its claims through small-model experiments across 18 PDE domains and original-scale model experiments on three PDE domains, obtaining good results.
Critical weaknesses
  1. The theoretical analysis does not explain the gains from the chosen default pairing in practical neural PDE surrogate training. The relevant conclusions depend on the covariance structure of the target errors, but the default pairing does not automatically ensure such a covariance structure during actual training. The experimental results also include cases that require further explanation, such as Acoustic Maze in Table 1.
  2. Table 2 reports only one-step prediction errors. However, one-step errors do not adequately represent the model's performance in long-horizon rollouts. And on some datasets, the performance gap between SL and TD is small, whereas the reported increase in training time on a particular dataset reaches 37%–46%.
Questions for authors
  1. The theoretical gains depend on the covariance structure of the target errors. How should actual training residuals be related to these target errors? Could you clarify under what conditions the default pairing is effective and explain the rollout degradation on Acoustic Maze in Table 1?
  2. Table 2 reports only one-step prediction errors. Could you compare the default pairing with strong alternatives on long-horizon rollouts at the original model scale? For a representative task, could you also provide SL/TD training and validation curves, and rollout error curves for a fixed configuration and checkpoint, to clarify where the long-horizon gains come from?
  3. The scaling experiments report a 37%–46% increase in training time per epoch for TD, while its accuracy gains are small on some tasks. Could you report the training time and hyperparameter-search cost for the settings in Table 1, and compare accuracy under equal training-time budgets? For settings with small gains, could you also provide results across multiple random seeds to establish whether the improvements are consistent?
2 comments
AuthorsOpenReview ↗

Response to Reviewer RUft

Thank you for your questions. We added equal-time, multi-seed experiments and rollout curves. The summary above and Appendix F of the revised manuscript report the results.

Q1 and W1. Target errors, training residuals, and Acoustic Maze. Training residuals combine errors in the training targets with prediction errors from the fitted model. Appendix A.4 considers a specific target-error model: errors have equal variance and equal pairwise correlation across trajectories at the same time step, and are uncorrelated across time steps. Under this model, Cross pairing with a matching discount recovers GLS. Theorem 4.2 quantifies the effect of a mismatch between TD’s weighting and the inverse target-error covariance.

App. B.6 finds a stronger shared residual component in 15 of 16 eligible domains, implying a strong residual correlation across trajectories at the same time step. This supports the motivation for Cross pairing.

On Acoustic Maze, TD and SL have similar one-step error. At the original scale, TD increases rollout error by approximately 6% over steps 1–20 and 11% over steps 21–60 (Table 1). At the reduced scale, TD lowers error by 6% and 9%, respectively (Table 5).

Q2 and W2. Long rollouts and training curves. Revised Figures 12 and 13 show training and validation curves; Figure 10 shows per-step rollout errors. Each seed uses the same selected model at all horizons. TD has lower mean error over steps 101–200 on four of six 1D PDEs and all six 2D PDEs, with small gains on 2D dispersion and KS (3.0% and 2.0%; revised Table 21).

The curves show means and standard errors across 20 seeds in 1D and five in 2D.

These APEBench experiments use Cross pairing. Table 2 in the submission compares alternative pairings only for one-step prediction.

Q3 and W2. Training cost, search budget, and random seeds. The original Table 1 experiments trained both methods for 100 epochs with a single seed. SL searched two learning rates per domain; TD searched the same rates with two discounts and three EMA coefficients (App. E.5).

The new 1D and 2D experiments use the same training-time budget per run and 32 randomly sampled hyperparameter configurations per PDE for each method, with 20 paired seeds in 1D and five in 2D. TD has lower mean error on all six PDEs in both 1D and 2D, with average reductions of 15.8% and 10.9% (revised Table 20).

Revised Figure 11 reports validation performance versus search budget. Due to the limited computation resources, we did not repeat the original-scale experiments with multiple seeds.

Thank you for the detailed rebuttal and additional experiments. The new experiments on computational cost addressed my concern.According to the author's experiment, TD does have a clear overall improvement compared to SL under the same budget.However I am not convinced on the other two issues. First, can you explain the conditions under which the Cross pairing is effective? It is said that most domains have strong shared residuals, so there is a motivation. I wish the authors could give a stronger theoretical analysis. Second, the authors added evidence for long-horizon rollouts, but only include a comparison between default pairing and SL.While I understand the computational constraints,the evidecne may not establish whether the cross pairing is better than other pairings.Therefore I maintain my score of 2.

Reviewer_xCaBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Virtual-Transition Temporal-Difference (TD) Learning, a training strategy for improving the predictive performance of autoregressive neural PDE models. Building on the Markov reward process (MRP) formulation of supervised learning introduced in [1], the proposed approach constructs virtual transitions between training examples and uses the prediction residual of a paired example to modify the current training target. This allows the model to exploit statistical relationships between prediction errors during training. The authors provide theoretical analyses of the statistical efficiency of the proposed TD estimator, establishing bounds on its covariance relative to oracle generalized least squares (GLS) under approximate inverse-covariance matching. The analysis is further extended to finite-horizon autoregressive rollout errors and locally nonlinear models. The authors evaluate the proposed method across 18 PDE domains and several neural PDE solver architectures, demonstrating generally improved one-step and autoregressive rollout accuracy compared with conventional supervised learning.

Reference

  1. An MRP Formulation for Supervised Learning: Generalized Temporal Difference Learning Models, Pan et al., JAIR 2025
Critical strengths
  • The paper proposes a relatively simple, plug-and-play approach to improving the predictive performance of neural PDE solvers trained with regression losses. Since the method modifies only the training objective, it can be incorporated into existing architectures without introducing additional inference overhead.
  • The authors provide theoretical analyses connecting TD learning to statistical efficiency and autoregressive rollout error. These results are supported by controlled experiments involving linear and nonlinear models, as well as empirical evaluations across 18 PDE domains.
Critical weaknesses
  • While I appreciate the extensive theoretical analyses presented in Sec. 4 and App. B, the strongest theoretical guarantees are established for linear models under specific assumptions about target-error covariance. Although the authors also provide a local nonlinear extension, this analysis relies on additional assumptions, including local identifiability and small perturbations around a correctly specified reference model. It remains unclear to what extent these assumptions hold for practical neural PDE solvers.
  • The introduction discusses several approaches to improving long-horizon dynamics prediction, including rollout-aware training, architectural stabilization, and distribution-level regularization. However, the main experimental comparisons focus on supervised learning. Although Sec. 5.5 includes comparisons involving multistep supervised training, the paper does not directly compare against several other approaches discussed in the introduction, such as PDE-Refiner and DySLIM. As I understand, limitations of these prior work and why the proposed approach can be better suited for dynamics modeling are not discussed in depth. That said, additional comparisons with these methods would help establish the practical effectiveness of the proposed approach beyond improvements over standard supervised learning.
  • Sec. 4 and Sec. 5 are somewhat difficult to follow, particularly for readers who are not familiar with TD learning and statistical efficiency analysis. Sec. 4 would benefit from a more intuitive explanation of the main theoretical insights and their relevance to PDE dynamics prediction before presenting the formal results. Similarly, Sec. 5 could more clearly connect each experiment to the theoretical claims it aims to validate. For example, Figure 1(a) examines covariance bounds under different mismatch levels, but the meaning of the covariance mismatch parameter δ\delta and its construction are not explicitly explained in the main text.
Questions for authors
  • In Sec. 5.3, the authors report that TD training incurs approximately 1.37–1.46 times the per-epoch training cost of standard supervised learning, while the main experiments compare the two approaches using the same number of training epochs. Could the authors provide comparisons under matched wall-clock training time or total computational budgets?
  • While the proposed method generally improves predictive performance in the Flower-Micro experiments summarized in Fig. 3, there are notable cases where TD learning performs worse than standard supervised learning. For example, Gray–Scott exhibits a 34.7% increase in VRMSE over rollout steps 21–60, while Rayleigh–Bénard exhibits a 13.0% increase over steps 1–20. Could the authors elaborate on why the proposed method struggles in these settings?
  • According to the caption of Tab. 1, model configurations are selected separately for one-step prediction and different rollout intervals. App. E.6 (and accordingly Tab. 14) further confirms that the selected hyperparameters can differ substantially across evaluation horizons. While this evaluation protocol is applied to both TD and supervised learning, it remains unclear whether the reported improvements can be achieved consistently using a single model configuration. Could the authors report one-step and rollout performance using the same validation-selected checkpoint and hyperparameters across all evaluation horizons?
  • Beyond the controlled linear and nonlinear experiments, could the authors provide further empirical analysis of the conditions under which virtual-transition TD learning is expected to improve performance on practical PDE datasets? My understanding is that the proposed method benefits from statistical dependencies between the prediction errors of paired training examples. Although App. B.6 and Fig. 8 provide a diagnostic analysis of residual structure, it remains unclear how directly these measurements explain the observed performance improvements or degradations across domains. Would it be possible to conduct additional stress tests on PDE benchmarks by introducing controlled perturbations with different correlation structures, or by varying the diversity of training trajectories in terms of their initial and boundary conditions? Such experiments could help clarify when cross-trajectory residual coupling is beneficial and when it may become less effective.
3 comments
AuthorsOpenReview ↗

Response to Reviewer xCaB (1/2): Q1–Q3

Thank you for your questions. Our new equal-time 1D and 2D experiments are summarized above, with full results in Appendix F of the revised manuscript.

Q1. Matched training time. The new experiments summarized in the general comment match both training-time and hyperparameter-search budgets between SL and TD. TD retains lower mean test error on all six PDEs in both 1D and 2D.

Q2. Gray–Scott and Rayleigh–Bénard.

While TD underperforms SL over the reported rollout intervals on these two datasets, it still delivers broad improvements across the benchmark: one-step and short-horizon accuracy improve on 15 of 18 domains, and accuracy over steps 21–60 improves on 13 of 15 domains. Our additional experiments strengthen this evidence under matched training-time and hyperparameter-search budgets. Across six PDEs in both 1D and 2D, TD achieves lower mean test error on every task, with average reductions of 15.8% and 10.9%, respectively. These gains are supported by repeated runs with 20 seeds in 1D and 5 seeds in 2D: TD outperforms SL in 108 of 120 paired PDE–seed comparisons in 1D and 25 of 30 in 2D (Table 20). These results demonstrate that the improvements extend across PDE settings and random seeds.

Q3. One selected model across horizons.

In the new 1D and 2D experiments, we select one hyperparameter configuration per method and PDE and one checkpoint per seed using validation data. The same selected models are used for all reported rollout evaluation without horizon-specific selection or tuning. TD achieves lower mean test error over steps 1–20 and 21–60 on all six PDEs in each dimension (Table 21).

AuthorsOpenReview ↗

Response to Reviewer xCaB (2/2): Q4 and W1–W3

Q4. Conditions under which TD helps. Our theory suggests that TD can improve prediction accuracy when the errors in training targets are correlated across samples. With our default pairing rule, TD uses the prediction error from another trajectory at the same time step to adjust the current training target. This reduces the influence of errors that are similar across many trajectories relative to errors that differ between trajectories. Our theory shows that this can lead to more accurate estimates of the underlying dynamics when the correlations between target errors satisfy certain conditions.

We investigated whether such an error pattern—prediction errors shared across different trajectories at the same time step—also appears in practical PDE datasets (App. B.6). Specifically, at each time step, we first average the prediction errors across trajectories and then measure how much each trajectory's error differs from this average. We compare the strength of these two error patterns across time. In 15 of 16 eligible domains, the error pattern common to different trajectories is stronger than the differences between individual trajectories. This is consistent with our motivation for coupling errors across trajectories, although it is only a suggestive diagnostic in practical deep-learning-based PDE models.

W1. Assumptions of the nonlinear analysis. Our nonlinear theory extends the statistical analysis beyond linear predictors. It examines how accurately the model parameters can be estimated from noisy training data, and how errors in these estimated parameters affect predictions over multiple time steps (Theorem B.3 and Corollary B.4). Importantly, TD does not need to perfectly capture every correlation between target errors to achieve the same statistical efficiency as the theoretically optimal estimator (GLS). In other words, TD can perform as well as GLS even if it does not account for every correlation between training errors, as long as those differences do not affect how the model parameters are estimated when the target noise is small.

Although these theoretical guarantees apply under specific conditions, Fig. 2 directly verifies their predictions in a controlled nonlinear experiment. Furthermore, TD improves short-horizon rollout accuracy on 15 of 18 PDE domains and achieves lower test error on all six PDEs in our new equal-time UNet study (15.8% ± 2.6% average reduction). These results demonstrate the practical benefits of TD for nonlinear dynamics prediction, complementing our controlled theoretical verification.

W2. Comparisons with PDE-Refiner and DySLIM. PDE-Refiner improves predictions through iterative denoising refinement, while DySLIM adds invariant-measure matching regularization to improve long-term statistical fidelity. TD instead couples residuals across training examples while preserving the prediction architecture and inference procedure. Our comparisons with matched SL training isolate the benefit of this residual coupling across several backbones, and our multistep experiments demonstrate compatibility with rollout-aware training (Sec. 5.5). These results motivate combining TD with other approaches; direct comparisons with PDE-Refiner and DySLIM remain outside the current study.

W3. Presentation. We have added an intuitive introduction before the formal results in Sec. 4 and clarified the noise construction and interpretation of Figs. 1 and 2 in Sec. 5.1. Appendices C.3 and C.4 now explain the noise variance, the distinction between the covariance bound and exact rollout error, and the α² normalization.

Dear authors,

Thank you for your thorough response to my initial review.

After carefully reviewing your rebuttal, I have decided to maintain my original rating, due to my remaining concerns about the practical significance of the proposed method:

  • While I appreciate the matched-time results summarized in Table 20, the differences in the reported test nRMSE (which I assume stands for "normalized RMSE") do not appear to be substantial. Specifically, even after normalization, the paired differences across the 1D and 2D benchmarks are on the order of 10−310^{-3}, raising questions about the practical benefits of the proposed approach.
  • The per-step rollout error curves for TD and SL in Fig. 10 nearly overlap across most benchmarks, with Burgers, Dispersion, and KdV being among the few exceptions.
  • In accordance with my concerns above regarding the empirical results, I remain unconvinced that the theoretical analysis presented throughout the paper translates into meaningful improvements in practical PDE solving.

I apologize for my delayed response. Nevertheless, I remain open to further discussion and would appreciate any clarification from the authors if I have misunderstood or overlooked any important aspects of the results.

Reviewer_Wgh4OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a Temporal-Difference learning scheme borrowed from Reinforcement learning to stabilize longer rollouts of (neural) surrogates.

Critical strengths
  • possibly the mathematical derivations, however I have to admit I lack the background to fully judge the novelty and correctness of the derivations
Critical weaknesses
  • I found the paper very hard to read. First, it does not clearly state in the abstract which problem it tries to solve. The clearest paragraph about this is the second paragraph in the introduction (line 43-46), however it's still quite indirect "Existing approaches improve long-horizon dynamics prediction through...". Second, the paper generally does not abstract the important ideas well to make it easy to follow. E.g. it uses metrics in y-axes of plots that are not explained (e.g. Fig 1 and 2).

  • the experiments are weak. The main baseline is just naive supervised learning which is known to be unstable for longer rollouts. The paper should compare to at least 1-2 prior methods for regularization (e.g. the mentioned pushforward trick). A simple baseline can also be just adding noise to the previous state similar to [1].

  • the metrics are not explained. As for long rollouts, the states are prone to drift from a ground truth trajectory, a distributional metric shall be included (e.g. mean and variance over time)

[1]: Kohl, Georg, Li-Wei Chen, and Nils Thuerey. "Benchmarking autoregressive conditional diffusion models for turbulent flow simulation." Neural Networks (2026): 108641

Questions for authors

None

Comments on the paper

1 comment
AuthorsOpenReview ↗

Additional experiments: matched training budgets and fixed model selection

We added 1D and 2D APEBench experiments using UNets on six PDEs: advection, Burgers, diffusion, dispersion, KdV, and KS. SL and TD receive the same training-time and random-search budgets. TD achieves lower mean test error on all six PDEs in each dimension (Table 20 of the revised manuscript).

Summary of the new experiments

Setting or result1D2D
Training-time budget400 seconds per group of 20 seeds1200 seconds per seed
Seeds per configuration205
Random configurations per method and PDE3232
Average test-error reduction (mean ± SE)15.8% ± 2.6%10.9% ± 3.5%
Paired comparisons with lower TD error108 / 12025 / 30

We compare SL and TD using the same UNet architecture and datasets on six PDEs in both 1D and 2D. All experiments run on NVIDIA L40S GPUs. The methods use matched training-time budgets and 32 random-search configurations per PDE, with paired seeds sharing initial weights and minibatch order. We use 20 seeds in 1D and 5 seeds in 2D.

For each method and PDE, we select one configuration and one checkpoint per seed using 100-step validation rollouts, then evaluate the same models for all reported comparisons without further tuning. TD achieves lower mean test nRMSE over steps 1–100 on all six PDEs in each dimension, with average reductions of 15.8% in 1D and 10.9% in 2D. Appendix F provides the full protocol and results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.