- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a training approach for improving autoregressive neural PDE surrogates. For a given training sample, it considers samples at the same time from different trajectories and uses residuals from an EMA target network to construct a coupled residual. The analysis shows that, under the stated conditions, when the induced weighting approximates the inverse covariance of the target errors, the covariance of the TD estimator approaches that of GLS, yielding better statistical efficiency. The paper also presents experiments demonstrating gains over ordinary supervised training (SL) across multiple PDE domains.
Critical strengths
- The paper presents a relatively novel approach that uses residuals from different trajectories at the same time to correct training-sample errors, incorporating TD into the training of neural PDE surrogates. This has the potential to improve predictive accuracy and statistical efficiency.
- The experiments have broad coverage and are reasonably convincing. The paper supports its claims through small-model experiments across 18 PDE domains and original-scale model experiments on three PDE domains, obtaining good results.
Critical weaknesses
- The theoretical analysis does not explain the gains from the chosen default pairing in practical neural PDE surrogate training. The relevant conclusions depend on the covariance structure of the target errors, but the default pairing does not automatically ensure such a covariance structure during actual training. The experimental results also include cases that require further explanation, such as Acoustic Maze in Table 1.
- Table 2 reports only one-step prediction errors. However, one-step errors do not adequately represent the model's performance in long-horizon rollouts. And on some datasets, the performance gap between SL and TD is small, whereas the reported increase in training time on a particular dataset reaches 37%–46%.
Questions for authors
- The theoretical gains depend on the covariance structure of the target errors. How should actual training residuals be related to these target errors? Could you clarify under what conditions the default pairing is effective and explain the rollout degradation on Acoustic Maze in Table 1?
- Table 2 reports only one-step prediction errors. Could you compare the default pairing with strong alternatives on long-horizon rollouts at the original model scale? For a representative task, could you also provide SL/TD training and validation curves, and rollout error curves for a fixed configuration and checkpoint, to clarify where the long-horizon gains come from?
- The scaling experiments report a 37%–46% increase in training time per epoch for TD, while its accuracy gains are small on some tasks. Could you report the training time and hyperparameter-search cost for the settings in Table 1, and compare accuracy under equal training-time budgets? For settings with small gains, could you also provide results across multiple random seeds to establish whether the improvements are consistent?
2 comments
Thank you for the detailed rebuttal and additional experiments. The new experiments on computational cost addressed my concern.According to the author's experiment, TD does have a clear overall improvement compared to SL under the same budget.However I am not convinced on the other two issues. First, can you explain the conditions under which the Cross pairing is effective? It is said that most domains have strong shared residuals, so there is a motivation. I wish the authors could give a stronger theoretical analysis. Second, the authors added evidence for long-horizon rollouts, but only include a comparison between default pairing and SL.While I understand the computational constraints,the evidecne may not establish whether the cross pairing is better than other pairings.Therefore I maintain my score of 2.
Response to Reviewer RUft
Thank you for your questions. We added equal-time, multi-seed experiments and rollout curves. The summary above and Appendix F of the revised manuscript report the results.
Q1 and W1. Target errors, training residuals, and Acoustic Maze. Training residuals combine errors in the training targets with prediction errors from the fitted model. Appendix A.4 considers a specific target-error model: errors have equal variance and equal pairwise correlation across trajectories at the same time step, and are uncorrelated across time steps. Under this model, Cross pairing with a matching discount recovers GLS. Theorem 4.2 quantifies the effect of a mismatch between TD’s weighting and the inverse target-error covariance.
App. B.6 finds a stronger shared residual component in 15 of 16 eligible domains, implying a strong residual correlation across trajectories at the same time step. This supports the motivation for Cross pairing.
On Acoustic Maze, TD and SL have similar one-step error. At the original scale, TD increases rollout error by approximately 6% over steps 1–20 and 11% over steps 21–60 (Table 1). At the reduced scale, TD lowers error by 6% and 9%, respectively (Table 5).
Q2 and W2. Long rollouts and training curves. Revised Figures 12 and 13 show training and validation curves; Figure 10 shows per-step rollout errors. Each seed uses the same selected model at all horizons. TD has lower mean error over steps 101–200 on four of six 1D PDEs and all six 2D PDEs, with small gains on 2D dispersion and KS (3.0% and 2.0%; revised Table 21).
The curves show means and standard errors across 20 seeds in 1D and five in 2D.
These APEBench experiments use Cross pairing. Table 2 in the submission compares alternative pairings only for one-step prediction.
Q3 and W2. Training cost, search budget, and random seeds. The original Table 1 experiments trained both methods for 100 epochs with a single seed. SL searched two learning rates per domain; TD searched the same rates with two discounts and three EMA coefficients (App. E.5).
The new 1D and 2D experiments use the same training-time budget per run and 32 randomly sampled hyperparameter configurations per PDE for each method, with 20 paired seeds in 1D and five in 2D. TD has lower mean error on all six PDEs in both 1D and 2D, with average reductions of 15.8% and 10.9% (revised Table 20).
Revised Figure 11 reports validation performance versus search budget. Due to the limited computation resources, we did not repeat the original-scale experiments with multiple seeds.