acceptodds
Under review as a conference paper at ICLR 2027

PAIR: Prediction with Additive Intermediate Readouts for Long-Horizon PDE Forecasting

Abstract

Long-horizon accuracy in autoregressive PDE forecasting depends on both prediction errors and their propagation through the dynamics. Multiscale architectures construct coarse-grid features, but these representations typically contribute to the predicted physical field implicitly through the remaining decoder. In our periodic Navier–Stokes simulations, we observed that low-frequency initial perturbations produce larger deviations for long-horizon responses than equal-norm high-frequency perturbations. Motivated by this scale dependence, we investigate whether giving intermediate features an explicit coarse-scale prediction role can improve both coarse-scale learning and the full-resolution predictor. To this end, we introduce **Prediction with Additive Intermediate Readouts (PAIR)**, which maps intermediate features to a coarse physical-field contribution, upsamples it with a fixed operator, and adds it to the backbone’s full-resolution prediction. A unified prediction loss on the combined output jointly trains both paths and their shared features, providing an additional route for the prediction loss to reach intermediate representations, allowing the coarse readout and the full-resolution path to adapt together without a separate coarse-scale supervision objective. Across periodic Navier–Stokes flow, reaction–diffusion, and heterogeneous acoustic scattering, PAIR reduces mean endpoint error by 72.8%, 87.5%, and 72.6%, respectively, relative to the corresponding U-Net baselines after 99–100 autoregressive steps. Further experiments show that joint training improves the full-resolution path’s predictions in a high-frequency band outside the coarse readout’s spectral support, suggesting our paired design decomposing the dynamics offer a better control of error propagation for long-term rollout.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Scientific machine learning, PDE surrogate modeling, Long-horizon forecasting, Autoregressive prediction, Multiscale learning, Additive intermediate readouts

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yC43OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposed a new method for long-horizon prediction on physical systems where low-frequence presents a strong signal in prediction than high-frequency signals. This frequence-dependent behaviors then inspired a simple design of making use of intermmediate readouts from the neural operator to correct low-frequency predictions in the final readout. The idea is simple. On the tested examples, its improvements over long horizon are significant.

Critical strengths

The proposed research can be seen as effort in multi-scale modeling in neural operators. Also, long-horizon prediction has been particularly difficult for prediction on physical systems with strong perturbations. So the importance of the targeted problem has been established and heavily studied.

The propose solution is lightweight. This is an advantage in that it will not add excessive computational workload.

The results are improved greatly in long-horizon prediction.

Critical weaknesses

When positioned within the context of multi-scale modeling in neural operators, it is difficult to see why PAIR is categorically different from existing work. The limitation of existing work seems to be 'At these hidden layers, the features typically have no direct mapping to the physical field and no additional prediction constraint. The final prediction loss acts on them through the remaining decoder. ' But I am afraid that this just points out a difference, not a problem or limitation. In other words, why is it a fundamental limitation such that existing mutl-scale operators cannot predict frequence-dependent signal well? Research like [1-3] explicitly trains models on multi-resolution. Admittedly, some focus on the spatial resolution, but they could natually deal with learning at different frequencies in the examples employed in this paper.

Also, the paper shows the intermediate features are useful. However, which intermediate features? Is this architecture specific? If so, this is a strong limitation on the generality of the proposed method; if not, why are these features selected and based on what criteria? I assume there must be architectural constraints implicitly assumed here, e.g. if it is upsampling operations, the features need to be a super-resolution representation of part of the whole solution space. In any way, it is hard to find the explaination and justificaiton.

Theorem 1 seems to be crucial for the justification of the methodological design and the guidance on later experiments. I have one question here. 'Assume that, over a local interval [0, τ ], K_{base} and K_{coarse} are fixed positive-semidefinite matrices that commute', what architectural or other constraints are assumed for this assumption to hold? Is there any guarantee that K_{base} and K_{coarse} can commute? Theoretically, it is possible construct two postive-semidfinite matrices that do not commute and do not share eigenvectors. I assume those matrices are not covered by Theorem 1. However, this brings the question of the validity of this assumption.

The evaluation is interesting. The choice of baselines is open to debate. Allow me to observe that the baseline methods employed here are not really designed for multi-scale modelling. The U-Net is supposed to be designed for capturing multi-scale features but the employed baseline is an unmodified U-Net which is not designed to deal with functions.

There seems to be a lack of verification of the proposed PAIR. There is a controlled training on coarse and full. However, it is difficult to see why this is not simply because the increase of the model capacity, or specifically attributed to the way the intermediate features are selected and used for corrections.

  1. Multiwavelet-based Operator Learning for Differential Equations
  2. Multi-Grid Tensorized Fourier Neural Operator for HighResolution PDEs
  3. HiLNO: A Hierarchical Latent Neural Operator with Multi-Scale Supervision for PDEs on General Geometries
Questions for authors
  1. What are the criteria of feature selection? Is this architecture specific? If not, is any constraint in selecting the intermediate features?
  2. What guarantees the validity of the assumption in Theorem 1 in 'Assume that, over a local interval [0, τ ], K_{base} and K_{coarse} are fixed positive-semidefinite matrices that commute'? While K_{base} and K_{coarse} are positive-semidienite, how do we know they commute for any or arbitrary neural networks?
  3. Can you employ baseline methods that are specifically designed to handle multi-scaling signals?
  4. How do we know PAIR works not simply because of the increase of the model compacity?
  5. How would you position the research with respect to multi-scale modeling in neural operators?
Reviewer_ejK8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PAIR, a multiscale prediction framework that introduces additive coarse-scale readouts from intermediate features. These contributions are combined with the full-resolution prediction and optimized using a unified loss. Experiments on three PDE benchmarks demonstrate substantial improvements in long-horizon forecasting accuracy.

Critical strengths
  • PAIR achieves substantial reductions in long-horizon prediction errors with a simple architectural modification and limited computational overhead.

  • The gradient-path ablations and frequency-domain analyses provide useful evidence that the intermediate readout influences the learning of the full-resolution prediction branch, rather than merely contributing an additional coarse-scale output.

Critical weaknesses
  • The analysis establishes faster local training-error reduction under a frozen-kernel approximation. However, this property follows from introducing an additional trainable prediction pathway and does not establish reduced error amplification during autoregressive rollout. The theoretical analysis does not directly explain the improvements in long-horizon forecasting.

  • The main comparisons focus on U-Net and FNO variants, without direct comparisons against approaches specifically designed to mitigate autoregressive error accumulation, such as pushforward training and PDE-Refiner. Consequently, the competitive advantage of PAIR for long-horizon forecasting remains unclear.

Questions for authors
  • Does PAIR reduce error amplification during autoregressive rollout, beyond improving one-step prediction accuracy? An analysis of perturbation growth or the Jacobian of the learned time-advancement operator would help distinguish these effects.

  • How does PAIR compare with pushforward training or PDE-Refiner under matched backbone architectures and training budgets? Are the improvements complementary?

1 comment

Thank you for your response. I have carefully read the rebuttal and appreciate your clarifications.

Reviewer_QV3oOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method PAIR (Predictions with a Additive Intermediate Readouts), for improving the accuracy of multiscale neural network architectures by introducing a path for coarse resolution intermediate outputs to provide an alternate route for optimizing jointly at two different scales via an additive prediction criteria. The main contribution of the work is the application of this method to a PDE forecasting tasks long horizon prediction, showing both the improvements the method brings over baseline methods as well as analysis of why the method improves accuracy. They show this by provide experiments for several different types of PDEs where the method improves over baseline architectures.

Overall, this paper shows how additive prediction methods and joint optimization can be applied in a useful way to to add a feedback path for learning to incorporate coarse information information. The experiments provided show the utility of the coarse path provides to improve learning.

One major thing not addressed in the experiments, is the question of how the method would perform on higher and lower resolved PDEs. An experiment showing how the method performs over different resolutions would show in what cases the method would expected to be useful.

For the experiments the following should be added for clarity:

  1. For results in Table 2a, a bit more explanation on the auxiliary supervision should be made to not force the reader to have to look into the appendix for the details.
  2. For results in Table2b, pointing the reader to Appendix C.4 would be useful to show why 64^2 was used as the intermediate readout

Minor comments:

  1. Caption of Figure 4, I don't see any triangles in the plots, should the triangle statement be removed
Critical strengths

The strong set of thorough experiments that exemplify how the method improves accuracy over the baseline.

Critical weaknesses

The paper shows cases where PAIR improves accuracy, but there is no analysis of the effectiveness of the method in general, in terms of what conditions this method will and will not help to improve accuracy.

Questions for authors

For results in Table 2c, naively I was think that removing the gradient path for the coarse resolution, would result in accuracy similar to baseline UNET. Why is that not the case?

How would you expect PAIR to perform in higher and lower resolved PDEs, e.g. 512x52, 64x64?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.