acceptodds
Under review as a conference paper at ICLR 2027

When Are Physics Gradients Corrective? Update-Space Dependence in Autoregressive Neural PDE Operators

Abstract

Autoregressive neural PDE predictors can remain locally accurate under a physical-coefficient shift while their long rollouts deteriorate. Governing-equation residuals offer label-free test-time objectives, but a small residual does not establish that its gradient will correct predictive error, nor does a corrective direction guarantee that an optimizer can use it. We separate these questions in controlled viscous Burgers and forced Navier–Stokes systems. Our central experiment holds real update-space dimension fixed. In two Fourier-operator training regimes, architecture-defined pointwise ((d=648)) and output-head ((d=1,249)) spaces have larger mean horizon-100 corrections than ten frozen random spaces at the same size. A two-way model-seed/initial-condition bootstrap supports the pointwise effect under both one- and five-step protocols; the head effects remain positive in mean but statistically inconclusive. A structured (d=4,096) counterexample rules out a blanket preference for structure. Complementary interventions separate scalar residual value from gradient direction: pooled residual–error association is near zero but heterogeneous across seeds, while equal-norm physics updates rank above matched random motion and reversal in aggregate. Rank-four adapters can nevertheless have high alignment and almost no rollout gain. A well-fit non-spectral DilResNet also exhibits positive alignment, directional specificity, and a 12.10% mean physics-only adaptation gain. Correctability is therefore a joint property of the objective and accessible update space, not residual magnitude, cosine, or dimension alone.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
neural operators, physics-informed machine learning, test-time adaptation, update subspaces, PDE surrogate modeling, autoregressive prediction, Navier–Stokes equations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_f8tbOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks when a physics-residual gradient actually improves the long rollout of a neural PDE predictor. It distinguishes the residual value, agreement with an analysis-only supervised gradient, and improvement after adaptation. Its main experiment compares update spaces of equal dimension, with additional direction controls and a non-spectral architecture check. The contribution is an empirical account of correctability, rather than a new adaptation algorithm.

Critical strengths
  1. The equal-dimension comparison addresses a real confound. A small adapter and a large trainable block differ in too many ways to isolate the role of the update space. Here, random-space controls and a norm-matched single step make the comparison much more informative. The fact that the finding persists without Adam is a useful part of the evidence.

  2. The paper is careful about its negative results. It distinguishes supported pointwise-space effects from inconclusive head-space effects, includes a poor structured-space example, and acknowledges that oracle diagnostics cannot be used at deployment. I found this restraint helpful: the paper gives reasons to reconsider simple adaptation rules without claiming that one parameter subset is always preferable.

Critical weaknesses
  1. The main result has limited independent replication. The central matched-dimension study is concentrated in one controlled system and two training regimes of the same spectral architecture. The non-spectral experiment shows that physics adaptation can help, but does not repeat that central comparison. There are also few independently trained seeds, and the low-viscosity reference has a documented resolution discrepancy. These limitations do not invalidate the reported contrast, but they leave uncertainty about how broadly the preferred update spaces will transfer.

  2. Some definitions needed to audit the experiment are inconsistent. Section 3 defines a gradient and motivates a descent step, then describes the beneficial intervention with the opposite sign notation. Appendix C also uses apparently different conventions for real and complex parameter counts. The exact weak residual is only partly specified in the PDF. Since the claim depends on direction, coordinate dimension, and objective choice, I would want these points settled before treating the experiment as fully reproducible.

Questions for authors
  1. Does “+physics” mean the descent direction? Please state the actual displacement used and confirm that the reversal differs only in sign.
  2. How should the last-block and parent-space parameter counts be reconciled? Which reported dimensions expand complex entries into real coordinates?
  3. Can the weak-residual objective be specified completely, including test functions, normalization, and the consistency anchor?
  4. What are the seed-specific paired effects for the main architectural-versus-random comparison? These would help assess independent consistency beyond the pooled state count.
  5. Do existing runs separate low- and high-viscosity shifts or vary the residual basis? If so, does the main ordering survive those choices?
2 comments

Thank you for the careful review and for pointing out these issues. We went back to the existing raw results to verify the numerical details and provide the following clarifications.

Question 1 and Weakness 2 (sign convention). This is a notation error in Section 3. The applied update is the descent step Δθ = −η∇L_phys. The reversal is +η∇L_phys with the same norm, so the two differ only in sign. The random and orthogonal controls use the same norm. We will correct the text and the figure captions.

Question 2 and Weakness 2 (parameter counts). Appendix C mixes two counting conventions. 166,536 (last block) and 667,513 (full model) count each complex weight once. All matched-dimension experiments instead use real coordinates, where each complex weight counts as 2. The parent is 2×165,888 (spectral) + 648 (pointwise) + 1,249 (head) = 333,673, and every reported d is a real dimension. We will label both conventions explicitly.

Question 3 and Weakness 2 (weak residual). L_weak is the trapezoidal vorticity residual on a 10-step predicted rollout (Δt = 0.05, known forcing), projected onto 13 fixed Fourier modes k ∈ {(1,0),(0,1),(1,1),(1,−1),(2,0),(0,2),(2,1),(1,2),(2,2),(3,0),(0,3),(4,0),(0,4)}. Its value is the mean squared coefficient magnitude (forward-normalized FFT) over modes and steps, divided by the forcing mean-square (0.005). The anchor is the mean squared, std-normalized deviation from the frozen model's rollout, with weight 1. TTA minimizes L_weak + L_anchor. The one-step test uses ∇L_weak only. We will add this specification to Appendix E.

Question 4 (per-seed effects). Structured-minus-random gain in percentage points, shown as seeds 0/1/2 (60 paired states per seed):

FNO 1-stepFNO 5-stepHERO 1-stepHERO 5-step
pointwise, d=648.31/.37/.302.09/2.32/2.30.97/.64/.88.58/.49/.49
head, d=1,249.51/.42/.423.71/2.39/2.731.01/.01/.88.52/.56/.53

All 24 seed-level means are positive, including those for the head comparisons whose clustered intervals include zero.

Question 5 (shift direction and residual basis). The ordering holds in all 16 cells when low- and high-viscosity shifts are analyzed separately. This includes the high-viscosity shifts, where the N=64 vs. N=128 reference discrepancy is below 0.1%. We did not vary the residual basis in this experiment and will state this as a limitation.

Weakness 1 (scope of replication). We agree that the matched-dimension result covers one system and one spectral architecture under two training regimes with three seeds, and we will state this scope explicitly. We present the DilResNet result only as an existence result.

We hope these clarifications address the reviewer's concerns and appreciate the opportunity to clarify the experimental setup and scope of our findings.

Reviewer_f8tbOpenReview ↗

Official Comment by Reviewer f8tb

Thanks to the author for the refutation. After reading the rebuttal carefully, I will raise the score.

Reviewer_Crr9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies physics-based gradient-corrective methods to improve long-term prediction of neural operators. It is known that neural operators can develop large errors in long-range predictions as the errors build up. Physics-based test time adaptation is generally used to correct the model's prediction. The paper further studies if physics based residual error, gradient direction, or update space lead to better predictions. The paper does not propose a new algorithm but explores why existing physics-based adaptation method succeed or fail.

Critical strengths
  • The study has good experimental controls making the comparisons useful.
  • The questions are clearly separated in terms of study and experimentation.
  • The study includes negative results
Critical weaknesses
  • Limited generalization for a benchmark study
  • Only three model seeds. What is the statistical power of the results?
  • No reliable automatic decision rule in the end.
Questions for authors
  • It is shown that the 648 parameter architectural space performs better than random space of equal dimensions. Can you identify where this advantage comes from?
  • Can you predict when physics-based adaptation will help without ground truth labels?
2 comments

Thank you for the review. New numbers are from our existing raw results.

Question 1 (why d=648 wins). The advantage comes from gradient magnitude, not direction:

  • Similar direction: physics–oracle cosine is 0.73 for the pointwise space vs 0.62–0.72 for random spaces (FNO).
  • Much larger magnitude: the pointwise layer holds 0.44 (FNO) and 0.37 (HERO) of the oracle-gradient norm, vs 0.02–0.04 for random spaces of the same size.
  • Predicts gain: across all 87 subspaces, this share predicts gain after controlling for dimension (partial Spearman 0.73–0.91; weaker for HERO one-step, 0.12).

Why this layer concentrates the gradient remains open. We will add this to Section 7.

Question 2 and Weakness 3 (label-free rule). No. A label-free gate (Appendix G) reached ROC-AUC 0.641 but gain Spearman 0.009, and it did worse than always adapting. Our results support choosing the update space and step size on in-range validation data, then adapting without labels.

Weakness 2 (three seeds). Agreed: three seeds cannot support population claims, and a seed-level sign test cannot reach p < 0.05 (the minimum is 0.125). We therefore use a two-way seed/initial-condition cluster bootstrap, under which the d=648 effect excludes zero for both models and both protocols. All 24 seed-level structured-minus-random means are also positive. We will state this limitation.

Weakness 1 (generalization). This is a controlled mechanism study, not a benchmark: two PDE families, three trained models, and documented failure cases. We will state that the matched-dimension result covers one system and one spectral architecture.

Thanks to the authors for their rebuttal. I consider the generalization and multiple run limitations to be significant and will therefore maintain my scores.

Reviewer_r1uUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper is a controlled mechanism study asking two questions about physics-based test-time adaptation for autoregressive neural PDE surrogates:

  • when does a label-free physics residual gradient agree with the unavailable supervised correction direction;
  • whether the update space (which parameter subspace is adapted) determines whether that correction is actually transferred.

The authors check this using periodic 1D viscous Burgers and a controlled periodic 2D forced Navier–Stokes vorticity system, with FNO, a HERO-compatible FNO training regime, and a non-spectral DilResNet.

Critical strengths
  1. The equal-norm interventional design cleanly separates direction from accessible magnitude.
  2. The overall study is very rigorous with high quality reporting.
Critical weaknesses
  1. While I think that the authors did significant efforts not to overstate and prove their statement, I feel that the main novelty as stated "controlled mechanism study", while being interesting, does not contribute strongly to the field. I suggest authors to reframe the paper with more sound novelty justification.

  2. The paper establishes that update-space choice matters, but not how to choose. The first-order Taylor argument is local. Nothing in the analysis yields a quantity computable without labels that forecasts whether the pointwise space will help on a new PDE, architecture, or observation condition.

  3. "Treat update-space choice and step size as architecture-specific hyperparameters calibrated on in-range validation trajectories" is an good advice, however, it is not a new capability, and it does not change what a practitioner can do that they could not already do by sweeping adaptation hyperparameters on held-in data.

Questions for authors
  1. The paper establishes that update-space choice matters but derives no label-free criterion. Is there any signal in the reported measurements that could be turned into a computable pre-deployment predictor, even imperfect?

  2. How sensitive is the 12.10% gain to the interpolation-selected learning rate and anchor weight, and would a different equally defensible selection rule change which protocol appears best?

2 comments

Thank you for the careful review. New numbers are from our existing raw results.

Question 1 and Weakness 2 (label-free criterion). Not reliably. We report this as a negative result (Section 9, Appendix G):

  • Imperfect signal: a frozen five-feature label-free diagnostic, trained on interpolation data, detects whether adaptation helps with ROC-AUC 0.641.
  • No ranking of gains: its score–gain Spearman is 0.009, and gating with it gave a higher pooled error than always adapting (0.6401 vs 0.6110). A weak-basis consensus score had gain Spearman −0.088, and the residual magnitude is not a stable ranking rule across seeds.
  • The explanatory quantities need labels: the oracle-gradient share and Q explain update-space differences but require labels, and they are not sufficient on their own (the full network has share 1 but a smaller gain).

We agree that the Taylor argument is local and that the paper does not provide a transferable label-free criterion.

Question 2 (sensitivity of 12.10%). The learning rate was chosen from {1e-6, 3e-6, 1e-5, 3e-5} by minimum mean interpolation error. The anchor weight was fixed at 1, not tuned. OOD results exist only at the selected rate, so OOD sensitivity is not yet measured. On interpolation data, sensitivity is high: at 1e-5 the full-model gain is −27%. The selection rule also matters. The three update spaces (full, last block, last block + head) differ by <1% in interpolation error, and ranking by mean relative gain would instead select last-block at 1e-6. So 12.10% is the result of one interpolation-only selection rule, not evidence that full-model adaptation is best. We will state this selection sensitivity explicitly.

Weaknesses 1 and 3 (novelty, practical value). We agree that calibrating on held-in data is not new. The contribution is showing what to calibrate:

  • At fixed dimension, which parameters are adapted changes gain by >10× (2.40% vs 0.16% at d=648).
  • Common defaults underperform: a rank-4 adapter gains 0.12% and the full network 3.74%, vs 8.43% for the last block (FNO).
  • The natural selection signals are insufficient: the adapter has a high gradient cosine (0.695) yet almost no gain, and the residual magnitude does not rank errors.

We will reframe the contributions around these points.

I want to thank the authors for the responses. After careful reading of their answers and other reviewers' posts, I will maintain the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.