acceptodds
Under review as a conference paper at ICLR 2027

Continuum-Equivalent Neural Operators Can Learn Different Interactions

Abstract

Neural operators should recover the same physical interaction as input resolution increases, yet vanishing weight decay can erase interactions that remain fully expressible in the continuum. Guarantees of continuum expressivity and consistent representation cannot predict this selection: a fixed interaction may require diverging coefficients, and standard parameter penalties charge those coefficients rather than the finite interaction they encode. This mismatch makes architecture and regularization decisive for whether learned continuum models preserve the underlying physics. We establish a realization-cost principle that characterizes consistent recovery through target approximations with vanishing penalized parameter cost. Counterterm-coupled regularization implements this principle in finite Wick coordinates, preserving the intended interaction objective and restoring consistency. We demonstrate the mechanism using cubic responses of two-dimensional Gaussian free fields with a fixed continuum target and smoothing observation, where continuum-equivalent architectures erase or recover the same target under identical vanishing weight decay. For standard ReLU channel networks, the principle yields the sharp threshold , linking penalty strength , field variance at input cutoff , and depth . Exact population analysis, matched architectural controls, and independently verified polynomial-network training demonstrate the predicted separation and its correction at moderate resolutions.

Primary area:
learning theory
Keywords:
physics-informed learning, neural operators, renormalization, continuum limit, cutoff dependence, weight decay, inductive bias, Gaussian free field, Wick polynomials, Wiener chaos, learnability, model selection, certified computation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_JjNqOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper claims to establish a "realization-cost principle" and it characterizes consistent recovery through target approximations with vanishing penalized parameter cost. Also, the authors give implementation of this principle based on Wick polynomial. They also claim to demonstrator the mechanism using cubic responses of two-dimensional Gaussian free fields.

Critical strengths

(S1) The starting point of discussion that neural operators should recover the same physical interaction as input resolution increases is good.

(S2) Related to (S1), when we consider neural simulator such as neural operators for physical phenomena, preserving the underlying physics is significant problem. Thus, the motivation of this paper is understandable, as far as I understand correctly.

Critical weaknesses

(W1) First of all, at least for me, this paper is not well-organized. It is quite difficult for me to understand correctly motivation, contribution, and main claim. As a result, it is almost impossible for me to judge the true contribution of this paper and that this is due to the quality of this paper or my understanding.

(W2) The discussion on the related work and the position of this work is quite limited. As a consequence, it will make difficult readers catch motivation and contribution of this paper.

(W3) The quality of the figures is not so good. To be honest, I cannot understand the meaning of the figures (for example Figure 1). It is better for the authors to reconsider the figures that would be easy to understand for non-expert readers.

Overall, I personally feel like that this paper dose not meet the standards of ICLR paper.

Questions for authors

(Q1) Please explain and clarify the motivation and contribution of this paper. I would like to know the main concept and results of this paper intuitively since my understanding may be not quite well.

(Q2) Please reconsider the organization and each component of each figures which can be understood for broad ML readers. Perhaps, the figures are created by AI?

(Q3) To clarify the background of this work, please discuss the related work more throughly and carefully.

2 comments

Dear Reviewer,

Thank you for your review. First of all, we apologize for the overall presentation of our paper. It was not well written and not very clear. We rewrote it entirely (including the figures) without using AI (except for a few typo fixes, cf. rebuttal to reviewer nUj2) and added a clear AI statement. Now the paper should be easier to understand for non-expert readers. We therefore ask you to reconsider our paper, and we hope that it now meets the ICLR quality bar.

We added back the discussion about related work in the first paragraph. And we enhanced the conclusion and limitations part (last paragraph of the main text). We also made the figures much clearer and easier to understand.

For your first question, we aim to repply to this: As the input gets more finely resolved, does the network keep learning the same thing?

Our findings: Not always, the weights needed to express the right answer can grow in magnitude with the resolution, and weight decay will then penalize large weights. So training may settle on a different answer, even when the weight decay is quite weak. In other words, what the network can express is not the same as what it will learn. We make this precise, and give a fix.

For your third question:

We agree so to clarify the background of this work, to clarify the background of this work, we discuss related literature in more depth. Appendix G contains a complete related-work discussion, and we provide a compact pointer to it in the Introduction. Our contribution centers on the realization cost of an interaction as resolution grows.

We hope we answered to you main concerns about the paper's format. We kindly ask you to reconsider the paper if you think it cross the ICLR bar.

Thank you for your response. I have read the rebuttal and quickly gone through the revisions. Even now, it is hard to change my decision. The quality of the manuscript is not good enough to accept as the ICLR paper. Also, importantly, I believe that the paper with significant revisions is better to get another review cycle.

Reviewer_HCUXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper shows that two networks that can give the same answers can still learn different things when weight decay is turned down.

They claim that this happens because sharper inputs force a hidden helper number to grow, and weight decay punishes it so hard that the network may drop the right answer and learn nothing. They show that what decides the outcome is how costly the answer is to build, and they give a clear rule for when learning works and when it fails. Deeper networks hold up better.

They offer two fixes, charging the real answer instead of the helper number or adding a part that makes the answer cheap, and back them with proofs, exact calculations, and real training runs.

Critical strengths
  • There are best-case results computed directly with no training, and a fair-test control: an add-on of the same size that doesn't lower the cost still fails. That control rules out "more parts" as the reason the fix works.
Critical weaknesses
  • The training was only done on a 129^2 grid, a medium level of detail. The very sharp case, where the effect is supposed to be strongest, is shown only by calculation and never by actual training.

  • For the deep ReLU networks, the authors say they could not fully confirm that training had finished, which weakens the depth claim that is checked against real data.

Questions for authors
  • If you train at finer grids, does the plain network get (steadily) worse as the theory predicts? Does the gap between the plain and fixed networks grow?
  • The penalty fix needs the field var to be known. Does it still work when the var has to be estimated, or on standard NOs trained on real equation data?
1 comment

Dear reviewer,

Thank you for your comprehensive review. Here are our answers for your concerns:

For the weakness 1

The training was only done on a 129^2 grid, a medium level of detail. The very sharp case, where the effect is supposed to be strongest, is shown only by calculation and never by actual training.

We have expanded Section 7.1 to show that every model was trained on 2572257^2 and 5132513^2 grids, alongside the original 1292129^2 grid. Every run achieved within the stated tolerance the best penalized training loss possible. For the extreme-cutoff example, we provide a separate mathematical certificate.

for weakness 2

For the deep ReLU networks, the authors say they could not fully confirm that training had finished, which weakens the depth claim that is checked against real data.

In Appendix F.6, we prove that the best deep loss is below the best shallow loss. The best deep model that we found by training was not assumed to be the best. Two of ten runs now pass the full gradient check, although even those do not establish that training had attained the global optimum.

Question 1

If you train at finer grids, does the plain network get (steadily) worse as the theory predicts? Does the gap between the plain and fixed networks grow?

Yes, for every tested refinement and every random seed, the mean cubic recovery for the plain network decreased (from 0.31304 to 0.24075), while the gap between that and every other network increased, both in terms of recovery and prediction error. We have added this to Appendix E.4, where we show paired comparisons of the plain network against every alternative.

We hope that it reply well to your concerns and questions.

Question 2

The penalty fix needs the field var to be known. Does it still work when the var has to be estimated, or on standard NOs trained on real equation data?

The penalty-fix works when spatial variances are estimated. In Appendix B.2.3 we treat spatial variance estimation for the Gaussian cubic model, proving that recovery is preserved as the penalty vanishes. We tested 16 calibration fields and the largest recovery loss was 0.010430. For full neural operators trained on PDE solution data we have not tested this correction.

Reviewer_qYLFOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how vanishing parameter regularization affects the continuum limits learned by neural operators on singular Gaussian fields. For cubic interactions of a 2D Gaussian free field, the baseline cubic architecture requires a diverging raw linear counterterm to represent a fixed renormalized interaction with a nonzero trace component. Standard squared-parameter regularization can therefore introduce a systematic bias even as its strength tends to zero. Depending on the scaling of λεCε2\lambda_\varepsilon C_\varepsilon^2, the cubic trace component is recovered, attenuated, or removed. To address this issue, the authors propose a counterterm-coupled penalty that preserves the intended regularized objective in Wick coordinates.

The paper also constructs architectures with the same continuum prediction class but different recovery costs. It gives a general consistency criterion based on the vanishing penalized realization cost. The main theoretical extension establishes the sharp recovery-cost scaling Cε3/LC_\varepsilon^{3/L} for scalar-input ReLU networks with LL affine layers, when all weights and biases are penalized. The theoretical results are supported by exact population analysis, controlled polynomial-network experiments, and finite-resolution ReLU comparisons with certified bounds on the objective and representation costs.

Critical strengths

1. A sharp separation between expressivity and regularized recovery

Theorem 6.4 is the strongest contribution. Although the unions over finite widths at different fixed depths have the same scalar piecewise-affine prediction class, the minimum cost of recovering the target scales as Cε3/LC_\varepsilon^{3/L}. The width-independent lower bound rules out insufficient shallow width as the explanation, while the explicit width-four construction establishes the matching upper bound. This gives a precise condition under which prediction-equivalent architectures recover different limiting interactions.

2. A controlled analysis that isolates the proposed mechanism

The polynomial analysis identifies exactly which component of the interaction is suppressed and derives the relevant penalty scale. The continuum-null extension, Wick-quintic placebo, and corrected penalties help separate realization cost from added parameters and expressivity. The numerical study retains trainable neuron factorizations and independently checks empirical optimization gaps. These controls make the claimed mechanism convincing within the constructed setting. The finite-sample result in Appendix B.3 further shows that increasing the number of sampled fields does not remove the structural bias.

Critical weaknesses

1. How far the results extend beyond the setting studied here remains unclear

The paper assumes known Gaussian-field structure and a fixed smoothing observation. The architectures are pointwise polynomial networks or scalar-input ReLU networks. It remains unclear whether the same bias occurs when the network learns the spatial processing or the SPDE solution map itself. Even in the scalar ReLU case, the lower bound for the 3/L3/L exponent relies on penalizing every bias as well as every weight. The general criterion tells us what to check in a new architecture, but the main work is still to determine its realization cost.

2. Recovery under fixed-penalty training

The successful ReLU runs use an unregularized warm-up, followed by penalty continuation and substantial refinement. In the matched control, only one of five runs at 2572257^2 meets the recovery criteria when the final penalty is applied from the start. The comparison with the unrestricted-width shallow optimum shows that continuation can find a deep solution with a lower objective value. How close that solution is to the deep optimum is a separate question. The paper gives no bound on that gap, and the primary deep runs do not meet the registered population-stationarity criterion. I therefore find the evidence for the shallow/deep separation stronger than the evidence that fixed-penalty training reliably reaches the near-minimizers in Theorem 6.4.

Questions for authors

1. Regularization of biases

The lower bound in Theorem 6.4 uses the squared penalty on all biases. If only the weights are penalized, the same construction appears to recover the target at a lower cost. How does the 3/L3/L threshold change in that case?

2. Optimality of the deep solutions

The paper certifies that the deep endpoints achieve a lower objective value than the shallow optimum. Does the submission also bound their gap to the deep global optimum? Also, can the authors tell whether the population-stationarity check failed because of residual optimization error or is it because the numerical bounds were too conservative?

1 comment

(Numbering follows the revised PDF: Theorem 6.4 of the submission is Theorem 5.2, Theorem 6.3 is Theorem 4.1.)

Dear Reviewer,

We would like to thank you for your complete review, here is our replies to you concerns:

If only the weights are penalized, the same construction appears to recover the target at a lower cost. How does the threshold change in that case?

You are right: with free biases (no bias penalty) it is cheaper to recover the target. The cost goes like Cε2/LC_\varepsilon^{2/L} instead of Cε3/LC_\varepsilon^{3/L}. The reason is that the hinges of the ReLU network are now placed by the (free) biases; only the product of the weights grows with ε\varepsilon. The lower bound now follows from the network Lipschitz constant and holds for any width. The threshold becomes λεCε2/L→0\lambda_\varepsilon C_\varepsilon^{2/L}\to0. Depth still separates: our schedule with two layers is critical, while three or more layers can recover the target. We add this in Appendix D.6 and point to it from Section 5.2.

I therefore find the evidence for the shallow/deep separation stronger than the evidence that fixed-penalty training reliably reaches the near-minimizers in Theorem 6.4.

We agree that Theorem 5.2 is about near-minimizers and not about a training procedure. The fixed-penalty control's low performance can be attributed to the fact that four of the five runs collapse to the zero network. We think this happens because, in the presence of any positive penalty, the zero network is a strict local minimum when the depth is at least three. We have now provided a proof of this fact in the new Appendix F.6. We also have added a few sentences in Section 7.2 to explain how the warm-up and then growing penalty schedule steers the training away from this local minimum.

Does the submission also bound their gap to the deep global optimum? Also, can the authors tell whether the population-stationarity check failed because of residual optimization error or is it because the numerical bounds were too conservative?

The first question: yes, we now bound the gap. We have a new lower bound on the deep optimum, valid at every width. The gap between the shallow optimum and the deep optimum is between 0.31 and 0.43 at 65265^2, and between 0.38 and 0.48 at 2572257^2. The bound holds regardless of what training does. The bounds for the individual endpoints are still loose, which we discuss in Appendix F.6. For the second question, the bounds that appear in the paper are still conservative, but they are not related to training. We computed the truncated gradient below 2×10−82\times10^{-8} for all ten endpoints we identified. We also computed bounds on the excess terms beyond degree 511. We performed a computation with degree raised to 2047, and two of the ten endpoints are now certified to have gradient below 10−610^{-6}, and none are certified to have gradient above 10−610^{-6}.

It remains unclear whether the same bias occurs when the network learns the spatial processing or the SPDE solution map itself.

You are right: our sharp results are specific: they apply to pointwise interactions, a known Gaussian field, and fixed smoothing. We have made this explicit in Section 8. The general part of our results is Theorem 4.1, which applies to any model meeting its scaling assumptions. To give you a concrete example, we computed the realisation cost in the "weights-only" case in Appendix D6. Determining the realization cost for new architectures would tell us if a similar bias occurs. We also hope to return to studying learned spatial processing and SPDE solution maps in future work.

Thank you for your review. We hope that eveyrhting is clearer now.

Reviewer_nUj2OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies preserving the continuum expressivity as the input resolution changes in neural operators.

Critical strengths

I cannot assess the strength of this paper.

Critical weaknesses
  • In this day and age, we can pose any question we want, and LLMs can cook up any theorems and proofs. So what is important is whether the question posed is worth answering. There is absolutely no discussion on why we should care about cubic responses of singular stochastic fields through a fixed smoothing observation.
  • The main text says "Appendices E–F supply the numerical evidence" and Appendix F says "the preregistered joint outcome remains UNRESOLVED." This makes me question if the authors ever even read the text in the submitted manuscript.
Questions for authors

N/A

1 comment

Dear Reviewer,

Thank you for taking the time to try to understand our paper. We apologize that the goals and objectives of the study were not clear enough. We did use AI to rewrite a few parts of the manuscript, but it was mainly written by humans. After finishing the manuscript, we reviewed it, but the number of pages exceeded the ICLR page limit, so we decided to shorten some interpretations and discussions that took up a large part of the main text. At the same time, we used AI to shorten the main text, but we reviewed it twice, before and after shortening it. Since we already knew the topic, it was easy for us to understand. We understand that it was hard to follow for a new reader or non-expert, so we have brought back the interpretations and the justifications for each section, a clear conclusion, and a clearer novelty and impact statements to improve the quality.

We rewrote the paper entirely (without changing the results and claims, except for changes made in response to the rebuttals) so it is easier to understand, and we moved unnecessary parts to the appendix. This time, AI was only used to enhance a very few parts to improve clarity, make sentences better, correct typos, or improve logical flow. We have now added a clear AI statement.

And we hope that now it will be better for you to understand the claims. We sincerely apologize for any time wasted. At the same time, we think that the contribution of this paper is valuable to the scientific community, especially people working on physics-informed machine learning, including both physicists and machine learning researchers. We kindly ask you to reconsider again our paper.

If we can summarize the main contribution is two sentences it would be :

Machine learning can fail to learn a target interaction as the input resolution grows, even though the corresponding neural network is capable of expressing it. By introducing realization cost (the size of the weights needed to represent an interaction), we find that, under suitable scaling, a network learns the target if and only if this cost, scaled by the penalty strength, goes to zero.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.