acceptodds
Withdrawn from ICLR 2027

Interaction Degree Predicts When Nonlinear Capacity Improves Neural Operators

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Neural-operator capacity is usually chosen from errors observed after training, although the target function itself determines whether nonlinear interactions are needed. We show that interaction degree, measured from reference-normalized response profiles, predicts when added nonlinear capacity will improve accuracy. Whitening removes invertible affine coordinate effects, and homogeneity carries the profile across input radii. Complete homogeneous regression with independently sampled signed rays recovers synthetic polynomial support, agrees across viscous Burgers regimes, and keeps a matched heat operator at degree one. Across 27 Burgers regimes, interaction degree correlates 0.986-0.998 with the gain from a denser polynomial approximation. Across Burgers and Allen-Cahn, the profile identifies 10 of 12 beneficial settings near the training radius; under larger shifts, the mean gain reverses. Interaction degree thus directs nonlinear modules to regimes where they help and warns when distribution shift makes the same capacity harmful.

Primary area:
learning on time series and dynamical systems
Keywords:
neural operators, nonlinear capacity, interaction degree, operator learning, distribution shift, polynomial approximation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4Vb1OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a diagnostic metric called interaction degree to predict under which problem conditions adding higher-order nonlinear computational capacity to neural operators improves performance. The authors approximate the target operator using polynomials over a specified input range and reference distribution, and quantify the importance of higher-order interactions by measuring the magnitude of components at each degree. They also conditionally analyze the relationship between coordinate invariance of the metric and polynomial approximation error, and compare two estimation methods: direct polynomial regression and directional probing. In the experiments, the authors measure the contribution of higher-order components under various conditions of the Burgers and Allen–Cahn equations and use these measurements to select conditions in which to apply the same additional residual module. Under the selected conditions, the method achieves an average error reduction of approximately 1.2% for distribution shifts close to the training range, but no improvement is observed under larger distribution shifts. Confirmatory experiments on FNOs operating on full spatial fields remain incomplete, so the empirical evidence for practical neural operators is primarily limited to module placement experiments in a restricted coefficient space.

Critical strengths
  1. The paper presents a clear analytical framework for measuring higher-order interactions. It explicitly specifies the input space, reference distribution, and output norm, and defines interaction degree in terms of the magnitude of components at each polynomial degree. By analyzing its invariance under affine coordinate transformations that represent the same problem and its conditional relationship with polynomial approximation error, the authors clarify the interpretation of the metric and the conditions under which it is valid.

  2. The paper validates the metric through independent estimation methods and controlled module placement experiments. The authors compare direct polynomial regression with separately collected directional probing results to assess the consistency of the measurements. They also fix the architecture of the additional module and the number of conditions selected for its application, and evaluate a selection rule established before observing performance outcomes. These experiments provide preliminary empirical evidence that the relative importance of higher-order components can be useful for guiding module placement.

Critical weaknesses
  1. The necessity of measuring interaction degree separately from learning higher-order modules is not sufficiently established. Given the existence of approaches capable of learning higher-order interactions, such as TESLA (Ko et al., 2026) [1], it remains unclear what additional benefits are provided by measuring interaction degree in advance. A separately trained selector based on physical parameters also selects the same conditions, making it difficult to isolate the practical contribution of the proposed measurement procedure. A direct comparison with strategies that learn and validate module placement without prior degree estimation is needed, while matching module capacity, the number of selected conditions, and validation budgets, and accounting for the data collection and regression costs required to estimate interaction degree.

  2. The contribution beyond expected effects of polynomial approximation is not sufficiently validated. It is reasonable to expect that higher-degree polynomial approximations perform better for targets with larger higher-order components. Therefore, a strong correlation between interaction degree and improvements in polynomial approximation performance is insufficient to establish that the metric provides additional information useful for allocating capacity in actual neural networks. The paper itself also shows that the performance gains of a particular module can differ even under the same degree distribution. The central issue is not merely whether interaction degree can be measured, but whether this measurement leads to better decisions than those based on existing information or conventional validation procedures.

  3. The limited scope of validation on actual neural networks leaves practical utility and scalability uncertain. The main module placement experiments focus on low-dimensional coefficient representations of two PDEs. Under the selected conditions, the method achieves an average error reduction of approximately 1.2% for distribution shifts close to the training range, but no improvement is observed under larger distribution shifts. Furthermore, confirmatory experiments on FNOs operating on full spatial fields remain incomplete, making it difficult to assess the effectiveness of the approach for general neural operators. In high-dimensional problems such as image modeling or LLMs, additional challenges arise in defining input transformations and reference distributions, as well as managing measurement costs. Consequently, the applicability supported by the current evidence remains limited.

References

[1] Daehwa Ko, JaeHyeon Kim, SeungHyun Ham, and Jay Hoon Jung. TESLA: Taylor Expansion of Sinusoidal Learnable Activations. AISTATS 2026, PMLR 300:5257–5265.

Questions for authors
  1. Under what conditions is measuring interaction degree separately more advantageous than directly using learnable higher-order modules? Compared with approaches capable of learning higher-order interactions, such as TESLA (Ko et al., 2026), what benefits does prior degree estimation provide in terms of accuracy or computational cost? Can the authors provide a comparison that matches module capacity, the number of selected conditions, and training and validation budgets, while also accounting for the cost of degree estimation?

  2. What additional value does interaction degree measurement provide when a selector based on physical parameters identifies the same conditions? Since the parameter-based selector uses performance labels, the two approaches differ in their information requirements. Considering this distinction, along with the data collection and computational costs required for degree estimation, can the authors quantitatively identify the conditions under which the proposed method offers an advantage?

  3. How can the information contributed by interaction degree to predicting actual module performance gains be isolated? The paper shows that performance gains can differ even under the same degree distribution, depending on the directions within each degree and their alignment with the module. Does interaction degree retain its predictive power when these factors are controlled for or measured? Beyond the strong correlations observed in the polynomial approximation experiments, please clarify what evidence supports this relationship in actual neural networks.

  4. Why do the improvements observed under nearby distribution shifts not extend to larger distribution shifts? Can the authors investigate whether this limitation primarily arises from errors in degree estimation, limitations of polynomial approximation, or difficulties in training and generalizing the additional module? Furthermore, please clarify how measurement costs and estimation stability are expected to change when extending the current low-dimensional coefficient experiments to full spatial fields or higher-dimensional inputs.

Reviewer_VbwSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a reference-normalized interaction-degree profile for studying when additional nonlinear capacity may improve operator approximation. It decomposes a finite-radius polynomial surrogate into homogeneous components and summarizes their realized output norms. The theoretical results establish coordinate invariance under a consistently transformed reference experiment, a radial transport identity, bounds relating tail energy to nested polynomial approximation gain, and a counterexample showing that degree mass alone cannot identify the benefit of a particular augmentation.

The profile is estimated using complete degree-graded regression and independently sampled signed rays. Synthetic experiments test support recovery, while Burgers and matched heat-equation experiments examine estimator agreement and nonlinear structure. Further experiments find strong correlations with gains from extending complete polynomial spaces. A condition-level Burgers/Allen–Cahn allocation study reports a modest benefit from a fixed Waring residual near the reference radius, with loss of benefit under a farther shift. The planned controlled full-field FNO intervention remains incomplete.

Critical strengths
  1. The measurement object is carefully specified and supported by appropriate independent checks. The paper distinguishes finite-radius projections from Taylor coefficients, realized homogeneous functions from their nonunique factorizations, and transporting a fixed surrogate from refitting at a new radius. Complete regression and independently sampled signed rays agree closely on the tested Burgers slices, while the matched heat equation remains degree one. The synthetic noise, sample-size, and rank-misspecification controls further identify where estimation becomes unreliable. Together, these results provide credible evidence that the proposed local descriptor can be measured reproducibly in the settings studied.

  2. The evaluation clearly separates prospective selection, statistical dependence, and unsuccessful outcomes. The allocation score is fixed before the held-out residual outcomes, competitors receive the same selection quota, and uncertainty is computed using complete data/model-replicate blocks. The paper also checks the polynomial correlations at the physical-regime level and explicitly retains the far-shift failure, unmet joint criteria, identical selections by competing policies, and incomplete full-field experiments. This transparency makes the supported local findings substantially easier to assess.

Critical weaknesses
  1. The contribution to selecting nonlinear capacity in trained neural operators remains insufficiently established. The formal results primarily provide transformation identities, projection bounds, and an orientation counterexample; they do not supply an evaluated architecture-specific predictor. The strongest correlations concern extending complete polynomial spaces, whose organization is closely aligned with the degree diagnostic. The independent allocation experiment adds useful evidence, but it uses a four-input Waring coefficient map. Appendix B.3 and Table 4 show that the earlier full-field comparisons did not establish the benefit or necessity of the higher-order structure, and the final controlled FNO comparison never trained its residual or control arms because the backbones did not meet the saturation criterion. The proposed degree-plus-alignment approach therefore remains a direction for future validation. These limitations substantially reduce the architecture-facing significance of the current paper.

  2. The incremental decision value and claimed distribution-shift warning are not demonstrated. On the selected twelve-condition subset, the near-OOD error ratio is 0.98841 [0.98301, 0.99368], corresponding to a modest 1.159% improvement. The metadata-based and outcome-trained selectors choose exactly the same conditions. Although those comparators require labeled development data, the present study establishes no additional held-out allocation value from measuring degree, whose stated protocol uses thousands of reference responses per condition. Moreover, the far-shell ratio of 1.00459 [0.99953, 1.00983] documents loss of the near-shell benefit, with uncertainty crossing one. I could not identify an outcome-independent rule that issued the abstract’s claimed warning of harmful augmentation before the errors were evaluated. The reported joint criterion is also unmet. The evidence supports a local diagnostic and a restricted allocation association, but does not yet establish the broader predictive capability suggested by the framing.

Questions for authors
  1. Clarification of the shift-warning claim: What quantity or decision rule, computed before evaluating augmentation outcomes, identifies a condition or operating radius at which added capacity will become harmful? Was such a warning rule fixed and tested on held-out outcomes? The reported procedure appears to reuse the near-shell selection and subsequently observe its far-shell reversal. Please clarify the predictive capability that is actually established.

  2. Additional analysis using existing responses and outcomes: At the same reference-response budget and selection quota, how does the proposed homogeneous profile compare with a simpler diagnostic such as the held-out residual of a low-degree fit or an orthogonal-polynomial tail-energy score? This comparison would help determine whether the specific normalization and homogeneous decomposition contribute predictive information beyond measuring ordinary polynomial approximation difficulty.

  3. Clarification of incremental value: Beyond the identical twelve-condition selections, do the existing held-out results show additional predictive information from degree over the metadata-based policy—for example, in condition-level gain prediction or ranking within each PDE? Please distinguish demonstrated advantages from proposed benefits in interpretability, development-label requirements, or transfer. Evidence of incremental prediction would materially affect my assessment of the method’s significance.

Reviewer_3mEQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at ways to measure interaction degree i.e to determine the number of nonlinear parameters. The strength of this is before running a training procedure it allows one to determine if the nonlinear models will help or not.

Critical strengths
  • Clear, well-defined object: whitening gives affine invariance, and the radial law (Prop. 1) is clean and exact.

  • Independently designed estimators (group lasso vs. signed rays) with strong correlation

  • The paper (Prop 3) also has a counterexample to their claim along with general weakness of their claim.

Critical weaknesses
  • The paper lacks similar simpler methods like ANOVA / Linear fit to observe how far the prior work and current work are off. Similarly no prior Neural Operators are compared ?

  • If a problem has a lot of high-degree content, then adding high-degree terms obviously helps. So the strong correlation (about 0.99) is less surprising than it sounds.

  • The selection method picks the same conditions that simple physical information already picks.

Questions for authors
  • Does the method beat simple linear-fit check at the same cost? (Weakness 1)

  • Does it offer better gains compared to trained FNO or DeepONet? (Weakness 2.) Even if they are not comparable a line to say why would be useful

  • How much of the 0.99 correlation is built from degree or how is this actually built ?

  • What does the results add beyond physical parameters ? Can it tell a smooth saturating nonlinearity apart from a truly high-degree one?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.