- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a diagnostic metric called interaction degree to predict under which problem conditions adding higher-order nonlinear computational capacity to neural operators improves performance. The authors approximate the target operator using polynomials over a specified input range and reference distribution, and quantify the importance of higher-order interactions by measuring the magnitude of components at each degree. They also conditionally analyze the relationship between coordinate invariance of the metric and polynomial approximation error, and compare two estimation methods: direct polynomial regression and directional probing. In the experiments, the authors measure the contribution of higher-order components under various conditions of the Burgers and Allen–Cahn equations and use these measurements to select conditions in which to apply the same additional residual module. Under the selected conditions, the method achieves an average error reduction of approximately 1.2% for distribution shifts close to the training range, but no improvement is observed under larger distribution shifts. Confirmatory experiments on FNOs operating on full spatial fields remain incomplete, so the empirical evidence for practical neural operators is primarily limited to module placement experiments in a restricted coefficient space.
Critical strengths
-
The paper presents a clear analytical framework for measuring higher-order interactions. It explicitly specifies the input space, reference distribution, and output norm, and defines interaction degree in terms of the magnitude of components at each polynomial degree. By analyzing its invariance under affine coordinate transformations that represent the same problem and its conditional relationship with polynomial approximation error, the authors clarify the interpretation of the metric and the conditions under which it is valid.
-
The paper validates the metric through independent estimation methods and controlled module placement experiments. The authors compare direct polynomial regression with separately collected directional probing results to assess the consistency of the measurements. They also fix the architecture of the additional module and the number of conditions selected for its application, and evaluate a selection rule established before observing performance outcomes. These experiments provide preliminary empirical evidence that the relative importance of higher-order components can be useful for guiding module placement.
Critical weaknesses
-
The necessity of measuring interaction degree separately from learning higher-order modules is not sufficiently established. Given the existence of approaches capable of learning higher-order interactions, such as TESLA (Ko et al., 2026) [1], it remains unclear what additional benefits are provided by measuring interaction degree in advance. A separately trained selector based on physical parameters also selects the same conditions, making it difficult to isolate the practical contribution of the proposed measurement procedure. A direct comparison with strategies that learn and validate module placement without prior degree estimation is needed, while matching module capacity, the number of selected conditions, and validation budgets, and accounting for the data collection and regression costs required to estimate interaction degree.
-
The contribution beyond expected effects of polynomial approximation is not sufficiently validated. It is reasonable to expect that higher-degree polynomial approximations perform better for targets with larger higher-order components. Therefore, a strong correlation between interaction degree and improvements in polynomial approximation performance is insufficient to establish that the metric provides additional information useful for allocating capacity in actual neural networks. The paper itself also shows that the performance gains of a particular module can differ even under the same degree distribution. The central issue is not merely whether interaction degree can be measured, but whether this measurement leads to better decisions than those based on existing information or conventional validation procedures.
-
The limited scope of validation on actual neural networks leaves practical utility and scalability uncertain. The main module placement experiments focus on low-dimensional coefficient representations of two PDEs. Under the selected conditions, the method achieves an average error reduction of approximately 1.2% for distribution shifts close to the training range, but no improvement is observed under larger distribution shifts. Furthermore, confirmatory experiments on FNOs operating on full spatial fields remain incomplete, making it difficult to assess the effectiveness of the approach for general neural operators. In high-dimensional problems such as image modeling or LLMs, additional challenges arise in defining input transformations and reference distributions, as well as managing measurement costs. Consequently, the applicability supported by the current evidence remains limited.
References
[1] Daehwa Ko, JaeHyeon Kim, SeungHyun Ham, and Jay Hoon Jung. TESLA: Taylor Expansion of Sinusoidal Learnable Activations. AISTATS 2026, PMLR 300:5257–5265.
Questions for authors
-
Under what conditions is measuring interaction degree separately more advantageous than directly using learnable higher-order modules? Compared with approaches capable of learning higher-order interactions, such as TESLA (Ko et al., 2026), what benefits does prior degree estimation provide in terms of accuracy or computational cost? Can the authors provide a comparison that matches module capacity, the number of selected conditions, and training and validation budgets, while also accounting for the cost of degree estimation?
-
What additional value does interaction degree measurement provide when a selector based on physical parameters identifies the same conditions? Since the parameter-based selector uses performance labels, the two approaches differ in their information requirements. Considering this distinction, along with the data collection and computational costs required for degree estimation, can the authors quantitatively identify the conditions under which the proposed method offers an advantage?
-
How can the information contributed by interaction degree to predicting actual module performance gains be isolated? The paper shows that performance gains can differ even under the same degree distribution, depending on the directions within each degree and their alignment with the module. Does interaction degree retain its predictive power when these factors are controlled for or measured? Beyond the strong correlations observed in the polynomial approximation experiments, please clarify what evidence supports this relationship in actual neural networks.
-
Why do the improvements observed under nearby distribution shifts not extend to larger distribution shifts? Can the authors investigate whether this limitation primarily arises from errors in degree estimation, limitations of polynomial approximation, or difficulties in training and generalizing the additional module? Furthermore, please clarify how measurement costs and estimation stability are expected to change when extending the current low-dimensional coefficient experiments to full spatial fields or higher-dimensional inputs.