acceptodds
Withdrawn from ICLR 2027

Mixtures of Neural Operators Reduce Active Complexity in Operator Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Operator-learning systems are not governed solely by total parameter count; for one query, the relevant bottleneck can be the model that must be loaded and evaluated. We study this distinction for classical neural operators on compact Sobolev subsets through a constructive comparison between routed mixtures of neural operators (MoNOs) and a fixed single-neural-operator construction. A MoNO routes each input function through a tree to one expert. Our main theorem shows that every scalar uniformly continuous nonlinear operator with bounded output Sobolev radius on the approximation set admits a MoNO approximation whose active expert has smaller depth and width scaling, and no larger rank scaling, than the analyzed single-neural-operator construction; for Lipschitz targets these expert quantities are bounded by . A coefficient-quantization tree also gives polynomial routing work for Lipschitz targets under an explicit coefficient-access model, while storage remains exponential. For bounded functionals, affine experts improve the exponential storage power over constant lookup under target-independent routing. We also prove a quantitative universal approximation theorem for the underlying neural-operator architecture, with explicit dependence on compact-set diameter and modulus of continuity.

Primary area:
learning theory
Keywords:
Mixture of Experts, Neural Operators, Distributed Computing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QserOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method to learn neural operators by Mixtures of Neural Operators (MoNOs), where a routing mechanism is used to select a specialized neural operator for a given input. The paper presents theoretical analysis to show that MoNOs achieve much smaller expert complexity than a single unified neural operator, with a small cost of additional experts and routing complexity. Then, controlled experiments on Burgers equation are presented to illustrate the potential benefits of MoNOs.

Critical strengths
  • The question of whether conditional computation can reduce the complexity of neural operators is novel and interesting, and leads to important gains in terms of computational complexity and representational efficiency.
  • The paper is theoretically founded. It presents substantial and rigorous analysis to quantify the approximation power of the proposed mixture-of-experts approach.
Critical weaknesses
  • There is still a gap between the theory and practice of the proposed method. In particular, the theoretical construction relies on covering a compact function space and the nearest-center routing mechanism; however, the experiments use only a small number of experts and rely on a classifier that is trained on predefined regime labels. It remains a question of whether the theoretical construction can really translate to the training dynamics of the model.
  • The experiments are done mainly on the Burgers equations, limiting the scope of the empirical study. Moreover, the heterogeneous setting couples the viscosity with initial-condition steepness. While this is a setting that explicitly tailors the mixture-of-experts idea, it would be interesting to see if similar benefits arise in more natural and realistic operator-learning problems.
  • While the paper is theory-minded, I find that the theoretical exposition is still not absolutely comprehensive. In particular, the complexity improvement is established relative to a specific constructive upper bound for a single neural operator, rather than a lower bound or an optimal approximation rate.
  • The complexity of the method does not only relate to the lightweighted experts, but is also related to concerns such as the complexity of the router, the space complexity needed for each expert, and so on. The paper would benefit from a more comprehensive evaluation of all these considerations.
Questions for authors
  1. In practical applications, an appropriate decomposition of the input space is generally unknown a priori. How does the proposed method identify useful experts and routing decisions without predefined regime labels? Would the theoretical advantages remain to hold when the decomposition and the router must be learned entirely from data?
  2. Related to the complexity discussion in the weakness section, under what circumstances does the reduction in active expert size translate into a meaningful efficiency improvement? Can the authors provide some explicit evidence regarding the wall-clock measurement, space complexity, and storage requirement?
  3. The heterogeneous Burgers experiment deliberately introduces distinct regimes through viscosity selection. Can the authors demonstrate improvements (or not) on a more naturally heterogeneous operator-learning problem, or investigate how the benefits vary with the degree of heterogeneity?
Reviewer_RGiJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the distinction between active expert complexity, routing work, and total storage in operator learning. On compact Sobolev approximation sets, it constructs mixtures of neural operators that approximate uniformly continuous operators while keeping the selected expert small. The basic construction uses constant experts storing projected outputs at covering centers. A coefficient-quantization tree provides polynomial routing work for Lipschitz targets under an explicit coefficient-access model, while storage remains exponential.

The paper also establishes a quantitative approximation theorem for the underlying finite-rank neural-operator architecture, derives a fixed-target separation from a specified class of single ReLU models using an existing lower bound, and shows that affine experts improve the exponential storage power over constant lookup for a bounded C1,1 functional class with target-independent routing.

Controlled one-dimensional Burgers experiments examine specialization across prescribed viscosity regimes. The strongest comparison shares routing and label information between the mixture and a global FNO, finding a moderate accuracy improvement at comparable active size and increased total storage.

Critical strengths
  1. The resource accounting makes the theoretical conclusions precise and interpretable. Section 3.3.1 separates active expert parameters, routing work, and total storage, and Theorem 4.8 includes the coefficient-access and precision assumptions behind polynomial routing. Corollary 4.5 gives a fixed-target separation against the stated single-ReLU model class, with an appropriate bounded scalar readout. Proposition 4.7 also carefully states the common-router quantifier required for its constant-expert lower bound. These details are consequential because they establish a coherent approximation result with identifiable costs and limitations.

  2. The experimental controls substantially improve the credibility of the specialization results. In particular, Appendix F.3 uses training-only thresholds and gives the MoNO and global FNO the same supervised routing information and predicted labels. The paper also includes constant and affine experts, input-space routing, and a homogeneous control, and reports unfavorable outcomes and training-budget limitations. This makes the observed benefit easier to attribute and prevents the earlier, larger performance gaps from being interpreted without the relevant qualifications.

Critical weaknesses
  1. The incremental theoretical contribution over existing localization and lookup constructions appears limited. The main active-expert result is attained by storing a projected output at each covering center; polynomial routing is obtained by digitwise coefficient quantization; and the affine-expert storage improvement follows from a quadratic Taylor remainder combined with Sobolev covering entropy. The trade-off between expert size and the number of local regions is already central to the cited finite-dimensional MoE approximation literature. I appreciate the explicit treatment of growing coefficient dimension, represented centers, and precision, but the paper does not make sufficiently clear which quantitative conclusions go beyond combining those existing ideas with Sobolev truncation. Consequently, I find the new insight into neural-operator approximation more limited than the breadth of the formal development suggests.

  2. The empirical contribution provides only limited additional support for the paper’s significance. In the strongest shared-router comparison, relative L2 error improves from 0.0452 to 0.0388 at comparable active size, with a reported 2.8% MSE reduction and 3.36 times the stored values. This is a useful specialization result, but the experiments concern a small, discrete Burgers setting with prescribed hard viscosity switches, and validation optima remain close to the training limit. The discontinuous regime construction also does not test the uniform-continuity results or the smooth-target storage improvement. These acknowledged limitations leave a substantial gap between the general resource-separation theory and evidence about the practical accuracy–resource trade-off of learned neural experts. Together with the limited theoretical increment described above, this leads me to lean toward weak rejection.

Questions for authors
  1. Clarification of the theoretical increment: Could you provide a quantitative comparison between your constructions and applying the cited finite-dimensional MoE/local-approximation results after Sobolev coefficient truncation, explicitly retaining the dependence on the growing coefficient dimension? Which exponents, resource bounds, or representability guarantees in Theorems 4.1 and 4.8 and Proposition 4.7 require a substantive additional argument or improve on that composition? This would materially affect my assessment of the theoretical contribution.

  2. Targeted evaluation of existing checkpoints: Appendix F.3 reports 97.62% routing accuracy for a simple training-threshold rule based on an existing input feature, compared with 89.97% for the learned classifier. Could you evaluate both the existing MoNO and conditioned-global checkpoints using this same threshold rule, without retraining or checkpoint reselection, and report relative L2 error and MSE across seeds? Since misrouted examples have much larger errors for both predictors, this comparison would help determine how much of the practical limitation comes from routing and how much specialization benefit remains when routing is more accurate.

Reviewer_kXeTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper distinguishes a model's total stored size from the number of expert parameters loaded and evaluated for one operator-learning query. It tries to define a mixtures of neural operators in which a tree router selects one local expert for each input function. Its main theorem gives uniformly accurate MoNO constructions whose active expert is much smaller than the analysed global single-operator construction, although total storage remains exponential. It additionally proves polynomial routing work under coefficient access and shows that affine experts improve storage exponents for bounded C1,1C^{1,1} functionals under target-independent routing. Controlled Burgers experiments demonstrate strong gains in explicitly heterogeneous regimes, but also show substantial sensitivity to route labels, side information, supervision, and training budget.

Critical strengths

The paper tries to make a precise and useful distinction between active, distributed, routing, and storage complexity. Beyond universal approximation, the paper gives explicit dependence on the modulus of continuity and Sobolev dimensions, a hard-functional separation from single ReLU models, improved storage scaling with affine experts, and a coefficient-quantization router with explicit work and precision assumptions.

Critical weaknesses

The critical weakness of the paper is that the idea of MoNO is simply unclear. Too many important things are in the appendix and that makes the paper hard to understand beyond just a vague outline. How is one supposed to read Definition 3.4? Is this an algorithm or an architecture/function? Algorithm 1 is quite hard to read - things like f^i\hat{f}_i is not defined - and that routing is the most important algorithm. And this key routing mechanism is not even in the main text! The main text simply fails to explain what is the way to use this thing given in Definition 3.4.

Questions for authors

The paper needs significant rewriting!

How is one supposed to read Definition 3.4? Is this an algorithm or an architecture/function?

What is the loss function to train a MoNO?

The "balanced tree construction" seems important to read Theorem 4.1 but that is not defined in the main text!

What is this "realization" G of MoNO that Theorem 4.1 talks of?

This draft is far from being publishable and needs a lot of things to be clarified before much can be understood. As a general guidance, I think papers should not be written with forward referencing - which is used everywhere here! Its very hard to read a paper where things stated before is not enough to understand the current line.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.