- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a method to learn neural operators by Mixtures of Neural Operators (MoNOs), where a routing mechanism is used to select a specialized neural operator for a given input. The paper presents theoretical analysis to show that MoNOs achieve much smaller expert complexity than a single unified neural operator, with a small cost of additional experts and routing complexity. Then, controlled experiments on Burgers equation are presented to illustrate the potential benefits of MoNOs.
Critical strengths
- The question of whether conditional computation can reduce the complexity of neural operators is novel and interesting, and leads to important gains in terms of computational complexity and representational efficiency.
- The paper is theoretically founded. It presents substantial and rigorous analysis to quantify the approximation power of the proposed mixture-of-experts approach.
Critical weaknesses
- There is still a gap between the theory and practice of the proposed method. In particular, the theoretical construction relies on covering a compact function space and the nearest-center routing mechanism; however, the experiments use only a small number of experts and rely on a classifier that is trained on predefined regime labels. It remains a question of whether the theoretical construction can really translate to the training dynamics of the model.
- The experiments are done mainly on the Burgers equations, limiting the scope of the empirical study. Moreover, the heterogeneous setting couples the viscosity with initial-condition steepness. While this is a setting that explicitly tailors the mixture-of-experts idea, it would be interesting to see if similar benefits arise in more natural and realistic operator-learning problems.
- While the paper is theory-minded, I find that the theoretical exposition is still not absolutely comprehensive. In particular, the complexity improvement is established relative to a specific constructive upper bound for a single neural operator, rather than a lower bound or an optimal approximation rate.
- The complexity of the method does not only relate to the lightweighted experts, but is also related to concerns such as the complexity of the router, the space complexity needed for each expert, and so on. The paper would benefit from a more comprehensive evaluation of all these considerations.
Questions for authors
- In practical applications, an appropriate decomposition of the input space is generally unknown a priori. How does the proposed method identify useful experts and routing decisions without predefined regime labels? Would the theoretical advantages remain to hold when the decomposition and the router must be learned entirely from data?
- Related to the complexity discussion in the weakness section, under what circumstances does the reduction in active expert size translate into a meaningful efficiency improvement? Can the authors provide some explicit evidence regarding the wall-clock measurement, space complexity, and storage requirement?
- The heterogeneous Burgers experiment deliberately introduces distinct regimes through viscosity selection. Can the authors demonstrate improvements (or not) on a more naturally heterogeneous operator-learning problem, or investigate how the benefits vary with the degree of heterogeneity?