acceptodds
Under review as a conference paper at ICLR 2027

Beyond Boundary Agreement: Sharp Repair Costs for Hard-Routed Mixtures of Experts

Abstract

Can a hard-routed mixture of experts be made -Lipschitz without changing its selected expert set, while preserving its original function? We show that boundary agreement and transition-layer representation impose distinct architectural costs. A weighted projection inequality transfers approximation error of a forced, rescaled transition profile into excess repair risk. For an explicit two-expert Top-1 model, the original experts and the optimal boundary value use at most one ReLU, yet every width- shallow repair has an explicit leading-risk gap of at least . A self-contained differential certificate gives this bound without restrictions on network parameters. Consequently, asymptotically optimal shallow repair requires diverging width; matching the relative excess of a two-stage, two-ReLU repair requires width . In contrast, scalar one-dimensional repair admits exact convex endpoint optimization when expert Lipschitz constants are below , with at most six additional units per shallow ReLU branch. We also establish finite-budget scalar and vector trace oracles, a quantitative Top- trace-width obstruction, and a scalar polyhedral expansion with a sharp-order remainder. Selected expert calls, evaluated nonlinear units, and auxiliary computation are accounted for separately.

Primary area:
learning theory
Keywords:
mixture of experts, hard routing, Lipschitz approximation, neural network expressivity, boundary repair

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_atbzOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper theoretically analyzes how to approximate a hard-routed Mixture of Experts (MoE) model using a function that changes gradually (in a Lipschitz sense) instead of jumping discontinuously across routing boundaries. It does so with the target function and the router's expert selection choices frozen for each input rather than evolving with training. Under this model, the authors show that making neighboring experts "agree" on decision boundaries is not enough to satisfactorily approximate the hard-routed MoE. Instead, they show that achieving a near-optimal approximation in two or more dimensions forces a specific transition region between experts, which requires either an extra network layer or a growing number of hidden units in a single-layer network. The paper also gives explicit clipping methods to bridge jumps across one-dimensional and multi-region routers under their setup.

Critical strengths

This paper is honestly written. It explicitly states the assumptions and computational model under which its theory is conducted. It also supplements its mathematical proofs with deterministic numerical and symbolic checks.

Critical weaknesses

This paper is limited to a theoretical analysis of hard-routed MoEs via fixed-function approximation. There are no experiments on trained MoE checkpoints or real datasets to demonstrate whether the insights hold in practice. Because of the heavy mathematical notation and strong technical assumptions (such as enforcing a frozen discontinuous routing function), it is unclear how directly these theoretical bounds translate into practical design guidance for real-world MoEs.

Questions for authors

It is difficult for me to give this paper a positive recommendation given the lack of experiments and notational density of the results. However, I am open to considering any perspectives that I missed. In particular,

  • What is the practical takeaway for used-in-practice MoE architecture designs? As in, what design change (or re-enforcement of current design beyond just using multi-layers and many parameters) does the theory imply?
  • Why would we expect the Lipschitz bounds to retain their insight for real-world MoEs when key assumptions (frozen routing, shallow scalar experts, no nonlinear outputs, etc) no longer hold in practice? Especially when no empirical validation is provided?
Reviewer_vhnLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors study Lipschitz repair of hard-routed mixtures of experts while preserving the router’s selected expert sets and minimizing squared error to the original function. A boundary-layer projection inequality transfers approximation error into excess repair risk. For a simple two-expert model, the paper establishes a shallow-width lower bound, while a two-stage construction using two ReLUs achieves asymptotically optimal risk. Additional results cover Top-k trace complexity, exact one-dimensional repair, and scalar polyhedral asymptotics.

Critical strengths
  1. The paper distinguishes agreement at the routing boundary from representing the transition around it. The example uses simple original experts, so the obstruction arises from the repair problem itself.

  2. The differential certificate gives an explicit approximation lower bound without restricting parameter magnitudes. Combined with boundary-layer rigidity, it yields a quantitative width requirement.

  3. The one-dimensional result provides a useful contrasting case: exact repair reduces to convex endpoint optimization with constant additive width. The paper also accounts separately for expert calls and auxiliary computation.

Critical weaknesses
  1. The lower bounds concern shallow additive ReLU branches and exclude nonlinear mixture heads and input-dependent gate multiplication. Their implications for deeper experts and commonly used MoE architectures remain unclear.

  2. The explicit constants are conservative, and there is no matching width-dependent upper bound. The logarithmic necessary condition therefore does not establish the actual shallow width needed for near-optimal repair.

  3. They support implementation consistency but do not measure attainable shallow-repair error. A controlled approximation study across widths and Lipschitz budgets would help assess the bounds’ quantitative relevance.

  4. The vector oracle assumes flat, normal-constant branches, whereas the polyhedral expansion is scalar. A compact comparison of assumptions and conclusions would make these distinctions easier to follow.

Questions for authors
  1. Can an explicit shallow construction provide an upper bound in terms of width, allowing comparison with the necessary condition in Corollary 3?

  2. Which parts of the rigidity argument extend to deeper experts, other activations, or input-dependent mixture weights?

  3. Can a controlled numerical study compare shallow repairs with the two-stage construction under verified Lipschitz constraints?

Reviewer_98guOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors argue that in MoEs making experts agree at their boundaries is not sufficient for optimal stabilization. The network also needs enough expressive power to connect their outputs smoothly.

Critical strengths

Strongest aspect is the quantitative depth separation induced by a stability constraint. The geometry of the required transitions can impose fundamental depth and width costs on the architecture.

Critical weaknesses

No real-world experiments.

Other theoretical works on MoEs are missing such as 'Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Experts' (TMLR) --> the paper at least covers (P)Relu experts whereas the current papers is has many restrictive architectural assumptions (I would recommend the authors try to at least generalize to Prelu like the reference provided).

Questions for authors

How robust is the depth-separation result to more general MoE architectures? (beyond ReLU experts) Could you characterize when boundary agreement is sufficient for near-optimal repair? How tight is the quantitative shallow-width lower bound? Could you add some real-world MoE experiments? --> for example do real pretrained MoEs exhibit substantial discontinuities when expert selection changes, and can these be repaired while retaining the original router? (that would be insightful to see whether the theory actually applies in practice)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.