Beyond Boundary Agreement: Sharp Repair Costs for Hard-Routed Mixtures of Experts
Abstract
Can a hard-routed mixture of experts be made -Lipschitz without changing its selected expert set, while preserving its original function? We show that boundary agreement and transition-layer representation impose distinct architectural costs. A weighted projection inequality transfers approximation error of a forced, rescaled transition profile into excess repair risk. For an explicit two-expert Top-1 model, the original experts and the optimal boundary value use at most one ReLU, yet every width- shallow repair has an explicit leading-risk gap of at least . A self-contained differential certificate gives this bound without restrictions on network parameters. Consequently, asymptotically optimal shallow repair requires diverging width; matching the relative excess of a two-stage, two-ReLU repair requires width . In contrast, scalar one-dimensional repair admits exact convex endpoint optimization when expert Lipschitz constants are below , with at most six additional units per shallow ReLU branch. We also establish finite-budget scalar and vector trace oracles, a quantitative Top- trace-width obstruction, and a scalar polyhedral expansion with a sharp-order remainder. Selected expert calls, evaluated nonlinear units, and auxiliary computation are accounted for separately.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.