When Can Pairwise Routing Dynamics Be Tested in Mixture-of-Experts? Identifiability Conditions and Evidence from a Learned Router
Abstract
How can pairwise routing dynamics in a learned mixture-of-experts (MoE) model be tested against a reduced stochastic law? We provide a falsifiable measurement criterion for this question. The criterion is instantiated by a specified two-expert (E=2) score-gap chain with an exact endpoint probability and a central first-order blind spot. For the aggregate soft-routing schema in our historical eight-expert (E=8) records, we prove that the fixed hard count, one-selection transition, and specified endpoint are not identifiable; that is, they are not uniquely determined by the recorded quantities. This is a schema-specific boundary on a dynamics test, not a universal impossibility result for MoE-to-reduced-model transfer. We provide a logging contract for a finite-chain dynamics test and verify the resulting analysis procedure on an imposed E=2 subsystem (4,608 independently replayed trajectories). We then measure pairwise routing dynamics in a 17M-parameter, four-layer, character-level E=8 MoE that makes one hard expert choice per token during the forward pass. Training, expert-pair choice, and evaluation use disjoint token spans. Its predefined six-feature linear model did not improve mean leave-one-seed-out MSE over router learning rate alone (mean MSE difference ; 5 of 8 seed folds favor it). A narrower local effect, gain-dependent selected-pair soft-share movement, replicates: movement is larger at than at for 15 of 16 paired seeds on Shakespeare and an independent enwik8 prefix. This effect is reproducible local routing dynamics, not a validated E=2 transfer law.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.