Entropy–Contraction Coupling in Softmax-Routed Mixture Systems: Theory and Applications
Desk rejected at ICLR 2027, so left out of the paper lists and the map.
Abstract
We establish a general theoretical connection between the routing entropy of a softmax-gated mixture system and its global Lipschitz (contraction) constant. For any operator of the form , where and each component has Lipschitz constant , we prove that \[ L_T \;\leq\; c_\min \;+\; (c_\max - c_\min) e^H(\pi)K, \] where is the Shannon entropy of the routing distribution. This Entropy–Contraction Coupling (ECC) theorem reveals that routing specialization – a decrease in – automatically tightens the contraction bound, independently of the learning algorithm, loss function, or data distribution. We instantiate ECC across four settings: (i) Mixture-of-Experts (MoE) language models, where ECC explains training instability and suggests entropy-scheduled spectral constraints; (ii) Joint Embedding Predictive Architectures (JEPA), where ECC certifies stability for recursive latent predictors; (iii) diffusion model classifier-free guidance, where ECC characterizes score-mixture stability; and (iv) hierarchical reinforcement learning with options, where ECC gives Bellman-contraction guarantees under option switching. We propose Entropy-Aware Spectral Regularization (EASR), a practical training technique derived from ECC, and validate it empirically on MoE language model training and JEPA representation learning.