acceptodds
Desk rejected at ICLR 2027

Entropy–Contraction Coupling in Softmax-Routed Mixture Systems: Theory and Applications

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

We establish a general theoretical connection between the routing entropy of a softmax-gated mixture system and its global Lipschitz (contraction) constant. For any operator of the form , where and each component has Lipschitz constant , we prove that \[ L_T \;\leq\; c_\min \;+\; (c_\max - c_\min) e^H(\pi)K, \] where is the Shannon entropy of the routing distribution. This Entropy–Contraction Coupling (ECC) theorem reveals that routing specialization – a decrease in – automatically tightens the contraction bound, independently of the learning algorithm, loss function, or data distribution. We instantiate ECC across four settings: (i) Mixture-of-Experts (MoE) language models, where ECC explains training instability and suggests entropy-scheduled spectral constraints; (ii) Joint Embedding Predictive Architectures (JEPA), where ECC certifies stability for recursive latent predictors; (iii) diffusion model classifier-free guidance, where ECC characterizes score-mixture stability; and (iv) hierarchical reinforcement learning with options, where ECC gives Bellman-contraction guarantees under option switching. We propose Entropy-Aware Spectral Regularization (EASR), a practical training technique derived from ECC, and validate it empirically on MoE language model training and JEPA representation learning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Mixture-of-Experts (MoE), Joint Embedding Predictive Architectures (JEPA), diffusion model classifier-free guidance, hierarchical reinforcement learning with options, Entropy-Aware Spectral Regularization (EASR)

Related papers

Discussion (0)

Sign in to comment.