acceptodds
Under review as a conference paper at ICLR 2027

What Must Expert Formation Preserve? Formation–Deployment Compatibility in Channel-Composed Two-Expert Models

Abstract

Mixture-of-experts (MoE) systems often couple two distinct choices: how experts are formed during training and how the resulting experts are combined at inference. We separate these factors in a controlled channel-composed expert model and study how training-time formation affects effective post-training deployment. Matched formation interventions vary assignment information, exact mixing values, low-order statistics, online adaptation, reference dependence, and stochasticity; the resulting frozen checkpoints are then evaluated under multiple deployment rules. Across ESC-50, Speech Commands V2, FSD50K, CIFAR-100, and Tiny ImageNet, no single formation–deployment pair dominates. To test whether learned coefficient-to-channel correspondence matters beyond the coefficient values themselves, we perform a post-hoc exploratory assignment intervention that preserves the exact learned coefficient multiset while permuting only its correspondence with representation channels. The aligned assignment outperforms its permutation mean on ESC-50, Speech Commands, CIFAR-100, and Tiny ImageNet, but not on FSD50K. On ESC-50, the aligned-minus-permuted effect is Robust-B points with a descriptive fold-level 95% interval ; the corresponding learned-channel versus learned-scalar difference is points . Thus channel-assignment sensitivity is itself regime dependent. For the studied linear head, every fixed channel-wise composition compiles exactly into a single affine classifier. A separate reference-free stress test shows that increasing expert count can help or hurt, with no monotonic trend across task, formation, and deployment. Together, these results identify formation–deployment compatibility as a distinct design problem.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.