MoE Modular Post-Training: Improving and Composing Emergent Domain Experts

Modular pretraining enables Mixture-of-Experts (MoE) language models to form domain expert modules that can operate independently. Can these modules also be independently improved and composed? We study MoE modular post-training: training domain modules separately and integrating their updates into a shared model. We find that modules fine-tuned with all their parameters can lose much of their improvement when reinserted. Controlled interventions trace this loss not to router mismatch but primarily to co-adaptation between the domain experts and the model's interface, the parameters outside the routed experts: both change during fine-tuning, but only the expert updates are reinserted. We therefore introduce Interface-Frozen Fine-Tuning (IF-FT), which updates only the domain experts and keeps the interface fixed. Across seven domains on EMO, a 14B-parameter modular MoE, IF-FT preserves standalone accuracy after reinsertion to within test-set noise and improves reinserted accuracy by up to 10 points over full-parameter module fine-tuning. It closely matches training the same experts inside the full model while holding only 3.9B of the 14B parameters in memory during training, and it does not reduce accuracy on other domains. Independently trained modules also compose without further training, and no domain falls below the untrained model; their gains survive best when the modules share few experts. In a standard MoE trained on the same data, reinsertion is equally lossless, but even after training the extracted module falls below its own full model. Emergent domain experts can thus serve as independent units of post-training and composition, provided modularity is built in during pretraining.