CARE-COM: Capability-Aware Compression of Mixture-of-Experts Models

Sparse Mixture-of-Experts (MoE) models achieve high parameter capacity at low per-token cost, but their large expert pools create deployment memory pressure. Existing compression methods identify candidates for removal or merging using static signals computed from the initial model—parameter similarity, routing statistics, or output clustering. These signals do not directly measure the functional consequence of physically modifying an expert population, and they are not updated as compression proceeds. We study expert compression through a functional intervention lens. We represent each expert by its responses to a fixed probe set—a capability vector—and show that distances between these vectors contain substantial predictive information about merge-induced functional damage: capability geometry achieves a mean held-out Spearman correlation of $0.7504$ with observed merge damage on OLMoE-1B-7B-0924, versus $0.4797$ for a local descriptor baseline. However, among 186 pairs satisfying a near-equal-distance criterion ($|\Delta d|<10^{-4}$), observed KL damage spans $0.001522$ to $0.007096$, showing that geometry identifies a useful search neighborhood but does not uniquely determine intervention consequence. This motivates CARE-COM: at each step, capability geometry generates a small candidate pool, each candidate is physically merged and evaluated for actual functional damage, the lowest-damage merge is committed, and the capability state is recomputed before the next step. In a controlled comparison on OLMoE, adaptive recomputation reduces cumulative KL divergence by $17.8$, $24.6$, and $15.9$ at 60, 56, and 48 remaining experts relative to a frozen capability ranking, while also reducing perplexity at every target. On a common WikiText-2 benchmark, CARE-Adaptive achieves the lowest perplexity at 60 experts; Sub-MoE is slightly lower at 56 and 48 experts. These results support treating expert compression, in the evaluated OLMoE setting, as a sequential functional intervention problem in which the relevant functional state evolves after each physical modification.