acceptodds
Under review as a conference paper at ICLR 2027

CARE-COM: Capability-Aware Compression of Mixture-of-Experts Models

Abstract

Sparse Mixture-of-Experts (MoE) models achieve high parameter capacity at low per-token cost, but their large expert pools create deployment memory pressure. Existing compression methods identify candidates for removal or merging using static signals computed from the initial model—parameter similarity, routing statistics, or output clustering. These signals do not directly measure the functional consequence of physically modifying an expert population, and they are not updated as compression proceeds. We study expert compression through a functional intervention lens. We represent each expert by its responses to a fixed probe set—a capability vector—and show that distances between these vectors contain substantial predictive information about merge-induced functional damage: capability geometry achieves a mean held-out Spearman correlation of with observed merge damage on OLMoE-1B-7B-0924, versus for a local descriptor baseline. However, among 186 pairs satisfying a near-equal-distance criterion (), observed KL damage spans to , showing that geometry identifies a useful search neighborhood but does not uniquely determine intervention consequence. This motivates CARE-COM: at each step, capability geometry generates a small candidate pool, each candidate is physically merged and evaluated for actual functional damage, the lowest-damage merge is committed, and the capability state is recomputed before the next step. In a controlled comparison on OLMoE, adaptive recomputation reduces cumulative KL divergence by , , and at 60, 56, and 48 remaining experts relative to a frozen capability ranking, while also reducing perplexity at every target. On a common WikiText-2 benchmark, CARE-Adaptive achieves the lowest perplexity at 60 experts; Sub-MoE is slightly lower at 56 and 48 experts. These results support treating expert compression, in the evaluated OLMoE setting, as a sequential functional intervention problem in which the relevant functional state evolves after each physical modification.

Primary area:
optimization
Keywords:
Mixture-of-Experts, Model Compression, Expert Consolidation, Functional Geometry, Interpretability, Sparse Neural Networks, Large Language Models, Neural Network Pruning, Adaptive Compression

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_78RpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper views expert compression as a sequence of functional interventions. It uses expert responses on a fixed probe set to construct capability vectors, uses geometric distance to produce K=5 candidates, physically merges and measures output KL for each candidate, commits the merge with the least damage, and then recomputes capabilities on the modified model. The experiments focus on OLMoE-1B-7B-0924, comparing CARE-Static, which freezes the geometric ranking, with CARE-Adaptive, which recomputes it at each step, and comparing WikiText-2 perplexity against HC-SMoE, MC-SMoE, and Sub-MoE.

Critical strengths
  • The framework—geometry proposes candidates, measured functional damage determines the decision, and state is recomputed after model modification—is clear and more faithful to the actual effects of an intervention than committing merges using only static similarity.
  • Section 3 evaluates the ability of distance to predict damage using expert-disjoint cross-validation, rather than reporting only in-sample correlations.
  • The paper proactively limits its claims: it explicitly notes the single model, single run, unexplored K, dependence on probes and calibration data, and the fact that Sub-MoE is slightly better at 56/48 experts.
Critical weaknesses
  1. Core numerical results are internally inconsistent. The abstract, Section 4, and Figure 3 state that the KL damage of 186 nearly equidistant pairs ranges from 0.001522 to 0.007096. The same paragraph and figure caption then give examples with KL≈0.074 and 0.046, approximately an order of magnitude larger and clearly outside the stated range. The distribution in Figure 3 corresponds to a cluster at distance approximately 0.0022, while the example distances are 0.0156/0.0157, so the sample sets do not align either. It is currently impossible to determine the scale and samples used for the central evidence.
  2. The core capability representation is not reproducibly defined in the PDF. Equation 1 gives only the abstract function F(e_i;P). Appendix D merely states that the probes are “fixed synthetic inputs designed to cover diverse expert output behaviors,” with the complete construction to be released later with the code. The paper omits the probe texts, response aggregation, vector dimensionality, normalization, cross-layer combination, and generation seed. It also provides an anonymous code-repository link, so the tension between “code available” in the main text and “details to be released later” in the appendix requires clarification.
  3. The controlled comparison in which “only recomputation differs” is unclear. Section 6.2 states that both variants use K=5 and functional-damage evaluation. However, Figure 5 describes CARE-Static as directly merging after a one-time ranking, while candidate generation and functional evaluation are shown only for Adaptive. Appendix C further reports 115 exact evaluations for Static and exactly 16×5=80 for Adaptive, with Static being approximately 5× slower. If the only difference is recomputation in Equation 6, these discrepancies in evaluation counts and procedure need explanation; otherwise, attributing the 17.8%–24.6% gain to recomputation is unsupported.
Questions for authors
  1. What is the relationship between 0.001522–0.007096 and 0.046/0.074? Please provide the samples used for each figure and statistic, with consistent units.
  2. Please fully define F and probe set P, and report the correlations and final compression results under changes to the probes and calibration data.
  3. Why does CARE-Static perform 115 functional evaluations while CARE-Adaptive performs 80? At every step, do both variants apply the same temporary merge and KL evaluation to the same K candidates?
Reviewer_gScuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this paper the authors present how to use a sequence of functional interventions to compress a Multi-Expert (MoE) network by expert capability vectors. A capability vector represents what an expert can do based upon its response to a set of probes. There is a good relationship between the Euclidean distance of two capability vectors and the amount of KL divergence caused when merging them together. Using expert-disjoint cross validation they found a mean held out Spearman correlation of 0.7504. This was compared to a local baseline which had a Spearman correlation of 0.4797 and a combined predictor which had a Spearman correlation of 0.8146.

Additionally, CARE-Adaptive resulted in lower WikiText-2 perplexity than CARE-Static. Among other methods tested against CARE-Adaptive, Sub-MoE performed slightly better than CARE-Adaptive at 56 and 48 experts. However, CARE-Adaptive was still able to achieve the lowest perplexity value at 60 experts.

Critical strengths
  • Problem Statement: An intuitive explanation for how to make merge decisions using the current state of a modified model rather than just comparing it with the original model's geometry. It also describes the "Geometry Proposes, Physical Intervention Decides, State Recomputes" process.

  • Comparison Design: A sound comparison design was used to compare Static and Adaptive methods. Only one variable was changed in this study; that is, whether capabilities were recomputed.

  • Limited Scope of Claims: This research acknowledges that there was only one model and one run. In addition, it does not claim that their method will always perform better than Sub-MoE. Finally, they are honest about the limitations of their work.

Critical weaknesses
  • Limited evidence. A single model (OLMoE-1B-7B), a single run, wiki-text-2 perplexity only, up to 25% compression, no uncompressed baseline; therefore, it is difficult to determine if the 0.08 – 0.66 perplexity difference between sub-moe is statistically significant.

  • Lack of details. The capability probe set is not described. With 64 → 48 there are only 16 merge operations and 80 evaluation operations; therefore, it is not clear if compression occurs on just one layer or on the same index expert(s) for each layer. "Cumulative KL" has no definition; after four merge operations it is 2.477 but single-merge KL can be at most approximately 0.045.

  • Internal contradictions. According to descriptions both methods should have exactly the same results at the first stage; however, Figure 3/6 shows different pairs. Static takes nearly five times longer and performs approximately five times as many evaluations (115 versus 80), which is inconsistent with "no recomputation."

-Incorrect router merge. Summing router column entries produces logit zi+zjz_i + z_j, which does not maintain routing mass when using softmax top-k.

Questions for authors
  • What is the construction process for the capability probing set? In addition, what layer(s) are acted upon by each of the 16 merges?

  • How is Cumulative KL defined? Why would both approaches select different pairs at the initial stage?

  • Would it be possible to conduct these experiments across multiple seeds, model types, and zero-shot task categories using the same merging operation as Sub-MoE and REAP?

Reviewer_w7gAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Sparse Mixture-of-Experts (MoE) architectures achieve high parameter capacity with low per-token cost, but storing the full expert pool causes substantial deployment memory pressure. Prior compression techniques rely primarily on static signals computed from the initial model state (e.g., parameter similarity or output clustering). The core concept of CARE-COM is formulating expert compression as a sequential functional intervention problem. Each expert is characterized by a "capability vector" obtained from its activations across probe inputs. Recognizing that geometric similarity identifies a promising candidate pool but does not guarantee the lowest functional damage, CARE-COM evaluates candidate merges via physical interventions (measuring output KL divergence) and updates the capability state adaptively after each modification.

Critical strengths
  • The authors provide rigorous empirical justification for every stage of their pipeline. By verifying that capability distance strongly predicts damage (Claim 1) but still leaves substantial variance among near-equal distance pairs (Claim 2), they theoretically and empirically justify the necessity of combining geometric pre-filtering with explicit physical intervention testing.
  • Through controlled head-to-head comparisons on OLMoE-1B-7B-0924 holding all parameters and seeds constant, the paper demonstrates that adaptive recomputation yields substantial advantages over frozen rankings.
  • They reach State-of-the-Art quality at moderate compression rates. Under a standard WikiText-2 evaluation benchmark against competitive baselines (such as Sub-MoE, HC-SMoE, and MC-SMoE), CARE-Adaptive achieves the best overall perplexity (11.548) at the 60-expert level. It prevents 49.0% of perplexity degradation relative to random merging, decisively outperforming Sub-MoE (38.3%), CARE-Static (26.2%), and previous clustering baselines (18.4%).
Critical weaknesses
  • All empirical evaluations rely strictly on a single model architecture (OLMoE-1B-7B-0924) evaluated on a single dataset (WikiText-2) using a single fixed random seed. Multi-task performance across downstream benchmarks (e.g., MMLU, GSM8K) is completely absent, making it difficult to judge whether general capabilities are preserved uniformly.
  • CARE-Adaptive’s relative advantage diminishes as compression increases. When compressing to 56 and 48 experts, Sub-MoE achieves lower perplexity (13.735 vs. 13.812 at 56, and 19.477 vs. 20.136 at 48). The perplexity degradation between 60 and 48 experts is steeper for CARE-Adaptive (+8.588) than for Sub-MoE (+7.666).
  • CARE-COM innovates on candidate selection and update scheduling, but adopts an existing, basic operator (uniform parameter averaging and router column addition). Moreover, evaluating physical merges dynamically incurs non-trivial overhead. While the authors report that CARE-Adaptive ran faster than CARE-Static in Appendix C due to implementation quirks, comparing wall-clock efficiency against true one-shot or static clustering baselines (e.g., Sub-MoE, HC-SMoE) is omitted.
Questions for authors
  • In Section 5.3, the candidate pool budget is fixed at K=5 without ablation. As demonstrated by Claim 2, pairs with virtually identical capability distances can exhibit up to 5-fold differences in KL divergence. If geometric distance only weakly isolates fine-grained damage, is it not plausible that the globally optimal merge pair lies outside this top-5 pool? How sensitive are compression quality and evaluation runtime to varying K = 3, 5, 10, 20? Could an adaptive threshold based on distance distribution be more effective than a hard cutoff?
  • Equation 3 relies on a held-out pretraining calibration distribution to calculate KL divergence for intervention selection. If downstream deployments encounter domain shifts (e.g., specialized code, biomedical corpora, or conversational dialogue), how stable is the selected merge sequence? Does optimizing sequential merges against a general calibration distribution inadvertently over-prune expert capacities critical for out-of-distribution reasoning? Have the authors tested whether merge decisions or cumulative perplexity diverge significantly when utilizing alternative calibration sets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.