- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper views expert compression as a sequence of functional interventions. It uses expert responses on a fixed probe set to construct capability vectors, uses geometric distance to produce K=5 candidates, physically merges and measures output KL for each candidate, commits the merge with the least damage, and then recomputes capabilities on the modified model. The experiments focus on OLMoE-1B-7B-0924, comparing CARE-Static, which freezes the geometric ranking, with CARE-Adaptive, which recomputes it at each step, and comparing WikiText-2 perplexity against HC-SMoE, MC-SMoE, and Sub-MoE.
Critical strengths
- The framework—geometry proposes candidates, measured functional damage determines the decision, and state is recomputed after model modification—is clear and more faithful to the actual effects of an intervention than committing merges using only static similarity.
- Section 3 evaluates the ability of distance to predict damage using expert-disjoint cross-validation, rather than reporting only in-sample correlations.
- The paper proactively limits its claims: it explicitly notes the single model, single run, unexplored K, dependence on probes and calibration data, and the fact that Sub-MoE is slightly better at 56/48 experts.
Critical weaknesses
- Core numerical results are internally inconsistent. The abstract, Section 4, and Figure 3 state that the KL damage of 186 nearly equidistant pairs ranges from 0.001522 to 0.007096. The same paragraph and figure caption then give examples with KL≈0.074 and 0.046, approximately an order of magnitude larger and clearly outside the stated range. The distribution in Figure 3 corresponds to a cluster at distance approximately 0.0022, while the example distances are 0.0156/0.0157, so the sample sets do not align either. It is currently impossible to determine the scale and samples used for the central evidence.
- The core capability representation is not reproducibly defined in the PDF. Equation 1 gives only the abstract function
F(e_i;P). Appendix D merely states that the probes are “fixed synthetic inputs designed to cover diverse expert output behaviors,” with the complete construction to be released later with the code. The paper omits the probe texts, response aggregation, vector dimensionality, normalization, cross-layer combination, and generation seed. It also provides an anonymous code-repository link, so the tension between “code available” in the main text and “details to be released later” in the appendix requires clarification. - The controlled comparison in which “only recomputation differs” is unclear. Section 6.2 states that both variants use K=5 and functional-damage evaluation. However, Figure 5 describes CARE-Static as directly merging after a one-time ranking, while candidate generation and functional evaluation are shown only for Adaptive. Appendix C further reports 115 exact evaluations for Static and exactly 16×5=80 for Adaptive, with Static being approximately 5× slower. If the only difference is recomputation in Equation 6, these discrepancies in evaluation counts and procedure need explanation; otherwise, attributing the 17.8%–24.6% gain to recomputation is unsupported.
Questions for authors
- What is the relationship between 0.001522–0.007096 and 0.046/0.074? Please provide the samples used for each figure and statistic, with consistent units.
- Please fully define
Fand probe setP, and report the correlations and final compression results under changes to the probes and calibration data. - Why does CARE-Static perform 115 functional evaluations while CARE-Adaptive performs 80? At every step, do both variants apply the same temporary merge and KL evaluation to the same K candidates?