acceptodds
Under review as a conference paper at ICLR 2027

CORTEX: COmpressing RouTed EXperts for MoE-based LLMs under the Block Loss

Abstract

Mixture-of-Experts (MoEs) scale the sizes of LLMs, thus studying their compression becomes crucial. One distinctive feature of MoEs is that the experts of a layer are redundant in their weights and correlated in their routing: a token activates several of them at once, and the subsequent layer receives only their gated sum. Existing low-rank compression methods exploit the redundancy through parameters shared across experts, but the routing correlation rarely enters their objectives: each expert is fitted on its own inputs. In this work, we propose to fit the low-rank compression under the error of the routed MoE block, in which the errors of co-routed experts are coupled. The exact second-order metric of this loss, however, couples all experts and admits no closed-form truncation. Besides, its per-expert blocks are estimated from few tokens and overfit the calibration set. Our method therefore uses two surrogates of this metric, one keeping the co-routing interference and the other the per-expert input covariances. Under the surrogate metrics with a Stein-type shrinkage, each component of the low-rank decomposition is fitted in a closed form, and the output factors of all experts are then re-solved jointly under the exact metric. Finally, a role-split rank allocation from a first-order analysis of the block output completes the method. We demonstrate the method at –% compression on Mixtral-87B, Qwen2-57B-A14B, and DeepSeekMoE-16B-Base, where it achieves the strongest performance in language modeling and zero-shot reasoning in most settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model compression, Mixture-of-Expert, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_x9BdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulate MoE compression under the routed block loss and introduce two surrogates both for the co-routing interference and for the per-expert covariances. It further derives closed-form solutions under the surrogates. Results shows that the proposed method achieves better results comparing to baselines.

Critical strengths
  • the block loss definition over routing MoE with two surrogates sounds OK to me, especially the consideration of between experts co-routing structure is novel in MoE modeling to my knowledge.
  • It derives a closed-form solution for the shared base.
Critical weaknesses
  • It compares the accuracy, and compressing time; but does not give the comparison on inference time. Does it have advantages on this point?
  • What is the range of \epsilon, in Appendix-B2, it says (1−ϵ)L^(1-\epsilon) \hat L < L < (1+ϵ)L^(1+\epsilon) \hat L. In Table-5, you have $\epsilon_{cov} $, and its median value much larger than 1. What is the relationship between ϵ\epsilon and ϵcov\epsilon_{cov}. Does the value in Table-5 sounds?
  • In Line-099, the paper states a closed related work HOPE, but does make any comparison in the experimental studies.
Questions for authors

check weakness for more comments

3 comments

Thank you for the review and the questions. Besides the three points below, the general comment lists the other new results (two newer checkpoints with 128 and 256 experts, repeated calibration draws, and controls that change one component at a time).

W1

W1. It compares the accuracy, and compressing time; but does not give the comparison on inference time. Does it have advantages on this point?

CORTEX has the same deployed form as D2D^2-MoE, except that we have a different rank allocation strategy; so at the same compression ratio the two have similar inference costs. The table compares the methods on Qwen2-57B-A14B at 40% (one GB200, bf16, forward pass over 16×204816 \times 2048 tokens without a KV cache); we will add it to the appendix.

MethodExpert FLOPs per token and layer (multiply-adds)Expert FLOPs at 40%, ×\times originalPeak GPU memory (GiB)Forward throughput (10310^3 tokens/s)
Original3kmd3kmd1.00116.520.5
MoBEkmd+2kr(m+d)kmd + 2kr(m + d)0.78as Original†as Original†
RFID-MoE3kmd3kmd1.00as Original†as Original†
TD-MoE3(r1rmrd+d rd+k m rm)3(r_1 r_m r_d + d\, r_d + k\, m\, r_m)0.9776.53.1
D2D^2-MoE3md+k(m+d)(rgate+rup+rdown)3md + k(m + d)(r_{\text{gate}} + r_{\text{up}} + r_{\text{down}})0.6374.023.8
CORTEXas D2D^2-MoE0.6373.823.1

Here kk experts are active per token, the expert matrices are m×dm \times d, and rr are the ranks. The FLOPs are theoretical counts for all methods. For MoBE, the count assumes that the combined basis matrix of each expert is precomputed and kept in memory. The second column evaluates the first with the ranks used at 40% and sums over all layers; layers that a method leaves uncompressed count as the original. The memory is the peak GPU memory allocated during this forward pass. † Evaluated as the dense reconstruction that the official code provides, so memory and speed are those of the original model.

CORTEX and D2D^2-MoE need 0.63×0.63\times the expert FLOPs of the original, and their peak GPU memory is 74 GiB. MoBE and TD-MoE save less. RFID-MoE saves none: it adds to every expert a full-size residual whose entries are tied by a hash, which reduces storage but has to be applied as a dense matrix.

Finally, we would like to highlight that the throughput is measured with plain PyTorch code. None of the implementations has an optimized kernel, so the measured numbers depend strongly on the implementation and should be read together with the FLOPs.

W2

W2. What is the range of ϵ\epsilon, in Appendix-B2, it says (1−ϵ)L^(1-\epsilon)\hat{L} < L < (1+ϵ)L^(1+\epsilon)\hat{L}. In Table-5, you have ϵcov\epsilon_{cov}, and its median value much larger than 1. What is the relationship between ϵ\epsilon and ϵcov\epsilon_{cov}. Does the value in Table-5 sounds?

ε\varepsilon ranges over [0,∞)[0, \infty); it is not restricted to [0,1][0, 1]. With λ\lambda the eigenvalues of M^−1/2MM^−1/2\hat{\boldsymbol{M}}^{-1/2} \boldsymbol{M} \hat{\boldsymbol{M}}^{-1/2}, ε=max⁡(λmax⁡−1, 1−λmin⁡)\varepsilon = \max(\lambda_{\max} - 1,\, 1 - \lambda_{\min}). Since λmin⁡≥0\lambda_{\min} \ge 0, a value above 1 always equals λmax⁡−1\lambda_{\max} - 1. In that case the lower bound (1−ε) L^≤L(1 - \varepsilon)\,\hat{\mathcal{L}} \le \mathcal{L} is vacuous, and the upper bound L≤(1+ε) L^\mathcal{L} \le (1 + \varepsilon)\,\hat{\mathcal{L}} remains informative.

εcov\varepsilon_{\text{cov}} is a lower bound on the ε\varepsilon of the Kronecker surrogate: ε⊗≥εcov=max⁡kεk\varepsilon_{\otimes} \ge \varepsilon_{\text{cov}} = \max_k \varepsilon_k (Prop. 4(i)), where εk\varepsilon_k is the same quantity for expert kk alone, comparing its input covariance with the pooled one.

The values of Table 5 are therefore sound: εk>1\varepsilon_k > 1 means that along some input direction the covariance of expert kk is 1+εk1 + \varepsilon_k times the pooled one. This holds for the median expert as well as for the maximum, so the pooled covariance is a poor substitute for the per-expert ones, which is why each private residual is fitted under its own covariance (Sec. 4.2). The range of ε\varepsilon and the one-sided bound are now stated in App. B.2.

W3

W3. In Line-099, the paper states a closed related work HOPE, but does make any comparison in the experimental studies.

HOPE is concurrent work: it appeared on arXiv on 16 September 2026, nine days before the submission deadline. It also addresses a different kind of compression. HOPE removes whole experts, and its objective decides which experts to delete; our method keeps every expert and compresses its weight matrices by low-rank factorization, as do the five baselines of Table 1. The two are complementary and can be combined. This would be an interesting future direction to explore, but is out of the scope of this work. We will expand this discussion in Sec. 1.2.

Thanks for the detailed responses, it address part of my concerns. I still have some followup questions

  • Is the inference cost and speed reported by original pytorch? Are there any special overhead when porting to industrial inference engine like sglang?
  • As the method gives similar inference cost as D^2-MoE, while according to Table-1, the improvement over D^2-MoE is relatively marginal.
  • The results are reported on general tasks, will the block-loss based method impact the long-horizon agentic related task like SWEBench, TeriminalBench on most recent agentic models?

Thank you for the follow-up questions.

Inference engine. Yes, all numbers are measured with plain PyTorch (Hugging Face Transformers), without an inference engine or a fused kernel. The compressed layer applies one shared dense matrix per projection and two low-rank matrices per active expert, all of which are ordinary matrix products. Attention, the KV cache, batching and scheduling are unchanged. To use engines such as SGLang, the factorized layer has to be added as a new layer type. This is the same for D2D^2-MoE and for any low-rank method. We have not done this, so we cannot report engine numbers; the FLOPs and the number of parameters do not depend on the implementation.

Improvement over D2D^2-MoE. The inference cost is the same, but the compression is cheaper in time and in GPU memory, and the clearest difference is the shared base. D2D^2-MoE merges the experts with Fisher weights. This needs a backward pass over the calibration set that keeps the weights, their gradients and the Fisher accumulators in GPU memory, about three times the size of the model, so on Qwen2 it runs on four GPUs (App. D.1). Our base is a closed form (Eq. 9) of the gate statistics that the forward profiling pass collects on one GPU.

Qwen2-57B-A14B, 40%GPUs for the baseBase step (min)Profiling (min)Fit (min)Total (min)
D2D^2-MoE431.180.556.8137.3
CORTEX1inside profiling11.697.7109.3

The base step is part of the profiling time; the last three columns are those of Table 2.

As for the results, the margin is small in a few settings, such as Mixtral at 40%, but it is substantial on the models with more experts, and it holds in every setting:

  • Size of the gain (Table 1). On Qwen2 and DeepSeekMoE (64 experts), the PPL of CORTEX + refit is 3.4–19.1% and 9.7–22.2% lower than that of D2D^2-MoE; for example, on DeepSeekMoE at 60% it is 11.18 / 22.06 / 26.37 against 14.32 / 28.35 / 33.89. The average accuracy is 0.5–2.2 points higher in every block of Table 1.
  • New models in the rebuttal (reply to Reviewer Qkmd, W2/Q3). On Qwen3-30B-A3B (128 experts) and Qwen3.5-35B-A3B (256 experts), the accuracy of CORTEX is 2.2–3.0 points higher than that of D2D^2-MoE in all four settings. With the refit, its PPL is 6–17% lower in all twelve columns, with accuracy still at or above that of D2D^2-MoE.
  • Consistency. Without the refit, CORTEX already has lower PPL than D2D^2-MoE in all 18 columns of Table 1 and higher average accuracy in all six settings. D2D^2-MoE is a strong baseline: none of the three more recent methods in Table 1 (MoBE, RFID-MoE, TD-MoE) improves on it consistently.

Agentic tasks. We have not evaluated agentic benchmarks, so we cannot answer this with data. The block loss itself is not tied to a task: it is fitted on WikiText-2 only, and the gains hold on PTB, C4 and the seven reasoning tasks, none of which is in the calibration set. Whether this carries over to long-horizon agentic tasks is untested.

Reviewer_QkmdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper studies training-free low-rank compression of MoE experts under a routing-weighted reconstruction objective. Building on a shared dense base with expert-specific low-rank residuals, it proposes a routed-centroid base, covariance-shrunk expert truncations, joint refitting of down-projection output factors, and sensitivity-inspired rank allocation across projection roles. Experiments cover three MoE models, three language-modeling corpora, and seven zero-shot tasks. The method improves the quality–parameter tradeoff over several baselines, particularly on Qwen2 and DeepSeekMoE at high compression ratios.

Critical strengths
  1. A relevant MoE-specific objective. Errors from co-activated experts are summed before reaching downstream layers, so independent expert reconstruction can miss error cancellation or amplification. Explicitly considering this interaction is a useful perspective, and the projected refit provides a tractable implementation.

  2. Useful improvements on the evaluated models. At 60% parameter removal on DeepSeekMoE, perplexities improve from 14.32/28.35/33.89 for D²-MoE to 11.18/22.06/26.37, with average accuracy increasing from 40.7 to 42.9. The component ladders and calibration-size experiments are informative, and the paper reports settings where refitting or stronger role allocation does not help.

Critical weaknesses
  1. The incremental contribution of the routed formulation is insufficiently established. Proposition 1 expresses a Kronecker quadratic form as a whitened norm; the closed-form base in Proposition 2 is a directly derived weighted mean; and Equation (13) is regularized least squares once the input factors are fixed. I view these as standard tools applied to the routed objective, rather than independent theoretical advances. Their combination may be useful, but the experiments do not adequately isolate the benefit of the paper’s central distinction: cross-expert coupling. As Appendix D.2 explains, refitting can also compensate for errors introduced by the compressed gate/up projections by targeting the original block output. A matched expertwise refit using the same compressed intermediate activations, retained subspaces, ranks, and ridge-selection procedure would quantify the additional benefit attributable specifically to coupling. The existing component ablations do not provide this comparison.

    The connection between some theoretical statements and the implemented method also needs clarification. Proposition 2(ii) explicitly requires centered residual approximations, but Algorithm 1 does not enforce this condition; it is unclear which conclusions from that decomposition apply to the truncated factors. More directly, Proposition 4 bounds metric mismatch and does not establish the claimed bound on refit gains in Sections 4.3 and 6.

  2. Generalization beyond the three older checkpoints remains untested. All evaluated checkpoints date to 2023–2024: Mixtral-8×7B, Qwen2-57B-A14B, and DeepSeekMoE-16B-Base. These choices enable controlled comparisons and cover different routing regimes, but provide limited evidence of transfer to newer checkpoints and larger expert pools. Closely related methods already include such evaluations: MoBE (ICLR 2026) evaluates Qwen3-family models, DeepSeek-V3-0324, and Kimi-K2, while RFID-MoE (ICML 2026) also evaluates Qwen3. This gap matters because CORTEX relies on routing correlations and expert-specific covariance estimates, whose estimation and usefulness can depend on expert-pool size and activation patterns. Evaluating the language backbone of a newer MoE checkpoint, such as Qwen3.6-35B-A3B with its 256 routed experts, would materially strengthen the evidence for the method's generality.

Questions for authors
  1. How much additional benefit does joint refitting provide over a matched expertwise refit using the same compressed gate/up activations, input factors, ranks, and ridge-selection procedure? Could you also report block error on a separate set of windows not used to select the ridge strength, since Figures 1 and 4 report errors on the ridge-selection windows?

  2. Which conclusions from the conditional orthogonal decomposition in Proposition 2(ii) apply to the truncated factors, given that Algorithm 1 does not explicitly enforce centering? Please also clarify how Proposition 4 supports the stated bound on stage-two gains, rather than only a bound on metric mismatch.

  3. Does the advantage over the closest low-rank baselines persist when compressing the language backbone of a newer MoE checkpoint, such as Qwen3.6-35B-A3B? Such a comparison would help establish whether the method transfers beyond the evaluated 8- and 64-expert checkpoints to larger expert pools.

5 comments

Thank you for the careful reading, in particular of the propositions.

W1a / Q1

W1. The incremental contribution of the routed formulation is insufficiently established. Proposition 1 expresses a Kronecker quadratic form as a whitened norm; the closed-form base in Proposition 2 is a directly derived weighted mean; and Equation (13) is regularized least squares once the input factors are fixed. I view these as standard tools applied to the routed objective, rather than independent theoretical advances. Their combination may be useful, but the experiments do not adequately isolate the benefit of the paper’s central distinction: cross-expert coupling.

Q1. How much additional benefit does joint refitting provide over a matched expertwise refit using the same compressed gate/up activations, input factors, ranks, and ridge-selection procedure? Could you also report block error on a separate set of windows not used to select the ridge strength, since Figures 1 and 4 report errors on the ridge-selection windows?

We ran the experiment of using expert-wise refit. Results are shown in the following table. At the block level the cross-expert coupling helps on every model, in nearly every layer. End to end, at both ratios, most of the gain of the refit comes from re-solving the down factors against the original block output with the compressed gate/up in the loop, as you suspected, and the cross-expert terms add a smaller, model-dependent part. This corrects our explanation of the refit. However, it does not alter the main claim of the paper: the improvement over D2D^2-MoE and other baselines.

Revision. Secs. 4.3, 5.2, 5.3 and 6 attributed the gain of the refit to the cross terms. They are changed as follows.

  • Sec. 4.3: stage two is no longer said to "restore the cross terms". It re-solves the down factors under the exact metric of the block output, which changes three things at once: the factors are fitted to the original block output with the compressed gate/up in the loop, the diagonal blocks are exact, and the cross terms are included.
  • Secs. 5.2, 5.3 and 6: the statements that the gain follows the co-routing mass of Γ\boldsymbol{\Gamma}, or is bounded by it, are removed. Secs. 5.3 and 6 now state the result of this control: most of the gain does not come from the cross terms.
  • The table below is added as Table 13 (new App. G).
PPL (WikiText-2 / PTB / C4), acc.Stage one+ refit+ expert-wise refitBlock err. (+ refit / + expert-wise refit)
Mixtral-8$\times$7B, 40%5.41 / 21.07 / 9.55, 57.35.42 / 20.88 / 9.60, 56.95.41 / 20.93 / 9.58, 57.2−0.6% / −0.3%
Qwen2-57B-A14B, 40%6.84 / 10.75 / 12.05, 58.16.80 / 10.64 / 11.76, 58.46.80 / 10.58 / 11.81, 58.2−5.4% / −3.4%
DeepSeekMoE-16B, 40%8.56 / 13.16 / 14.92, 50.18.18 / 12.76 / 14.45, 50.18.30 / 12.84 / 14.60, 50.4−6.4% / −2.6%
Qwen2-57B-A14B, 60%10.18 / 18.97 / 20.76, 51.49.24 / 15.72 / 19.11, 51.59.19 / 15.81 / 19.77, 51.8−10.0% / −6.8%
DeepSeekMoE-16B, 60%13.47 / 25.40 / 29.43, 42.311.18 / 22.06 / 26.37, 42.911.46 / 22.50 / 26.76, 42.7−8.3% / −4.3%
  • Block level. The coupling helps on every model: the coupled refit has a lower held-out error than the per-expert refit in 31/31, 21/23 and 27/27 layers at 40%, and in 21/24 (Qwen2) and 27/27 (DeepSeekMoE) layers at 60%. Last column: median per-layer change of the held-out error of Figs. 1 and 4, relative to stage one.
  • End to end. On DeepSeekMoE the coupling lowers PPL by a further 1.4 / 0.6 / 1.0% beyond the expert-wise refit at 40% and by 2.4 / 2.0 / 1.5% at 60%. On Mixtral it adds nothing. On Qwen2 it adds nothing at 40% and is mixed at 60%: PPL is 3.3% lower on C4 and 0.5% lower on PTB, but 0.5% higher on WikiText-2, and the expert-wise refit has the higher accuracy.
  • Windows not used for selection. Figs. 1 and 4 do report the error on the 32 windows that select λ\lambda. On text that neither calibration nor selection has seen (WikiText-2 test, PTB test and C4 validation, 32 windows each), the refit lowers the block-output error by 2.6% on Mixtral, 0.9% on Qwen2 and 2.0% on DeepSeekMoE at 40%, summed over all layers (lower in 31/32, 22/24 and 26/27 layers).
  • Standard tools. We agree that Props. 1–2 and Eq. (13) are standard once the objective is fixed, and we do not claim them as independent theory. What is specific to the paper is the routed block objective and the choice of surrogate for each component; our reply to Reviewer qGQe (W1/Q1) changes one component at a time and measures what each choice contributes (on DeepSeekMoE, the base alone gives −4.9 / −6.3 / −8.3% PPL and the metric with shrinkage a further −1.5 / −3.3 / −5.4%).

W1b / Q2 (first part)

W1. The connection between some theoretical statements and the implemented method also needs clarification. Proposition 2(ii) explicitly requires centered residual approximations, but Algorithm 1 does not enforce this condition; it is unclear which conclusions from that decomposition apply to the truncated factors.

Q2. Which conclusions from the conditional orthogonal decomposition in Proposition 2(ii) apply to the truncated factors, given that Algorithm 1 does not explicitly enforce centering?

You are right. Prop. 2(ii) holds only for routing-centered residual approximations, and the truncated factors of Alg. 1 are not centered. For the truncated factors:

  • What still holds. Part (i), and the fact that with B=B⋆\boldsymbol{B} = \boldsymbol{B}^{\star} the surrogate loss equals ∥R−R^∥2\lVert \boldsymbol{R} - \hat{\boldsymbol{R}} \rVert^2, so every residual is fitted against the right target.
  • What does not hold. B⋆\boldsymbol{B}^{\star} is no longer exactly the optimal base, and our sentence "base and residuals do not compete for the same error directions" is too strong. Given the truncated R^\hat{\boldsymbol{R}}, the optimal base is B⋆−∑kckR^k\boldsymbol{B}^{\star} - \sum_k c_k \hat{\boldsymbol{R}}_k, a closed-form shift that costs no parameters. Summed over three layers of each model at 40%, this shift would lower the Kronecker surrogate loss by 4–12%. The reported results use B⋆\boldsymbol{B}^{\star} and do not include it.

In the revised PDF, Prop. 2(ii) states explicitly that the split requires a centered approximation, the sentence after Prop. 2 is restricted to centered approximations, and the corrected base is added with this measurement as a remark (App. A).

W1c / Q2 (second part)

W1. More directly, Proposition 4 bounds metric mismatch and does not establish the claimed bound on refit gains in Sections 4.3 and 6.

Q2. Please also clarify how Proposition 4 supports the stated bound on stage-two gains, rather than only a bound on metric mismatch.

We agree. Prop. 4(ii) bounds the mismatch of the block-diagonal surrogate, that is, the weight that the dropped cross terms can carry relative to the diagonal blocks. It does not bound the gain of the refit: such a bound would also need the smallest eigenvalue of the relative metric, which ktopk_{\text{top}} does not control, and the refit also changes more than the cross terms. The corresponding sentences in Secs. 4.3 and 6 are revised accordingly: Prop. 4(ii) limits the weight of the cross terms that stage one drops, which gives an expectation of where the refit can help, and the measured contribution of these terms is the control above.

W2 / Q3

W2. Generalization beyond the three older checkpoints remains untested.

Q3. Does the advantage over the closest low-rank baselines persist when compressing the language backbone of a newer MoE checkpoint, such as Qwen3.6-35B-A3B?

We ran two newer checkpoints with the protocol of Table 1 and the same hyper-parameters:

  • Qwen3-30B-A3B (128 routed experts, top-8, 48 MoE layers), the Qwen3 model that MoBE and RFID-MoE evaluate.
  • Qwen3.5-35B-A3B (256 routed experts and 1 shared expert, top-8, 40 MoE layers, Gated DeltaNet / attention hybrid). It has the architecture of Qwen3.6-35B-A3B.
ModelMethod40%: WikiPTBC4Acc.60%: WikiPTBC4Acc.
Qwen3-30B-A3BOriginal6.0910.4210.8563.5
D2D^2-MoE10.1119.3723.2552.318.2942.7454.7741.3
MoBE15.8927.4528.2455.1NANANANA
RFID-MoE10.0517.4520.4555.2NANANANA
CORTEX10.0418.5821.5855.318.5244.1052.3943.7
CORTEX + refit9.4117.7320.9154.015.3039.9446.4941.5
Qwen3.5-35B-A3BOriginal5.7310.1510.6166.6
D2D^2-MoE8.6115.2017.8355.914.5329.6136.9045.0
MoBE13.1523.2422.4653.5NANANANA
RFID-MoE10.2319.5721.0753.6NANANANA
CORTEX8.5815.0817.3258.314.9830.9537.1847.2
CORTEX + refit8.1214.1716.4457.612.2724.6131.2046.0

MoBE and RFID-MoE compress only the gate/up projections, so 60% is out of their reach, as in Table 1.

  • Against D2D^2-MoE, the closest baseline. Stage one has higher accuracy in all four settings. Its PPL is lower at 40% (by up to 7%) and within 5% of D2D^2-MoE at 60% (higher in five of the six columns). With the refit, PPL is lower than D2D^2-MoE in all twelve columns, by 6–17%, and accuracy is higher by up to 1.7 (a tie on Qwen3 at 60%).
  • Against MoBE and RFID-MoE (40%). MoBE's PPL is 1.2–1.6$\times$ that of D2D^2-MoE and of CORTEX on both checkpoints; its accuracy ties our stage-one row on Qwen3 and is 4.8 points lower on Qwen3.5. RFID-MoE is the strongest baseline on Qwen3: it ties our stage-one row on WikiText-2 and accuracy and has 5–6% lower PPL on PTB and C4; against the refit row it is 6.7% higher on WikiText-2, 1.5–2.2% lower on PTB and C4, and 1.2 points higher in accuracy. On Qwen3.5 it is behind both D2D^2-MoE and CORTEX (PPL +18–30%, acc. −2.3 and −4.7).
  • The refit is a PPL/accuracy trade-off on these models. It lowers PPL by 3–6% at 40% and 9–21% at 60%, and costs 0.8–1.3 accuracy points at 40% and 1.1–2.3 at 60%. We report both rows.

So the advantage over D2D^2-MoE holds on 128- and 256-expert checkpoints. We will add these results to the paper, as a new table in the appendix with a paragraph in Sec. 5.2.

Reviewer_qGQeOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CORTEX studies training-free low-rank compression of MoE language models using the routed block-output error. It retains a shared-base and expert-specific low-rank residual representation, fits these components using structured surrogate metrics and covariance shrinkage, and optionally jointly refits the down-projection output factors. It also introduces a role-based rank allocation strategy. Experiments on Mixtral, Qwen2, and DeepSeekMoE evaluate perplexity and zero-shot accuracy under several whole-model compression ratios.

Critical strengths
  1. The routed block-loss formulation has a clear motivation: the errors of co-routed experts can cancel or accumulate in their weighted sum. Accounting for this interaction provides a principled direction for improving expert compression.

  2. The method produces meaningful perplexity improvements in several high-compression settings. The evaluation includes multiple routing configurations and component ablations, providing useful evidence about when shrinkage and the coupled refit are beneficial.

Critical weaknesses
  1. The methodological advance is incremental, and its central contributions are not fully isolated experimentally. CORTEX retains the shared-base and low-rank residual structure of D²-MoE. Although the routed fitting objective and associated solutions are distinct, the first CORTEX configuration in Tables 3 and 10 simultaneously changes both the shared-base construction and the per-expert fitting metric. This makes it difficult to establish the separate contribution of the routed centroid. The complete pipeline also includes shrinkage, rank allocation, and checkpoint-specific live-unit handling, so its overall improvement cannot be attributed solely to cross-expert interaction modeling.

  2. The evidence for a consistently worthwhile practical improvement remains incomplete. Several average accuracy gains are modest, without uncertainty estimates, and the coupled refit lowers accuracy on Mixtral at both 40% and 60% compression despite improving the calibration-side block error. Furthermore, the reproducibility statement promises code release upon publication, but no implementation accompanies the provided review materials. Given the implementation details involved in covariance estimation, rank allocation, live-unit handling, and held-out refit selection, this limits independent verification of the reported gains.

Overall, I lean toward weak rejection. The approach is well motivated and some perplexity improvements are meaningful, but stronger isolation of the methodological contributions and more verifiable evidence of robust gains are needed to establish sufficient significance.

Questions for authors
  1. Could the authors separately evaluate the routed-centroid base and the gate-weighted per-expert metric, changing one component at a time relative to D²-MoE? This would help establish which proposed changes account for the improvement.

  2. How robust are the smaller accuracy gains to calibration-set sampling and evaluation uncertainty? Please also clarify how a practitioner should decide whether to use the coupled refit, given that lower held-out block error does not consistently translate into better downstream performance on Mixtral.

  3. Could the authors provide an anonymized implementation or a minimal reproducible package during review, including the rank-allocation procedure, live-unit handling, and held-out hyperparameter selection? This would materially improve the ability to verify the main results.

3 comments

Thank you for the detailed review.

W1 / Q1

W1. The methodological advance is incremental, and its central contributions are not fully isolated experimentally.

Q1. Could the authors separately evaluate the routed-centroid base and the gate-weighted per-expert metric, changing one component at a time relative to D²-MoE? This would help establish which proposed changes account for the improvement.

Our contribution is not a new compressed structure, but a more effective, routing-aware fitting objective for MoE layers. The base, the per-expert metric and the refit are all derived from this one objective (Eq. 5), each with the approximation that suits the component. The shrinkage is part of the covariance estimate in the per-expert metric (Eq. 11) and is essential to avoid over-fitting. The live-unit cap is part of the rank allocation (Eq. 18, App. C.2): Qwen2 has dead hidden units, so the rank of an expert is capped at its number of live units. It is read from the checkpoint and is used on Qwen2 only. The paper ablates the shrinkage, the rank allocation and the cap one at a time (Tables 3, 4 and 10, Fig. 2).

To isolate the effects of the routed-centroid base and the gate-weighted per-expert metric and answer Q1, we conduct the following experiment, in which every row changes one component, starting from D2D^2-MoE (Fisher-merged base; each residual truncated under the covariance of the tokens routed to its expert), at the same structure, ranks and parameter count: row 1 → 2 changes only the shared base (Sec. 4.1), and row 2 → 3 changes only the fitting of the per-expert residuals (Sec. 4.2, Eq. 11 with μ=0.5\mu = 0.5) at a fixed base. Row 2 is new; the first CORTEX row of Tables 3 and 10 changed both together.

40% compression

Model#Shared basePer-expert residualsWikiPTBC4Acc.
DeepSeekMoE-16B1D2D^2-MoED2D^2-MoE9.0614.4416.9548.4
2CORTEXD2D^2-MoE8.6213.5315.5549.7
3CORTEXCORTEX8.4913.0914.7149.9
Mixtral-8$\times$7B1D2D^2-MoED2D^2-MoE5.4522.689.7256.2
2CORTEXD2D^2-MoE5.4521.919.6556.9
3CORTEXCORTEX5.4322.139.5957.2
Qwen2-57B-A14B1D2D^2-MoED2D^2-MoE7.0411.1512.9057.6
2CORTEXD2D^2-MoE7.0311.1512.8557.7
3CORTEXCORTEX7.0111.0912.4957.7
  • Base alone (row 1 → 2). On DeepSeekMoE, using our metric on the shared base alone decreases the PPL by 4.9 / 6.3 / 8.3% and increases the accuracy by 1.3 points. On Mixtral and Qwen2, the swap is a tie or a smaller gain. The reason is the form of the base. D2D^2-MoE merges the experts entry by entry with Fisher weights, which ignores the input correlations that the block loss measures; under the routed objective the optimal base is a scalar mixture of the experts (Prop. 2).
  • Per-expert residuals alone, at a fixed base (row 2 → 3). The residuals are fitted under Eq. 11, which is one estimator: the gate-weighted covariance of each expert (the exact block of the routed loss), shrunk toward the pooled routed covariance. Comparing row 3 and row 2, we show that it further reduces the PPL and increases the acc.

W2 / Q2 (first part)

W2. The evidence for a consistently worthwhile practical improvement remains incomplete. Several average accuracy gains are modest, without uncertainty estimates [...]

Q2. How robust are the smaller accuracy gains to calibration-set sampling and evaluation uncertainty?

Calibration sampling and evaluation uncertainty. The compression pipeline mainly relies on SVD and has no random step other than the draw of the calibration dataset. The evaluation has no random step either: perplexity and accuracy are both computed from the model's log-likelihoods on fixed evaluation sets, and a multiple-choice question is answered by the option with the highest log-likelihood. The calibration sampling is therefore the only random step behind the reported numbers. We repeated the DeepSeekMoE-16B experiment at 40% with three independent draws, for CORTEX and for D2D^2-MoE:

DeepSeekMoE-16B, 40%WikiPTBC4Avg. acc.
D2D^2-MoE9.04 ±\pm 0.0214.35 ±\pm 0.0916.87 ±\pm 0.0748.7 ±\pm 0.3
CORTEX8.59 ±\pm 0.0413.16 ±\pm 0.0014.88 ±\pm 0.0450.1 ±\pm 0.1
CORTEX + refit8.20 ±\pm 0.0312.78 ±\pm 0.0214.42 ±\pm 0.0350.2 ±\pm 0.1

The spread is far below the gap. We will add this table to the paper.

W2 / Q2 (second part)

W2. [...] the coupled refit lowers accuracy on Mixtral at both 40% and 60% compression despite improving the calibration-side block error.

Q2. Please also clarify how a practitioner should decide whether to use the coupled refit, given that lower held-out block error does not consistently translate into better downstream performance on Mixtral.

We agree with the observation on Mixtral. The refit minimizes the block error on calibration text, so it mainly helps perplexity, and accuracy follows only loosely. We recommend it when many experts are active per token and not under top-2 routing. This can be checked before any evaluation. Moreover, we would like to remark that the refit does not repeat the compression: it starts from the stage-one factors and re-solves only the down factors, with one more forward pass over the calibration set. On one GPU it takes 7–18 minutes for the three models of the paper.

On Qwen2 and DeepSeekMoE, the refit lowers every PPL by up to 17% at unchanged or higher accuracy. On the two new checkpoints it lowers PPL by 3–21% but costs 0.8–2.3 accuracy points (reply to Reviewer Qkmd, W2/Q3). The stage-one row needs no such decision: it already has higher average accuracy than D2D^2-MoE in every block we ran (11 of 11). We will state this rule in Sec. 4.3 and keep reporting both rows.

W2 / Q3

W2. Furthermore, the reproducibility statement promises code release upon publication, but no implementation accompanies the provided review materials.

Q3. Could the authors provide an anonymized implementation or a minimal reproducible package during review, including the rank-allocation procedure, live-unit handling, and held-out hyperparameter selection?

Due to the limited time, we could not organize the code during the rebuttal period. We promise to release the code, including detailed settings to fully reproduce all results if the paper is accepted.

Dear Reviewer qGQe,

Thank you again for your time and constructive feedback. As the discussion period will close soon, we would like to kindly ask whether our response has addressed your concerns. If it has, we would be grateful if you would re-consider your score. If any questions remain, we would be glad to clarify them before the deadline.

Best regards

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of new results

We thank all reviewers for their careful and constructive reviews. Below, we summarize new results added in the rebuttal to address the concerns; the details are in the reply to the reviewer who raised each point.

  1. Each component changed one at a time relative to D2D^2-MoE (reply to Reviewer qGQe, W1/Q1). On DeepSeekMoE at 40%, the base and the per-expert metric each lower the perplexity when changed alone, so the gain does not come from a single component. The shared base part contributes the most. The per-expert metric works only as a whole, that is, together with the shrinkage (Eq. 11). The same study on Mixtral and Qwen2 shows similar effects.
  2. Expert-wise refit experiment (reply to Reviewer Qkmd, W1/Q1). The cross-expert coupling lowers the held-out block error in almost all layers of the three models. End to end it lowers the perplexity by a further 0.6–2.4% on DeepSeekMoE, adds nothing on Mixtral, and is mixed on Qwen2. Secs. 4.3, 5.3 and 6 are revised according to this result.
  3. Two newer checkpoints (reply to Reviewer Qkmd, W2/Q3). We test Qwen3-30B-A3B (128 experts) and Qwen3.5-35B-A3B (256 experts, the architecture of Qwen3.6-35B-A3B), with D2D^2-MoE, MoBE and RFID-MoE as baselines. CORTEX has higher accuracy than D2D^2-MoE in all settings, and with the refit its perplexity is lower in all twelve columns. RFID-MoE is on par with CORTEX on Qwen3-30B-A3B and behind it on Qwen3.5-35B-A3B.
  4. Uncertainty (reply to Reviewer qGQe, W2/Q2). We test three independent calibration draws for CORTEX and D2D^2-MoE on DeepSeekMoE at 40%. The standard deviations are far below the gaps.
  5. Inference cost (reply to Reviewer x9Bd, W1). CORTEX has the same deployed form as D2D^2-MoE: on Qwen2 at 40%, 0.63×0.63\times the expert FLOPs of the original model and 74 GiB of peak GPU memory against 116.5 GiB.
  6. Clarifications on Props. 2 and 4 (replies to Reviewer Qkmd, W1/Q2 and Reviewer x9Bd, W2). Prop. 4 is no longer used as a bound on the refit gain, the role of centering in Prop. 2(ii) is clarified, and the range of ε\varepsilon is stated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.