- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper formulate MoE compression under the routed block loss and introduce two surrogates both for the co-routing interference and for the per-expert covariances. It further derives closed-form solutions under the surrogates. Results shows that the proposed method achieves better results comparing to baselines.
Critical strengths
- the block loss definition over routing MoE with two surrogates sounds OK to me, especially the consideration of between experts co-routing structure is novel in MoE modeling to my knowledge.
- It derives a closed-form solution for the shared base.
Critical weaknesses
- It compares the accuracy, and compressing time; but does not give the comparison on inference time. Does it have advantages on this point?
- What is the range of \epsilon, in Appendix-B2, it says < L < . In Table-5, you have $\epsilon_{cov} $, and its median value much larger than 1. What is the relationship between and . Does the value in Table-5 sounds?
- In Line-099, the paper states a closed related work HOPE, but does make any comparison in the experimental studies.
Questions for authors
check weakness for more comments
3 comments
Thanks for the detailed responses, it address part of my concerns. I still have some followup questions
- Is the inference cost and speed reported by original pytorch? Are there any special overhead when porting to industrial inference engine like sglang?
- As the method gives similar inference cost as D^2-MoE, while according to Table-1, the improvement over D^2-MoE is relatively marginal.
- The results are reported on general tasks, will the block-loss based method impact the long-horizon agentic related task like SWEBench, TeriminalBench on most recent agentic models?
Thank you for the follow-up questions.
Inference engine. Yes, all numbers are measured with plain PyTorch (Hugging Face Transformers), without an inference engine or a fused kernel. The compressed layer applies one shared dense matrix per projection and two low-rank matrices per active expert, all of which are ordinary matrix products. Attention, the KV cache, batching and scheduling are unchanged. To use engines such as SGLang, the factorized layer has to be added as a new layer type. This is the same for -MoE and for any low-rank method. We have not done this, so we cannot report engine numbers; the FLOPs and the number of parameters do not depend on the implementation.
Improvement over -MoE. The inference cost is the same, but the compression is cheaper in time and in GPU memory, and the clearest difference is the shared base. -MoE merges the experts with Fisher weights. This needs a backward pass over the calibration set that keeps the weights, their gradients and the Fisher accumulators in GPU memory, about three times the size of the model, so on Qwen2 it runs on four GPUs (App. D.1). Our base is a closed form (Eq. 9) of the gate statistics that the forward profiling pass collects on one GPU.
| Qwen2-57B-A14B, 40% | GPUs for the base | Base step (min) | Profiling (min) | Fit (min) | Total (min) |
|---|---|---|---|---|---|
| -MoE | 4 | 31.1 | 80.5 | 56.8 | 137.3 |
| CORTEX | 1 | inside profiling | 11.6 | 97.7 | 109.3 |
The base step is part of the profiling time; the last three columns are those of Table 2.
As for the results, the margin is small in a few settings, such as Mixtral at 40%, but it is substantial on the models with more experts, and it holds in every setting:
- Size of the gain (Table 1). On Qwen2 and DeepSeekMoE (64 experts), the PPL of CORTEX + refit is 3.4–19.1% and 9.7–22.2% lower than that of -MoE; for example, on DeepSeekMoE at 60% it is 11.18 / 22.06 / 26.37 against 14.32 / 28.35 / 33.89. The average accuracy is 0.5–2.2 points higher in every block of Table 1.
- New models in the rebuttal (reply to Reviewer Qkmd, W2/Q3). On Qwen3-30B-A3B (128 experts) and Qwen3.5-35B-A3B (256 experts), the accuracy of CORTEX is 2.2–3.0 points higher than that of -MoE in all four settings. With the refit, its PPL is 6–17% lower in all twelve columns, with accuracy still at or above that of -MoE.
- Consistency. Without the refit, CORTEX already has lower PPL than -MoE in all 18 columns of Table 1 and higher average accuracy in all six settings. -MoE is a strong baseline: none of the three more recent methods in Table 1 (MoBE, RFID-MoE, TD-MoE) improves on it consistently.
Agentic tasks. We have not evaluated agentic benchmarks, so we cannot answer this with data. The block loss itself is not tied to a task: it is fitted on WikiText-2 only, and the gains hold on PTB, C4 and the seven reasoning tasks, none of which is in the calibration set. Whether this carries over to long-horizon agentic tasks is untested.
Thank you for the review and the questions. Besides the three points below, the general comment lists the other new results (two newer checkpoints with 128 and 256 experts, repeated calibration draws, and controls that change one component at a time).
W1
CORTEX has the same deployed form as D2-MoE, except that we have a different rank allocation strategy; so at the same compression ratio the two have similar inference costs. The table compares the methods on Qwen2-57B-A14B at 40% (one GB200, bf16, forward pass over 16×2048 tokens without a KV cache); we will add it to the appendix.
Here k experts are active per token, the expert matrices are m×d, and r are the ranks. The FLOPs are theoretical counts for all methods. For MoBE, the count assumes that the combined basis matrix of each expert is precomputed and kept in memory. The second column evaluates the first with the ranks used at 40% and sums over all layers; layers that a method leaves uncompressed count as the original. The memory is the peak GPU memory allocated during this forward pass. † Evaluated as the dense reconstruction that the official code provides, so memory and speed are those of the original model.
CORTEX and D2-MoE need 0.63× the expert FLOPs of the original, and their peak GPU memory is 74 GiB. MoBE and TD-MoE save less. RFID-MoE saves none: it adds to every expert a full-size residual whose entries are tied by a hash, which reduces storage but has to be applied as a dense matrix.
Finally, we would like to highlight that the throughput is measured with plain PyTorch code. None of the implementations has an optimized kernel, so the measured numbers depend strongly on the implementation and should be read together with the FLOPs.
W2
ε ranges over [0,∞); it is not restricted to [0,1]. With λ the eigenvalues of M^−1/2MM^−1/2, ε=max(λmax−1,1−λmin). Since λmin≥0, a value above 1 always equals λmax−1. In that case the lower bound (1−ε)L^≤L is vacuous, and the upper bound L≤(1+ε)L^ remains informative.
εcov is a lower bound on the ε of the Kronecker surrogate: ε⊗≥εcov=maxkεk (Prop. 4(i)), where εk is the same quantity for expert k alone, comparing its input covariance with the pooled one.
The values of Table 5 are therefore sound: εk>1 means that along some input direction the covariance of expert k is 1+εk times the pooled one. This holds for the median expert as well as for the maximum, so the pooled covariance is a poor substitute for the per-expert ones, which is why each private residual is fitted under its own covariance (Sec. 4.2). The range of ε and the one-sided bound are now stated in App. B.2.
W3
HOPE is concurrent work: it appeared on arXiv on 16 September 2026, nine days before the submission deadline. It also addresses a different kind of compression. HOPE removes whole experts, and its objective decides which experts to delete; our method keeps every expert and compresses its weight matrices by low-rank factorization, as do the five baselines of Table 1. The two are complementary and can be combined. This would be an interesting future direction to explore, but is out of the scope of this work. We will expand this discussion in Sec. 1.2.