- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper formulates MoE compression as minimizing model size subject to reconstruction-error and compute constraints. PRIMO first constructs a shared base through a routing-weighted KL quadratic program, then compresses expert residuals using activation-weighted low-rank factors and rank search, builds a discrete size–error frontier, and selects a configuration according to the routing KL shift between the target task and the calibration set. Experiments cover four MoE models, multiple compression ratios, and 12 tasks.
Critical strengths
- Jointly considering a shared base, residual compression, size/error constraints, and task-distribution shift is an appealing problem formulation.
- The experiments cover four architectures and report perplexity, nine accuracy metrics, throughput, memory, ablations, and offline cost.
- The appendix provides the QP, the routing-shift bound, algorithm pseudocode, complexity analysis, and extensive sensitivity studies. The authors also candidly acknowledge the absence of calibration-budget and mixed-domain-batch experiments.
Critical weaknesses
- The purported PGD low-rank optimization is redundant under the paper's own objective. Let
B=ΔW SandṼ=S V; Equation 9 is then the standard fixed-rank problemmin ||B-U Ṽᵀ||²_F. Appendix A.8.5 also acknowledges that truncated SVD is globally optimal in the Eckart–Young sense. Algorithm 1 initializes with that truncated SVD, so it is already at a global optimum of the fixed-rank objective, the gradient should be zero, and subsequent PGD should not further reduce the same objective (unless the randomized SVD implementation is only an approximate solution, in which case the paper does not reformulate the argument accordingly). The 0.35 PPL improvement attributed to “PGD” in Tables 4/17 likely comes from different rank/threshold search rather than the PGD iterations themselves. This contradiction directly affects the core methodological contribution and technical correctness. - Under the stated formulas, the purported MI-constrained QP reduces to a routing-frequency-weighted average. Equations 6–7 imply
b=Qp, wherepis the routing-frequency vector on the probability simplex. Therefore,α=psatisfies the first-order optimality condition and is a global optimum of the positive-semidefinite quadratic objective, although the solution may be non-unique in the singular case. The resulting shared base is simplyW_b=Σ_i p_iW_i. The paper needs to explain what this QP adds beyond ordinary frequency-weighted merging. - The large gains from adaptive selection are not resource-matched. In Tables 5/18, the adaptive strategy selects more permissive, less compressed configurations, but the final parameter count, rank, memory, FLOPs, and compression ratio selected for each task are not reported. The HumanEval improvement from 21.5 to 33.5 may primarily result from deploying a larger model. Without comparison to static, routing-independent, and oracle selection at the same size, the effectiveness of the routing rule itself is not established.
- The theoretical bound for adaptive selection uses an error metric different from the algorithm's actual metric. Equation 10 and line 37 of Algorithm 1 use the unweighted mean
L(Θ)over layers and experts; Equations 97–104 in Appendix A.10.2 instead prove a result forL_ref, weighted by the reference routing distribution. The paper neither proves equivalence nor gives a conversion bound, so Equation 19 does not guarantee that the configuration actually selected by the algorithm satisfies the claimed target-task error bound. - Constants required by the theoretical guarantee are not verified, and the SNR notation is inconsistent. The error bound in Appendix A.10 requires the
ζin Equation 104 to satisfyζ ≥ ζ̄, but all models use a fixedζ=1.0without computing the actualζ̄. Appendix A.7.4 empirically tests only Gaussianity and the error range, and claims that 128 samples are sufficient, but does not report an empirical test of the SNR condition. Equations 52–53 first defineSNRas||G||²₂, but the derivation subsequently uses||G||₂=SNR; the correct relationship should involve a square root. Page 43 also gives a KL budgetδ≤0.1I(X;Y), but Equation 8 and Algorithm 1 merely minimize KL without explicitly enforcing or verifying this constraint. Moreover,ρin the MI bound diverges as reconstruction error approaches zero, potentially making the bound very loose. - “Pareto-optimal” refers only to nondominated points on a 4×4 discrete grid. Appendix A.9.5 explicitly states that its theorem guarantees only nondominance among evaluated configurations. This is not Pareto optimality over the continuous feasible region of the original problem. The title and abstract risk overstating the result.
Questions for authors
- Since truncated SVD is already the fixed-rank global optimum of Equation 9, is the PGD gradient zero at initialization? Of the gain in Table 17, how much is attributable to PGD iterations and how much to rank selection?
- Is
α=p, derived fromb=Qp, the solution returned by the implementation? If so, what is the substantive difference between MI-QP and frequency-weighted averaging? - How many parameters and FLOPs, and what compression ratio, are selected for each task in Tables 5/18? At matched resources, does adaptive selection still outperform static and random/task-label-only selection?
- How does the unweighted
L(Θ)used by the algorithm satisfy Equation 104, which the appendix derives for routing-weightedL_ref?
4 comments
(continued)
Table R2: Size deployed per task and score under each policy on Mixtral-8×7B at the 40% target; the last four rows use the adaptive mean size and are read from score curves.
| Row | WikiText-2 PPL | C4 PPL | MMLU acc (%) | GSM8K acc (%) | HumanEval Pass@1 (%) | Mean parameters (B) |
|---|---|---|---|---|---|---|
| Routing shift D_shift | 0.12 | 0.00 | 0.35 | 0.47 | 0.61 | - |
| Adaptive: compression ratio (%) | 31.3 | 40.0 | 23.2 | 23.2 | 23.2 | - |
| Adaptive: parameters (B) | 32.1 | 28.0 | 35.8 | 35.8 | 35.8 | 33.5 |
| Adaptive: rank ratio | 0.576 | 0.486 | 0.659 | 0.659 | 0.659 | - |
| Adaptive: parameters read per token (B) | 9.2 | 8.2 | 10.2 | 10.2 | 10.2 | - |
| Adaptive: weight memory, bfloat16 (GiB) | 59.8 | 52.2 | 66.7 | 66.7 | 66.7 | - |
| Static | 4.58 | 8.23 | 59.3 | 52.2 | 21.7 | 28.0 |
| Adaptive | 4.37 | 8.23 | 62.7 | 57.3 | 33.4 | 33.5 |
| Static at the adaptive mean size | 4.30 | 7.71 | 61.7 | 55.8 | 29.9 | 33.5 |
| Same sizes, random assignment | 4.30 | 7.71 | 61.7 | 55.8 | 29.9 | 33.5 |
| Same sizes, task-label rule (code, math and MMLU largest, then WikiText-2, C4 smallest) | 4.37 | 8.23 | 62.7 | 57.3 | 33.4 | 33.5 |
| Same sizes, largest summed relative gain (oracle) | 4.37 | 7.48 | 59.3 | 57.3 | 33.4 | 33.5 |
Following your suggestions, we have added these results to Appendix A.22 as Table 26 (Lines 2951–2961).
Summarized W4 & Q4: Uniform against routing-weighted
A4: The uniform error of the selector converts to the routing-weighted error up to a factor , namely with , which is 1 under balanced routing and at most the number of experts (8, 64, 16 and 64 for the four backbones). A configuration that passes line 38 of Algorithm 1 then satisfies by Eq. (104) under Assumption 7 and , and a selector testing directly removes the factor .
Following your suggestions, we have stated this bound in Section 4.4 (Lines 335–338) and Appendix A.10.2 (Lines 2139–2146).
Summarized W5: Constants of the theory and the SNR notation
A5: The SNR notation is consistent in the revised version, where Theorem 1 defines (Line 1627) and its proof uses (Line 1704). The factor grows as while the bound of Theorem 1 does not, because vanishes with . Corollary 2 is a bound in , and the revised Appendix A.12.2 (Lines 2335–2395) sets no numerical KL budget, while Algorithm 1 (Line 2982) minimizes the routing-averaged KL of Eq. (8). The constants , and are model-specific, set by per-expert errors and activation statistics; the experiments use (Section 5.1). Appendix A.7.4 (Lines 1737–1742) states conditions (i) and (iv) as assumptions of the analysis, the latter at 128 calibration samples, reports a per-layer Kolmogorov-Smirnov test passing in 28 of 32 layers for condition (i), and gives a range [0.18, 0.28] of for condition (ii).
Following your suggestions, we have restated before Corollary 2 (Lines 1635–1636) and revised Appendix A.7.4 (Lines 1737–1742).
Summarized W6: Pareto optimality over a discrete grid
A6: Pareto-optimal refers to non-dominance on the frontier of the configurations that the rank searches return, as you note from Eq. (15). Within the family the search covers (one shared base, one base sparsity, a uniform rank and factors at the optimum of Eq. (9)), the binary search returns the smallest feasible rank, and a configuration dominating it would need a smaller rank, contradicting minimality, or the same rank with a smaller error, contradicting the optimality of the truncated SVD. The guarantee covers this uniform-rank family with a single base, and with randomized SVD and PGD the computed error approximates its optimum.
Following your suggestions, we have stated this definition in Section 4.3 (Lines 288–289) and Appendix A.9.3 (Lines 2024–2032).
References
[1] Hao Gu et al. Delta decompression for MoE-based LLMs compression. ICML, 2025.
Best Wishes,
The Authors of Paper 52575
Thank you for the response. While I appreciate your detailed explaination and appended ablation, the new results substantially diminish the contributions of MI-constrained QP and PGD and rank search, which are the two major components of PRIMO. As a result, I maintain my rating of 2.
Dear Reviewer tsY4,
Thank you for the timely response, and for acknowledging the detailed explanation and the appended ablation.
As suggested, we will keep these analyses in the final version.
Thank you again for your constructive feedback.
Best Wishes,
The Authors of Paper 52575
Response to Reviewer tsY4
Dear Reviewer tsY4:
Thank you for recognizing the novel problem formulation, the broad evaluation, and the detailed appendix of our work. We have responded point by point to your concerns regarding the PGD step at the truncated-SVD initialization, the QP and frequency-weighted averaging, resource matching of adaptive selection, the two error metrics, the constants of the theory and the SNR notation, and the definition of Pareto optimality. In the marked copy appended to the revised PDF (pages 57–115), the revisions made in response to your comments are tagged with @Reviewer tsY4. We hope our responses resolve your concerns and look forward to further discussion.
Summarized W1 & Q1: PGD at the truncated-SVD initialization
A1: The exact truncated SVD of ΔWiS is a stationary point of Eq. (9), so at a fixed rank PGD keeps that point. With B=ΔWiS and V~i=SVi, both gradients of Eq. (12) (Lines 261–263) are products of the residual B−UiV~i⊤ with a retained singular factor and vanish by orthogonality, so PGD closes the gap left by the randomized SVD. Table R1 shows that this gap is 0.03 in WikiText-2 perplexity (4.61 without PGD against 4.58 with PGD), and how this 0.03 splits between the rank and PGD depends on the path: along the exact-SVD path (rows 3 and 4) it is the rank, along the randomized-SVD path (rows 5 and 6) it is PGD, with row 4 taking the values of row 6 under the assumption that PGD reaches the optimum of Eq. (9).
Table R1: Base, rank rule and PGD on Mixtral-8×7B at the 40% target (row 3 has 27.5B parameters at 41.2% compression, against 28.0B for the other rows).
Following your suggestions, we have revised Table 4 and its analysis (Lines 437–447) and Table 17 (Lines 2754–2765), which add the exact truncated SVD at the searched rank.
Summarized W2 & Q2: The QP and frequency-weighted averaging
A2: α = p solves Eq. (8), and the shared base is the average weighted by the routing weights pi (the gate weights averaged over the calibration tokens). With b=Qp, Eq. (50) gives α=p for nonsingular Q; for singular Q every minimizer gives the same Wb when ΣX≻0, so the base is unique in both cases. The QP adds the objective behind this base, which separates it from Fisher weighting [1], and the revised Algorithm 1 sets α∗=p at line 4 (Line 2982), with no solver needed.
Following your suggestions, we have stated the closed form in Section 4.1 (Lines 222–224) and Algorithm 1 (Lines 2981–2982).
Summarized W3 & Q3: Resource matching of adaptive selection
A3: The adaptive policy deploys more parameters on shifted tasks by design, and its HumanEval gain from 21.7% to 33.4% (the values of the revised Table 5 for your 21.5% and 33.5%) includes the larger configuration. As you suggested, we compare the policies at the same mean size and report the resources of each deployed member. Table R2 shows that at 33.5B mean parameters the static policy reaches 29.9% Pass@1 against 33.4% for the adaptive one and is ahead on WikiText-2 and C4, where HumanEval deploys a member with rank ratio 0.659, 10.2B parameters read per token and 66.7 GiB in bfloat16, against 0.486, 8.2B and 52.2 GiB for the static member. These rows are read from score curves; at the mean size random assignment coincides with the static policy, and a task-label rule gives the same assignment as the adaptive policy, which routing shift reaches from unlabeled target samples.