acceptodds
Under review as a conference paper at ICLR 2027

PRIMO: Pareto-Optimal Information-Constrained Compression for Mixture-of-Experts Language Models

Abstract

Mixture-of-Experts (MoE) architectures scale large language models (LLMs) efficiently, but every expert must be stored although only a few are active per token, so memory becomes the deployment bottleneck. Existing pipelines merge experts into a shared base by heuristics, truncate the deltas at energy thresholds that bound neither error nor cost, and deploy one fixed configuration for every task. We propose PRIMO (Pareto-optimal Information-constrained Mixture-of-experts Compression), which casts post-training MoE compression as one constrained problem: minimize model size subject to explicit bounds on reconstruction error and computational overhead. Its key insight is that the costly stages are task-independent while the right compression level is task-dependent, and routing links the two. Once per model, MI-constrained decomposition builds the shared base by minimizing a routing-weighted KL surrogate of information loss, PGD low-rank compression with a rank search fits the deltas to both bounds, and a Pareto frontier stores the non-dominated size-error configurations. At deployment, task-adaptive selection tightens the error bound by the target task's routing shift and picks a configuration by a linear scan after one forward pass, without re-optimization. On four MoE LLMs (Mixtral-87B, DeepSeek-MoE-16B, Phi-3.5-MoE, Qwen2-57B-A14B) and 12 tasks, PRIMO attains the lowest perplexity and highest average accuracy in all eight settings, and its lead over D-MoE widens with compression, reaching 5.12 against 6.46 WikiText-2 perplexity at 60% compression of Mixtral-87B. Under routing shift, task-adaptive selection raises HumanEval Pass@1 from 21.5% (static configuration) to 33.5%, and its gain grows with the shift.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture-of-experts compression, post-training model compression, delta decomposition, low-rank expert approximation, Pareto frontier selection, routing distribution shift

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tsY4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulates MoE compression as minimizing model size subject to reconstruction-error and compute constraints. PRIMO first constructs a shared base through a routing-weighted KL quadratic program, then compresses expert residuals using activation-weighted low-rank factors and rank search, builds a discrete size–error frontier, and selects a configuration according to the routing KL shift between the target task and the calibration set. Experiments cover four MoE models, multiple compression ratios, and 12 tasks.

Critical strengths
  • Jointly considering a shared base, residual compression, size/error constraints, and task-distribution shift is an appealing problem formulation.
  • The experiments cover four architectures and report perplexity, nine accuracy metrics, throughput, memory, ablations, and offline cost.
  • The appendix provides the QP, the routing-shift bound, algorithm pseudocode, complexity analysis, and extensive sensitivity studies. The authors also candidly acknowledge the absence of calibration-budget and mixed-domain-batch experiments.
Critical weaknesses
  1. The purported PGD low-rank optimization is redundant under the paper's own objective. Let B=ΔW S and Ṽ=S V; Equation 9 is then the standard fixed-rank problem min ||B-U Ṽᵀ||²_F. Appendix A.8.5 also acknowledges that truncated SVD is globally optimal in the Eckart–Young sense. Algorithm 1 initializes with that truncated SVD, so it is already at a global optimum of the fixed-rank objective, the gradient should be zero, and subsequent PGD should not further reduce the same objective (unless the randomized SVD implementation is only an approximate solution, in which case the paper does not reformulate the argument accordingly). The 0.35 PPL improvement attributed to “PGD” in Tables 4/17 likely comes from different rank/threshold search rather than the PGD iterations themselves. This contradiction directly affects the core methodological contribution and technical correctness.
  2. Under the stated formulas, the purported MI-constrained QP reduces to a routing-frequency-weighted average. Equations 6–7 imply b=Qp, where p is the routing-frequency vector on the probability simplex. Therefore, α=p satisfies the first-order optimality condition and is a global optimum of the positive-semidefinite quadratic objective, although the solution may be non-unique in the singular case. The resulting shared base is simply W_b=Σ_i p_iW_i. The paper needs to explain what this QP adds beyond ordinary frequency-weighted merging.
  3. The large gains from adaptive selection are not resource-matched. In Tables 5/18, the adaptive strategy selects more permissive, less compressed configurations, but the final parameter count, rank, memory, FLOPs, and compression ratio selected for each task are not reported. The HumanEval improvement from 21.5 to 33.5 may primarily result from deploying a larger model. Without comparison to static, routing-independent, and oracle selection at the same size, the effectiveness of the routing rule itself is not established.
  4. The theoretical bound for adaptive selection uses an error metric different from the algorithm's actual metric. Equation 10 and line 37 of Algorithm 1 use the unweighted mean L(Θ) over layers and experts; Equations 97–104 in Appendix A.10.2 instead prove a result for L_ref, weighted by the reference routing distribution. The paper neither proves equivalence nor gives a conversion bound, so Equation 19 does not guarantee that the configuration actually selected by the algorithm satisfies the claimed target-task error bound.
  5. Constants required by the theoretical guarantee are not verified, and the SNR notation is inconsistent. The error bound in Appendix A.10 requires the ζ in Equation 104 to satisfy ζ ≥ ζ̄, but all models use a fixed ζ=1.0 without computing the actual ζ̄. Appendix A.7.4 empirically tests only Gaussianity and the error range, and claims that 128 samples are sufficient, but does not report an empirical test of the SNR condition. Equations 52–53 first define SNR as ||G||²₂, but the derivation subsequently uses ||G||₂=SNR; the correct relationship should involve a square root. Page 43 also gives a KL budget δ≤0.1I(X;Y), but Equation 8 and Algorithm 1 merely minimize KL without explicitly enforcing or verifying this constraint. Moreover, ρ in the MI bound diverges as reconstruction error approaches zero, potentially making the bound very loose.
  6. “Pareto-optimal” refers only to nondominated points on a 4×4 discrete grid. Appendix A.9.5 explicitly states that its theorem guarantees only nondominance among evaluated configurations. This is not Pareto optimality over the continuous feasible region of the original problem. The title and abstract risk overstating the result.
Questions for authors
  1. Since truncated SVD is already the fixed-rank global optimum of Equation 9, is the PGD gradient zero at initialization? Of the gain in Table 17, how much is attributable to PGD iterations and how much to rank selection?
  2. Is α=p, derived from b=Qp, the solution returned by the implementation? If so, what is the substantive difference between MI-QP and frequency-weighted averaging?
  3. How many parameters and FLOPs, and what compression ratio, are selected for each task in Tables 5/18? At matched resources, does adaptive selection still outperform static and random/task-label-only selection?
  4. How does the unweighted L(Θ) used by the algorithm satisfy Equation 104, which the appendix derives for routing-weighted L_ref?
4 comments
AuthorsOpenReview ↗

Response to Reviewer tsY4

Dear Reviewer tsY4:

Thank you for recognizing the novel problem formulation, the broad evaluation, and the detailed appendix of our work. We have responded point by point to your concerns regarding the PGD step at the truncated-SVD initialization, the QP and frequency-weighted averaging, resource matching of adaptive selection, the two error metrics, the constants of the theory and the SNR notation, and the definition of Pareto optimality. In the marked copy appended to the revised PDF (pages 57–115), the revisions made in response to your comments are tagged with @Reviewer tsY4. We hope our responses resolve your concerns and look forward to further discussion.


Summarized W1 & Q1: PGD at the truncated-SVD initialization

A1: The exact truncated SVD of ΔWiS\Delta W_iS is a stationary point of Eq. (9), so at a fixed rank PGD keeps that point. With B=ΔWiSB=\Delta W_iS and V~i=SVi\tilde V_i=SV_i, both gradients of Eq. (12) (Lines 261–263) are products of the residual B−UiV~i⊤B-U_i\tilde V_i^\top with a retained singular factor and vanish by orthogonality, so PGD closes the gap left by the randomized SVD. Table R1 shows that this gap is 0.03 in WikiText-2 perplexity (4.61 without PGD against 4.58 with PGD), and how this 0.03 splits between the rank and PGD depends on the path: along the exact-SVD path (rows 3 and 4) it is the rank, along the randomized-SVD path (rows 5 and 6) it is PGD, with row 4 taking the values of row 6 under the assumption that PGD reaches the optimum of Eq. (9).

Table R1: Base, rank rule and PGD on Mixtral-8×7B at the 40% target (row 3 has 27.5B parameters at 41.2% compression, against 28.0B for the other rows).

#BaseDelta compressor and rankWikiText-2 PPLPTB PPLC4 PPLNote
1Fisher [1]truncation-aware SVD, D2-MoE's delta share (47.34%)5.2820.5410.10reported by D2-MoE [1]
2frequency merging (top-k counts f)truncation-aware SVD, D2-MoE's delta share (47.34%)6.4235.1213.79reported by D2-MoE [1]
3routing-weight average (α = p)truncation-aware SVD, D2-MoE's delta share (rank ratio 0.473)4.6115.458.31Table 17
4routing-weight average (α = p)exact truncated SVD, searched rank (rank ratio 0.486)4.5815.298.23equal to row 6 if PGD reaches the optimum of Eq. (9)
5routing-weight average (α = p)randomized SVD, searched rank, no PGD4.6115.348.26slightly above row 6
6routing-weight average (α = p)randomized SVD + PGD, searched rank (PRIMO)4.5815.298.23Table 2

Following your suggestions, we have revised Table 4 and its analysis (Lines 437–447) and Table 17 (Lines 2754–2765), which add the exact truncated SVD at the searched rank.


Summarized W2 & Q2: The QP and frequency-weighted averaging

A2: α = p solves Eq. (8), and the shared base is the average weighted by the routing weights pip_i (the gate weights averaged over the calibration tokens). With b=Qpb=Qp, Eq. (50) gives α=p\alpha=p for nonsingular QQ; for singular QQ every minimizer gives the same WbW_b when ΣX≻0\Sigma_X\succ0, so the base is unique in both cases. The QP adds the objective behind this base, which separates it from Fisher weighting [1], and the revised Algorithm 1 sets α∗=p\alpha^*=p at line 4 (Line 2982), with no solver needed.

Following your suggestions, we have stated the closed form in Section 4.1 (Lines 222–224) and Algorithm 1 (Lines 2981–2982).


Summarized W3 & Q3: Resource matching of adaptive selection

A3: The adaptive policy deploys more parameters on shifted tasks by design, and its HumanEval gain from 21.7% to 33.4% (the values of the revised Table 5 for your 21.5% and 33.5%) includes the larger configuration. As you suggested, we compare the policies at the same mean size and report the resources of each deployed member. Table R2 shows that at 33.5B mean parameters the static policy reaches 29.9% Pass@1 against 33.4% for the adaptive one and is ahead on WikiText-2 and C4, where HumanEval deploys a member with rank ratio 0.659, 10.2B parameters read per token and 66.7 GiB in bfloat16, against 0.486, 8.2B and 52.2 GiB for the static member. These rows are read from score curves; at the mean size random assignment coincides with the static policy, and a task-label rule gives the same assignment as the adaptive policy, which routing shift reaches from unlabeled target samples.

(continued)

Table R2: Size deployed per task and score under each policy on Mixtral-8×7B at the 40% target; the last four rows use the adaptive mean size and are read from score curves.

RowWikiText-2 PPLC4 PPLMMLU acc (%)GSM8K acc (%)HumanEval Pass@1 (%)Mean parameters (B)
Routing shift D_shift0.120.000.350.470.61-
Adaptive: compression ratio (%)31.340.023.223.223.2-
Adaptive: parameters (B)32.128.035.835.835.833.5
Adaptive: rank ratio0.5760.4860.6590.6590.659-
Adaptive: parameters read per token (B)9.28.210.210.210.2-
Adaptive: weight memory, bfloat16 (GiB)59.852.266.766.766.7-
Static4.588.2359.352.221.728.0
Adaptive4.378.2362.757.333.433.5
Static at the adaptive mean size4.307.7161.755.829.933.5
Same sizes, random assignment4.307.7161.755.829.933.5
Same sizes, task-label rule (code, math and MMLU largest, then WikiText-2, C4 smallest)4.378.2362.757.333.433.5
Same sizes, largest summed relative gain (oracle)4.377.4859.357.333.433.5

Following your suggestions, we have added these results to Appendix A.22 as Table 26 (Lines 2951–2961).


Summarized W4 & Q4: Uniform L(Θ)L(\Theta) against routing-weighted LrefL_{\text{ref}}

A4: The uniform error of the selector converts to the routing-weighted error up to a factor κ\kappa, namely Lref≤κL(Θ)L_{\text{ref}}\le\kappa L(\Theta) with κ=max⁡lNmax⁡ipref(l)(i)\kappa=\max_lN\max_ip^{(l)}_{\text{ref}}(i), which is 1 under balanced routing and at most the number of experts NN (8, 64, 16 and 64 for the four backbones). A configuration that passes line 38 of Algorithm 1 then satisfies Ltask≤κϵL_{\text{task}}\le\kappa\epsilon by Eq. (104) under Assumption 7 and ζ≥ζˉ\zeta\ge\bar\zeta, and a selector testing LrefL_{\text{ref}} directly removes the factor κ\kappa.

Following your suggestions, we have stated this bound in Section 4.4 (Lines 335–338) and Appendix A.10.2 (Lines 2139–2146).


Summarized W5: Constants of the theory and the SNR notation

A5: The SNR notation is consistent in the revised version, where Theorem 1 defines SNR=∥G∥22\mathrm{SNR} = \Vert G \Vert_2^2 (Line 1627) and its proof uses ∥G∥F≤m∥G∥2=m⋅SNR\Vert G \Vert_F \le \sqrt{m} \Vert G \Vert_2 = \sqrt{m \cdot \mathrm{SNR}} (Line 1704). The factor ρ\rho grows as ∥E∥F→0\Vert E \Vert_F \to 0 while the bound of Theorem 1 does not, because (1+ρ)E[DKL]=12∥E∥F2+∥G∥F∥E∥F(1+\rho) \mathbb{E}[D_{\mathrm{KL}}] = \frac{1}{2} \Vert E \Vert_F^2 + \Vert G \Vert_F \Vert E \Vert_F vanishes with EE. Corollary 2 is a bound in δ\delta, and the revised Appendix A.12.2 (Lines 2335–2395) sets no numerical KL budget, while Algorithm 1 (Line 2982) minimizes the routing-averaged KL of Eq. (8). The constants ζˉ\bar{\zeta}, SNRmax⁡\mathrm{SNR}_{\max} and δ/I(X;Y)\delta / I(X;Y) are model-specific, set by per-expert errors and activation statistics; the experiments use ζ=1.0\zeta = 1.0 (Section 5.1). Appendix A.7.4 (Lines 1737–1742) states conditions (i) and (iv) as assumptions of the analysis, the latter at 128 calibration samples, reports a per-layer Kolmogorov-Smirnov test passing in 28 of 32 layers for condition (i), and gives a range [0.18, 0.28] of ∥ΔW∥F/∥W∥F\Vert \Delta W \Vert_F / \Vert W \Vert_F for condition (ii).

Following your suggestions, we have restated SNRmax⁡\mathrm{SNR}_{\max} before Corollary 2 (Lines 1635–1636) and revised Appendix A.7.4 (Lines 1737–1742).


Summarized W6: Pareto optimality over a discrete grid

A6: Pareto-optimal refers to non-dominance on the frontier F\mathcal F of the configurations that the rank searches return, as you note from Eq. (15). Within the family the search covers (one shared base, one base sparsity, a uniform rank and factors at the optimum of Eq. (9)), the binary search returns the smallest feasible rank, and a configuration dominating it would need a smaller rank, contradicting minimality, or the same rank with a smaller error, contradicting the optimality of the truncated SVD. The guarantee covers this uniform-rank family with a single base, and with randomized SVD and PGD the computed error approximates its optimum.

Following your suggestions, we have stated this definition in Section 4.3 (Lines 288–289) and Appendix A.9.3 (Lines 2024–2032).


References

[1] Hao Gu et al. Delta decompression for MoE-based LLMs compression. ICML, 2025.


Best Wishes,

The Authors of Paper 52575

Thank you for the response. While I appreciate your detailed explaination and appended ablation, the new results substantially diminish the contributions of MI-constrained QP and PGD and rank search, which are the two major components of PRIMO. As a result, I maintain my rating of 2.

Dear Reviewer tsY4,

Thank you for the timely response, and for acknowledging the detailed explanation and the appended ablation.

As suggested, we will keep these analyses in the final version.

Thank you again for your constructive feedback.

Best Wishes,

The Authors of Paper 52575

Reviewer_qTd7OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PRIMO compresses MoE expert weights into a shared base and low-rank residuals. It constructs the base through a routing-weighted Gaussian KL objective, optimizes activation-weighted residual reconstruction, caches configurations trading model size against reconstruction error, and selects a configuration using target-task routing divergence. Experiments cover four MoE models, language modeling and downstream benchmarks, with component ablations, selector sensitivity, and efficiency measurements. The reported improvements are substantial at high compression, including Mixtral WikiText-2 perplexity of 5.12 versus D2-MoE's 6.46 at 60% compression.

Critical strengths
  1. Table 2 spans four architecture families and multiple compression levels, with improvements in perplexity and average task accuracy. This breadth makes the approach potentially useful beyond one particular MoE architecture.

  2. Caching candidate configurations permits inexpensive task-level selection without recompression. The selector sensitivity experiments and measured offline cost (Appendices A.19 and A.21) help assess this deployment trade-off.

Critical weaknesses
  1. From Eqs. (6)-(8), symmetry of the Gram matrix gives b=Qpb=Qp, so the objective equals 12(α−p)⊤Q(α−p)+constant\tfrac12(\alpha-p)^\top Q(\alpha-p)+\mathrm{constant}. Since routing frequencies lie on the simplex, α=p\alpha=p is already a global minimizer: frequency-weighted merging solves the stated problem. Table 16 compares full pipelines and does not isolate a base-specific advantage over this construction. Similarly, Appendix A.8.5 explicitly establishes that truncated SVD of ΔWiS\Delta W_iS globally minimizes Eq. (9), and Eq. (90) uses that solution for initialization. Randomized SVD could leave room for numerical refinement, but Tables 4/17 do not isolate this from rank selection or SVD accuracy. Consequently, the evidence does not establish the advertised advantages of MI optimization and PGD over frequency averaging plus accurate weighted SVD.

  2. Appendix A.10.2 bounds a routing-weighted reference error with fixed activation statistics and requires ζ≥ζˉ\zeta\geq\bar\zeta. Algorithm 1 instead selects using the uniformly averaged error in Eq. (10), with an unverified default ζ=1\zeta=1. Thus, the stated bound does not justify the actual selector, or control changes in activation statistics. Moreover, Tables 5/18 explicitly improve shifted tasks by selecting less-compressed models, without reporting their resulting sizes or comparison with a static policy at comparable memory cost. The reported gains therefore do not distinguish effective shift-aware selection from simply allocating more parameters.

Questions for authors

None

4 comments
AuthorsOpenReview ↗

Response to Reviewer qTd7

Dear Reviewer qTd7:

Thank you for recognizing the broad evaluation and the practical method design of our work. We have responded point by point to your concerns regarding the comparison with frequency averaging plus accurate weighted SVD, and the link between the bound, the selector and resource matching. In the marked copy appended to the revised PDF (pages 57–115), the revisions made in response to your comments are tagged with @Reviewer qTd7. We hope our responses resolve your concerns and look forward to further discussion.


Summarized W1: Frequency averaging plus accurate weighted SVD

A1: Both equivalences hold; PRIMO contributes the formulation that yields this base and fit, and the bound-driven search that sets the rank. Eq. (8) is minimized by the routing-weight average and both gradients of Eq. (12) vanish at the exact truncated SVD of ΔWiS\Delta W_iS; PRIMO adds the objective that links this base to the bound in Eq. (3), the rank chosen at the 40% target by binary search under explicit error and overhead bounds (ϵ\epsilon and γ\gamma) where D2-MoE [1] keeps a hand-set delta share, and shift-aware selection on the frontier. Table R1 shows that the searched rank lowers WikiText-2 perplexity by 0.03 (4.61 against 4.58, with row 4 taking the values of row 6 at the optimum of Eq. (9)), and the rest of the gain over D2-MoE lies in the base together with calibration, sparsity and whitening.

Table R1: Base, rank rule and PGD on Mixtral-8×7B at the 40% target (row 3 has 27.5B parameters at 41.2% compression, against 28.0B for the other rows).

#BaseDelta compressor and rankWikiText-2 PPLPTB PPLC4 PPLNote
1Fisher [1]truncation-aware SVD, D2-MoE's delta share (47.34%)5.2820.5410.10reported by D2-MoE [1]
2frequency merging (top-k counts f)truncation-aware SVD, D2-MoE's delta share (47.34%)6.4235.1213.79reported by D2-MoE [1]
3routing-weight average (α = p)truncation-aware SVD, D2-MoE's delta share (rank ratio 0.473)4.6115.458.31Table 17
4routing-weight average (α = p)exact truncated SVD, searched rank (rank ratio 0.486)4.5815.298.23equal to row 6 if PGD reaches the optimum of Eq. (9)
5routing-weight average (α = p)randomized SVD, searched rank, no PGD4.6115.348.26slightly above row 6
6routing-weight average (α = p)randomized SVD + PGD, searched rank (PRIMO)4.5815.298.23Table 2

Following your suggestions, we have added the exact-SVD row to Table 17 (Lines 2764–2765) and rows under C4 calibration to Table 16 (Lines 2710–2714), where the routing-weight average has 4.61 against 5.37 for the Fisher base and 6.54 for frequency merging.


Summarized W2: The bound, the selector and resource matching

A2: A configuration the selector accepts meets the bound up to a factor κ\kappa, and the adaptive gains include larger configurations by design. The bound covers the routing-weighted error of the delta factors before base sparsification, with per-expert errors held fixed (Assumption 7); since Lref≤κL(Θ)L_{\text{ref}}\le\kappa L(\Theta) with κ=max⁡lNmax⁡ipref(l)(i)\kappa=\max_l N\max_ip^{(l)}_{\text{ref}}(i), between 1 under balanced routing and the number of experts NN, a configuration passing the uniform-error test satisfies Ltask≤κϵL_{\text{task}}\le\kappa\epsilon for ζ≥ζˉ\zeta\ge\bar\zeta, where ζˉ\bar\zeta is set by the reference routing prefp_{\text{ref}} and the per-expert errors (Appendix A.10.2) and the experiments use ζ=1.0\zeta=1.0 (Section 5.1). Table R2 shows that at the same mean size of 33.5B parameters the static policy reaches 29.9% HumanEval Pass@1 against 33.4% for the adaptive one, while it is ahead on WikiText-2 and C4; these rows are read from score curves, and at this mean size a task-label rule gives the same assignment, which routing shift reaches from unlabeled target samples.

(continued)

Table R2: Size deployed per task and score under each policy on Mixtral-8×7B at the 40% target; the last four rows use the adaptive mean size and are read from score curves.

RowWikiText-2 PPLC4 PPLMMLU acc (%)GSM8K acc (%)HumanEval Pass@1 (%)Mean parameters (B)
Routing shift D_shift0.120.000.350.470.61-
Adaptive: compression ratio (%)31.340.023.223.223.2-
Adaptive: parameters (B)32.128.035.835.835.833.5
Adaptive: rank ratio0.5760.4860.6590.6590.659-
Adaptive: parameters read per token (B)9.28.210.210.210.2-
Adaptive: weight memory, bfloat16 (GiB)59.852.266.766.766.7-
Static4.588.2359.352.221.728.0
Adaptive4.378.2362.757.333.433.5
Static at the adaptive mean size4.307.7161.755.829.933.5
Same sizes, random assignment4.307.7161.755.829.933.5
Same sizes, task-label rule (code, math and MMLU largest, then WikiText-2, C4 smallest)4.378.2362.757.333.433.5
Same sizes, largest summed relative gain (oracle)4.377.4859.357.333.433.5

Following your suggestions, we have stated the bound in Section 4.4 (Lines 335–338) and Appendix A.10.2 (Lines 2139–2146), and added these results to Appendix A.22 as Table 26 (Lines 2951–2961).


References

[1] Hao Gu et al. Delta decompression for MoE-based LLMs compression. ICML, 2025.

Best Wishes,

The Authors of Paper 52575

Dear Reviewer qTd7,

Thank you for your time in reviewing our work.

We will keep these analyses in the final version.

Thank you again for your constructive feedback.

Best Wishes,

The Authors of Paper 52575

I really appreciate the author's effort and persistence. Although I think this paper did not meet the ICLR acceptance standards, my initial score was too harsh, so I am willing to raise my score from 1 to 2 to encourage the author.

Reviewer_jXD5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents PRIMO, a post-training compression method for mixture-of-experts language models that stores each expert as a shared base matrix plus a low-rank residual, which keeps every expert while cutting the stored parameters. The shared base is a convex combination of the expert weights, and a quadratic program picks the mixing coefficients by minimizing a KL surrogate of the information loss. Experiments on four MoE models demonstrate lower perplexity and higher average accuracy than existing pruning, merging, sparsity and low-rank methods, and show that the per-task selection recovers accuracy on code and math tasks.

Critical strengths
  • The paper shows that the accuracy loss of a compressed model grows with the change in expert routing between the calibration data and the target task, which explains why one fixed configuration loses most on code and math tasks.

  • The evaluation spans four MoE families.

Critical weaknesses
  • The shared base construction reduces to a known heuristic. The optimal mixing coefficients of the quadratic program equal the routing frequencies, so the base is the frequency-weighted average that previous work already uses.

  • The low-rank fitting step cannot improve on its own initialization. The paper initializes the factors at the truncated SVD of the whitened residual, and this is already the global optimum of the fitting objective.

  • The numbers in Table 2 seems to be the best accuracy with per-task selection, which the paper states deploys a less compressed configuration. Hence this 40% compression rate is not quite fair as a comparison with other method. If one use a static setting, PRIMO falls below the strongest baseline on mixtral, so the claim of the highest average accuracy does not hold in static setting.

  • The paper states that memory is the deployment bottleneck of MoE models, and it keeps all compressed weights in bfloat16. However, the quality comparison contains no quantization method, although quantization is the standard way to cut memory. Also, there is not latency/throughput profiling for the compressed model.

Questions for authors
  • Which compression ratio does the per-task selection deploy in Table 5?
5 comments
AuthorsOpenReview ↗

Response to Reviewer jXD5

Dear Reviewer jXD5:

Thank you for recognizing the valuable empirical finding and the broad evaluation of our work. We have responded point by point to your concerns regarding the base construction, the low-rank fitting, the selection policy behind Table 2, quantization and latency, and the compression ratio deployed per task. In the marked copy appended to the revised PDF (pages 57–115), the revisions made in response to your comments are tagged with @Reviewer jXD5. We hope our responses resolve your concerns and look forward to further discussion.


Summarized W1: The QP optimum equals the routing frequencies

A1: α = p is a global minimizer of Eq. (8) (Lines 219–220), and the QP derives this base from the routing-weighted KL term of the bound in Eq. (3). In the metric ⟨A,B⟩M=tr(Σξ−1AΣXB⊤)\langle A,B\rangle_M=\mathrm{tr}(\Sigma_\xi^{-1}A\Sigma_X B^\top), Eq. (42) equals 12∥Wb−∑ipiWi∥M2\tfrac12\|W_b-\sum_i p_iW_i\|_M^2 plus a constant, which the average Wb=∑ipiWiW_b=\sum_i p_iW_i minimizes, where pip_i is the gate weight of expert ii averaged over the calibration tokens, whereas frequency merging uses the top-kk count fif_i. This objective separates the base from Fisher weighting [1]. Table R1 shows that at the same delta share (47.34%) this base reaches a WikiText-2 perplexity of 4.61 against 5.28 for the Fisher base, while calibration, sparsity and whitening also differ between these rows.

Table R1: Base, rank rule and PGD on Mixtral-8×7B at the 40% target (row 3 has 27.5B parameters at 41.2% compression, against 28.0B for the other rows).

#BaseDelta compressor and rankWikiText-2 PPLPTB PPLC4 PPLNote
1Fisher [1]truncation-aware SVD, D2-MoE's delta share (47.34%)5.2820.5410.10reported by D2-MoE [1]
2frequency merging (top-k counts f)truncation-aware SVD, D2-MoE's delta share (47.34%)6.4235.1213.79reported by D2-MoE [1]
3routing-weight average (α = p)truncation-aware SVD, D2-MoE's delta share (rank ratio 0.473)4.6115.458.31Table 17
4routing-weight average (α = p)exact truncated SVD, searched rank (rank ratio 0.486)4.5815.298.23equal to row 6 if PGD reaches the optimum of Eq. (9)
5routing-weight average (α = p)randomized SVD, searched rank, no PGD4.6115.348.26slightly above row 6
6routing-weight average (α = p)randomized SVD + PGD, searched rank (PRIMO)4.5815.298.23Table 2

Following your suggestions, we have stated the closed form in Section 4.1 (Lines 222–224) and Algorithm 1 (Lines 2981–2982), and added these rows to Appendix A.22 as Table 23 (Lines 2919–2930).


Summarized W2: The low-rank fit starts at its optimum

A2: The exact truncated SVD is a stationary point of Eq. (9), and in the revised Tables 4 and 17 the step from the truncation-aware row to PRIMO comes from the rank. With B=ΔWiSB=\Delta W_iS and V~i=SVi\tilde V_i=SV_i, both gradients of Eq. (12) (Lines 261–263) are products of the residual B−UiV~i⊤B-U_i\tilde V_i^\top with a retained singular factor, and they vanish because this residual is orthogonal to the retained singular vectors, so PGD closes the gap left by the randomized SVD. As reported in Table R1 of our response to your W1, this gap is 0.03 in WikiText-2 perplexity (4.61 without PGD against 4.58 with PGD). Row 4 takes the values of row 6 under the assumption that PGD reaches the optimum of Eq. (9), and the 0.03 step from D2-MoE's delta share to the searched rank along the exact-SVD path (rows 3 and 4) is then the effect of the rank.

Following your suggestions, we have revised Table 4 and its analysis (Lines 437–447) and Table 17 (Lines 2754–2765), which add the exact truncated SVD at the searched rank.

(continued)

Summarized W3: The selection policy behind Table 2

A3: The revised Table 2 compares all methods under the static policy, namely one configuration per model and ratio for all tasks. Table R2 shows that the static average accuracy of PRIMO is the highest in all eight settings, with 0.58 against 0.56 for D2-MoE [1] at Mixtral-8×7B 40%. Its MMLU and GSM8K cells there are the static values 0.59 and 0.52, which differ from those of the reviewed Table 5 that you note were below D2-MoE's.

Table R2: Average accuracy of PRIMO under the static policy against the strongest baseline in each setting of Table 2.

ModelRatioStrongest baselineIts AvgPRIMO static AvgPRIMO MMLUPRIMO GSM8K
Mixtral-8×7B20%D2-MoE [1]0.590.610.640.58
Mixtral-8×7B40%D2-MoE [1]0.560.580.590.52
Mixtral-8×7B60%D2-MoE [1]0.510.540.510.44
DeepSeekMoE-16B20%D2-MoE [1]0.480.490.430.19
DeepSeekMoE-16B40%D2-MoE [1]0.440.460.400.17
DeepSeekMoE-16B60%D2-MoE [1]0.370.420.360.15
Phi-3.5-MoE40%D2-MoE [1]0.630.640.730.81
Qwen2-57B-A14B40%D2-MoE [1]0.620.630.750.79

Following your suggestions, we have stated the policy in the caption of Table 2 (Lines 378–379) and in Appendix A.12.3 (Lines 2426–2439), and added these results to Appendix A.22 as Table 24 (Lines 2932–2941).


Summarized W4: Quantization and latency

A4: Following your suggestion, we further report weight memory, WikiText-2 perplexity and batch-1 latency (including 16-bit and 4-bit weights) on Mixtral-8×7B. Table R3 shows that PRIMO composes with 4-bit quantization and lowers the estimated latency, where PRIMO at 40% with 4-bit weights needs 13.0 GiB against 21.7 GiB for the 4-bit full model, and PRIMO at 16 bits takes 1.05 s against 1.65 s for the full model. The two methods compose: the 4-bit GPTQ [2] full model is smaller and better than 16-bit PRIMO alone (4.07 against 4.58 in perplexity), and PRIMO with 4-bit weights is the smallest model in Table R3. For throughput, Table 15 (Lines 2677–2686) compares compressed Mixtral-8×7B with four baselines at three ratios at batch size 64, a different setting from the batch-1 latency that Table R3 estimates from the bytes read per token at a bandwidth of 2.0 TB/s.

Table R3: Weight memory, perplexity and estimated latency on Mixtral-8×7B.

MethodWeight bitsParameters (B)Weight memory (GiB)WikiText-2 PPLBatch-1 latency, 128 tokens (s)
Full model1646.787.03.981.65
Full model + RTN 4-bit446.721.74.150.41
Full model + GPTQ 4-bit [2]446.721.74.070.41
PRIMO 40%1628.052.24.581.05
PRIMO 40% + RTN 4-bit428.013.04.810.26

Following your suggestions, we have added these results to Appendix A.22 as Table 25 (Lines 2943–2949).


Summarized Q1: The ratio deployed per task in Table 5

A5: The adaptive policy deploys 31.3% compression on WikiText-2 and 23.2% on MMLU, GSM8K and HumanEval, against 40.0% for the static policy, because the selection rule of Eq. (19) (Lines 332–338) selects the most compressed stored configuration whose error meets the bound tightened by the routing shift. Table R4 shows that at the same mean size of 33.5B parameters, the static policy reaches 29.9% HumanEval Pass@1 against 33.4% for the adaptive one, while it is ahead on WikiText-2 and C4. These matched-size rows are read from score curves through the printed values of each task. At this mean size a task-label rule gives the same assignment, which routing shift reaches from unlabeled target samples.

(continued)

Table R4: Size deployed per task and score under each policy on Mixtral-8×7B at the 40% target; the last four rows use the adaptive mean size and are read from score curves.

RowWikiText-2 PPLC4 PPLMMLU acc (%)GSM8K acc (%)HumanEval Pass@1 (%)Mean parameters (B)
Routing shift D_shift0.120.000.350.470.61-
Adaptive: compression ratio (%)31.340.023.223.223.2-
Adaptive: parameters (B)32.128.035.835.835.833.5
Adaptive: rank ratio0.5760.4860.6590.6590.659-
Adaptive: parameters read per token (B)9.28.210.210.210.2-
Adaptive: weight memory, bfloat16 (GiB)59.852.266.766.766.7-
Static4.588.2359.352.221.728.0
Adaptive4.378.2362.757.333.433.5
Static at the adaptive mean size4.307.7161.755.829.933.5
Same sizes, random assignment4.307.7161.755.829.933.5
Same sizes, task-label rule (code, math and MMLU largest, then WikiText-2, C4 smallest)4.378.2362.757.333.433.5
Same sizes, largest summed relative gain (oracle)4.377.4859.357.333.433.5

Following your suggestions, we have stated the frontier in Section 5.1 (Lines 360–364) and Appendix A.12.3 (Lines 2411–2417), and added these results to Appendix A.22 as Table 26 (Lines 2951–2961).


References

[1] Hao Gu et al. Delta decompression for MoE-based LLMs compression. ICML, 2025.

[2] Elias Frantar et al. OPTQ: Accurate quantization for generative pre-trained transformers. ICLR, 2023.


Best Wishes,

The Authors of Paper 52575

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

General Response

We sincerely thank all reviewers for their time and constructive comments. Line references refer to the clean copy, PDF pages 1–56.

Summary

Mixture-of-Experts language models store every expert although each token activates only a few, and existing compression methods rely on heuristic merging, hand-set truncation ratios and one fixed configuration for every task. Our method, PRIMO, casts post-training MoE compression as one constrained problem, minimizing model size under explicit bounds on reconstruction error and computational overhead, and selects a stored configuration for each task by its routing shift. Under the static policy, PRIMO has the lowest perplexity and the highest average accuracy in all eight settings, with a WikiText-2 perplexity of 5.14 against 6.46 for D2-MoE at 60% compression of Mixtral-8×7B. The revision adds a C4-calibrated base comparison, the separation of rank search and PGD, 4-bit results, and a matched-size comparison of the selection policies.

Strengths

  • Novel formulation. Shared base, residual compression, constraints and task shift treated jointly (tsY4).
  • Broad evaluation. Four MoE families and several compression levels (jXD5, qTd7, tsY4).
  • Valuable finding. Accuracy loss grows with the routing change between calibration data and target task (jXD5).
  • Practical design. Cached configurations allow task-level selection without recompression (qTd7); detailed appendix (tsY4).

Weaknesses and Revisions

  1. Quantization and latency (jXD5). Added weight memory, perplexity and estimated batch-1 latency with 16-bit and 4-bit weights in Table 25 (Lines 2943–2949).
  2. Size per task and resource matching (jXD5, qTd7, tsY4). Added the size deployed per task and a comparison at equal mean size in Table 26 (Lines 2951–2961).
  3. Role of the QP (jXD5, qTd7, tsY4). Stated its closed form in Section 4.1 (Lines 222–224) and Algorithm 1 (Lines 2981–2982); added C4-calibrated rows to Table 16 (Lines 2710–2714).
  4. Role of PGD (jXD5, qTd7, tsY4). Revised Tables 4 and 17 (Lines 437–447 and 2754–2765) to state the setting of every row, and separated rank search from PGD in Table 23 (Lines 2919–2930).
  5. Selection policy of the main table (jXD5). Specified the static policy in the caption of Table 2 (Lines 378–379), Appendix A.12.3 (Lines 2426–2439) and Table 24 (Lines 2932–2941).
  6. Bound and selector (qTd7, tsY4). Explained the factor between the uniform and routing-weighted errors in Section 4.4 (Lines 335–338) and Appendix A.10.2 (Lines 2139–2146).
  7. Theory constants and SNR notation (tsY4). Clarified that Theorem 1 (Lines 1624–1634) and its proof (Line 1704) use one SNR definition.
  8. Pareto optimality (tsY4). Defined it as non-dominance on the stored frontier in Section 4.3 (Lines 288–289) and Appendix A.9.3 (Lines 2024–2032).

In the marked copy (PDF pages 57–115), every revision is tagged with the ID of the reviewer who raised it.

All experiments above will be kept in the final version.

Best Wishes,

The Authors of Paper 52575

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.