- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2(+1)
- Soundness
- 1
Read review
Paper summary
Budget allocation for MoE compression through joint rate–distortion optimization of shared-base sparsity and residual low-rank compression. The method evaluates sparsity–rank combinations per layer and uses a common Lagrange multiplier to select configurations under a global storage budget.
Critical strengths
None
Critical weaknesses
-
Unclear presentation and problem formulation. The abstract and introduction rely on terms such as “shared-base sparsity” and “expert-residual rank” before adequately explaining the underlying representation and compression operations. References such as “That staging” in line 61 further obscure the argument. Although the mechanism becomes more concrete in Section 4.2, the introduction is difficult to follow because it discusses the interaction between compression decisions before clearly establishing those decisions. This substantially increases the effort required to identify and assess the contribution.
-
Insufficiently established significance and advantage over staged allocation. The importance of MoE compression does not itself establish the importance of the specific allocation problem addressed here. The paper explains why sparsity and residual rank interact, but does not convincingly establish that handling this interaction provides a substantial advance over a carefully tuned sequential pipeline. The reported improvements and joint-versus-additive ablation are relevant, but the latter does not directly test a strong sequential or alternating optimizer under otherwise identical conditions. Since the allocation mechanism draws on established rate–distortion optimization, a clearer demonstration of this specific advantage is central to establishing the contribution’s significance. Nonetheless, their improvement over baseline method is incremental (see Table 2. for instance).
Questions for authors
-
Could the authors clearly define “expert bank,” “shared-base sparsity,” and “expert-residual rank,” and clarify which specific sequence of operations “That staging” refers to in line 61?
-
Beyond the general importance of reducing MoE storage, what evidence establishes sequential sparsity and rank allocation as a practically significant limitation? How much quality or storage efficiency is lost specifically because these decisions are made sequentially?
-
What practical benefit does joint allocation provide over a carefully tuned sequential pipeline, and why should this benefit be considered a substantial advance rather than an incremental improvement?
5 comments
Summarized W2: What is the novelty beyond established rate–distortion optimization?
A2: We respectfully clarify the significance and novelty of our work by addressing four arguments. In short, existing methods set the shared-base sparsity and the expert-residual rank separately, and our method prices the two jointly under one byte budget.
- Argue 1: Is the allocation problem unimportant? The allocation problem is important, since under a fixed byte budget, compressing an MoE model is essentially deciding where each byte goes, namely to which layer, and to the shared base or the expert residuals. Therefore, the allocation directly decides the compression quality, and replacing our allocation with a greedy allocation on the same per-layer tables and the same budget raises perplexity by up to 0.318 at 60% compression.
- Argue 2: Is handling the interaction no better than sequential pipelines? Handling the interaction between the shared-base sparsity and the expert-residual rank brings a clear gain over sequential decisions. Under identical grids, budget, and routing weights, separating the two decisions (scoring them on independent surfaces, Table 5, Lines 384–394) raises perplexity from 8.91 to 9.21, and the gap of the sequential pipeline D2-MoE grows from 0.20 to 0.88 perplexity as compression increases from 30% to 60%. We further compare with carefully tuned sequential and alternating pipelines in our response to your Q3.
- Argue 3: Is there no novelty beyond existing methods (i.e., established rate–distortion optimization)? Rate–distortion optimization is the tool we use, and our novelty lies in overcoming three obstacles when applying it to this problem. First, standard rate–distortion allocation assumes independent options, while here the shared-base sparsity changes every expert residual, so we score the two decisions jointly in one per-layer table. Second, this coupling makes exact search grow multiplicatively with depth, so we price all layers with one multiplier, which solves the allocation 19.6× faster than exact dynamic programming on the bounded instance. Third, since the reconstruction objective cannot tell which tasks get worse, we report the gain or loss on each task separately.
- Argue 4: Is the improvement marginal? The improvement is consistent and grows as compression tightens. Table R2 shows that our method reduces perplexity over the strongest baseline (D-Rank) in all eight settings by up to 8.4%, while keeping the aggregate score comparable or better.
Table R2: Perplexity reduction over the strongest baseline (D-Rank).
| Model | 30% | 40% | 50% | 60% |
|---|---|---|---|---|
| Qwen1.5-MoE-A2.7B | 8.65 → 8.51 (−1.6%) | 9.16 → 8.91 (−2.7%) | 10.34 → 9.94 (−3.9%) | 12.74 → 12.05 (−5.4%) |
| Mixtral-8×7B | 5.68 → 5.33 (−6.2%) | 5.88 → 5.49 (−6.6%) | 6.33 → 5.86 (−7.4%) | 7.18 → 6.58 (−8.4%) |
Following your suggestions, we have revised Section 1 (Lines 72–86) to state the significance and novelty more clearly.
Summarized Q2 (1): What evidence shows sequential allocation is a significant limitation?
A3 (1): We respectfully clarify that sequential allocation is a significant limitation, since fixing one decision before the other leaves quality behind even on identical surfaces. Two published pipelines that start from opposite halves of the decision (D2-MoE fixes the shared base first; D-Rank sets ranks with no shared base) disagree at every compression ratio, and their disagreement grows from 0.69% at 30% to 1.47% at 60% compression; on identical surfaces, a tuned sequential optimizer still trails our method by 0.23 and 0.52 perplexity at 40% and 60% (Table R4).
Following your suggestions, we have revised Section 2 (Lines 99–121) to make this evidence clearer.
Summarized Q2 (2): Quality and storage efficiency lost by sequential decisions
A3 (2): As you suggested, we quantify the quality and storage efficiency that the sequential pipeline D2-MoE loses against our method (i.e., higher perplexity and extra storage). Table R3 shows that the sequential pipeline D2-MoE loses up to 0.88 perplexity at the same budget and needs up to 5.0% extra storage at the same perplexity. The storage loss is the extra storage D2-MoE needs for the same perplexity, in percent of the uncompressed size; its 60% entry uses the perplexities of our method at 62%, 64%, and 66% compression (12.63, 13.33, and 14.06 perplexity). D2-MoE also differs from our method in its base and its factorization; the comparison on identical surfaces is in Table R4.
Table R3: Losses of the sequential pipeline D2-MoE.
| Loss | 30% | 40% | 50% | 60% |
|---|---|---|---|---|
| Quality loss | +0.20 PPL | +0.36 PPL | +0.52 PPL | +0.88 PPL |
| Storage loss | +5.0% | +3.5% | +2.5% | +2.9% |
Following your suggestions, we have revised Appendix A.15 (Lines 1654–1673) to make these losses clearer.
Summarized Q3 (1): Practical benefit over a carefully tuned sequential pipeline
A4 (1): Following your suggestion, we further compare our method with two strong optimizers (i.e., Sequential and Alternating) under identical grids, byte budget, and routing weights, as you recommended in W2. Table R4 shows that our method outperforms both optimizers, where even Alternating, the stronger one, remains 0.34 perplexity behind at 60% compression.
Table R4: Comparison with strong sequential and alternating optimizers.
| Method | 40% PPL ↓ | 40% Score ↑ | 60% PPL ↓ | 60% Score ↑ |
|---|---|---|---|---|
| Sequential | 9.14 (+0.23) | 55.5 | 12.57 (+0.52) | 49.4 |
| Alternating | 9.04 (+0.13) | 55.5 | 12.39 (+0.34) | 49.5 |
| Our method | 8.91 | 55.7 | 12.05 | 49.7 |
Following your suggestions, we have revised the paper and add these results to Section 5.3 (Lines 413–415) and Appendix A.15 (Lines 1674–1694).
Summarized Q3 (2): Why is the benefit substantial rather than incremental?
A4 (2): We respectfully clarify that the benefit is substantial rather than incremental for two reasons. First, the benefit grows as compression tightens, which is exactly where compression is most needed, and the gap of Alternating grows from 0.13 perplexity at 40% to 0.34 at 60% compression, as shown in Table R4. Second, the benefit comes at almost no extra cost, since the allocation of our method takes less than 15 seconds, as shown in Table 8 (Lines 422–424).
Following your suggestions, we have revised Section 1 (Lines 91–92) to state this benefit more clearly.
Best Wishes,
The Authors of Paper 53226
Response to Reviewer peZt
Dear Reviewer peZt:
Thank you for your careful review and constructive feedback on our work. We have responded point by point to your concerns regarding the presentation of the abstract and introduction, the significance and novelty of our work, and the advantage over sequential allocation. In the marked copy appended to the revised PDF (pages 34–66), the revisions made in response to your comments are tagged with @Reviewer peZt. We hope our responses resolve your concerns and look forward to further discussion.
Summarized W1 & Q1: Abstract and introduction difficult to follow, contribution hard to identify
A1: To make the contribution easier to identify and assess, we first summarize our paper as follows.
We understand that your main concern lies in the writing of the abstract and introduction, mainly because several terms are used before they are clearly explained. To make them easier to follow, we have thoroughly revised the abstract and introduction while keeping the overall logic unchanged, and we have also adjusted the rest of the paper accordingly for consistency. Please kindly refer to Pages 1 to 2 of the revised PDF (Lines 10–93). To make these changes easy to check, Table R1 lists the main term revisions, where the first column gives the terms used in the original paper, the second column gives their meaning, and the third column gives the unified term. In line 61 (line numbers of the reviewed version for previous wording, of the revised version otherwise), "That staging" referred to deciding which components survive before deciding how finely what survives is encoded (e.g., a shared-base decomposition fixes its base before it truncates the residuals against it); the revised Section 1 states this sequence explicitly (Lines 61–63).
Table R1: Main term revisions.