acceptodds
Under review as a conference paper at ICLR 2027

RDO-MoE: Rate-Distortion Optimized Joint Rank-Sparsity Allocation for Mixture-of-Experts Compression

Abstract

Sparse mixture-of-experts (MoE) language models store far more parameters than any one token activates; whether one runs on a commodity accelerator therefore hinges on how its expert bank is compressed, not on how sparsely it is routed. Existing pipelines settle that representation in stages, pruning before rating the survivors or scoring the base and the residual ranks under separate objectives, and can thereby miss the configurations favorable only when both are set together. We propose RDO-MoE (Rate–Distortion Optimized MoE compression), which prices shared-base sparsity and expert-residual rank together on one rate–distortion surface under a global serialized-byte budget. Joint per-layer surfaces turn each layer's weights and routing statistics into one finite table of nondominated pairs. Multiplier-indexed allocation dualizes that budget into one Lagrange multiplier and bisects it for a feasible integer allocation over the per-layer frontiers. A capability diagnostic outside the reconstruction objective reports signed per-task differences, not one score. The key insight is that the base and the rank index one surface rather than two budgets; pricing both against a single multiplier thus turns an ordered pair of searches into one query. On Qwen1.5-MoE-A2.7B at 40% compression it reaches 8.91 perplexity and a 55.7 aggregate against 9.16 and 55.4 for the best prior method; on Mixtral-87B it reaches 5.49 against 5.88 at the same budget and the lowest perplexity at every budget. On the instance exact search can enumerate it stays within 0.0057 in relative perplexity at 19.6 the solver speed, while the 0.9- and 1.3-point WinoGrande and BoolQ regressions at 40% stay visible rather than being averaged away by the nine-task aggregate.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture-of-experts compression, rate-distortion optimization, low-rank expert decomposition, Lagrangian budget allocation, shared-base sparsity, large language model compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_peZtOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2(+1)
Soundness
1
Read review
Paper summary

Budget allocation for MoE compression through joint rate–distortion optimization of shared-base sparsity and residual low-rank compression. The method evaluates sparsity–rank combinations per layer and uses a common Lagrange multiplier to select configurations under a global storage budget.

Critical strengths

None

Critical weaknesses
  1. Unclear presentation and problem formulation. The abstract and introduction rely on terms such as “shared-base sparsity” and “expert-residual rank” before adequately explaining the underlying representation and compression operations. References such as “That staging” in line 61 further obscure the argument. Although the mechanism becomes more concrete in Section 4.2, the introduction is difficult to follow because it discusses the interaction between compression decisions before clearly establishing those decisions. This substantially increases the effort required to identify and assess the contribution.

  2. Insufficiently established significance and advantage over staged allocation. The importance of MoE compression does not itself establish the importance of the specific allocation problem addressed here. The paper explains why sparsity and residual rank interact, but does not convincingly establish that handling this interaction provides a substantial advance over a carefully tuned sequential pipeline. The reported improvements and joint-versus-additive ablation are relevant, but the latter does not directly test a strong sequential or alternating optimizer under otherwise identical conditions. Since the allocation mechanism draws on established rate–distortion optimization, a clearer demonstration of this specific advantage is central to establishing the contribution’s significance. Nonetheless, their improvement over baseline method is incremental (see Table 2. for instance).

Questions for authors
  1. Could the authors clearly define “expert bank,” “shared-base sparsity,” and “expert-residual rank,” and clarify which specific sequence of operations “That staging” refers to in line 61?

  2. Beyond the general importance of reducing MoE storage, what evidence establishes sequential sparsity and rank allocation as a practically significant limitation? How much quality or storage efficiency is lost specifically because these decisions are made sequentially?

  3. What practical benefit does joint allocation provide over a carefully tuned sequential pipeline, and why should this benefit be considered a substantial advance rather than an incremental improvement?

5 comments
AuthorsOpenReview ↗

Response to Reviewer peZt

Dear Reviewer peZt:

Thank you for your careful review and constructive feedback on our work. We have responded point by point to your concerns regarding the presentation of the abstract and introduction, the significance and novelty of our work, and the advantage over sequential allocation. In the marked copy appended to the revised PDF (pages 34–66), the revisions made in response to your comments are tagged with @Reviewer peZt. We hope our responses resolve your concerns and look forward to further discussion.


Summarized W1 & Q1: Abstract and introduction difficult to follow, contribution hard to identify

A1: To make the contribution easier to identify and assess, we first summarize our paper as follows.

  • Problem. Existing methods set the shared-base sparsity and the expert-residual rank separately, missing their interaction.
  • Method. Our method prices the two (i.e., the shared-base sparsity and the expert-residual rank) jointly under one byte budget.
  • Experiments. We conduct experiments on twelve evaluation datasets with two MoE models (i.e., Qwen1.5-MoE-A2.7B and Mixtral-8×7B), compare against five baselines (two of them published in 2026), and use both performance metrics (e.g., perplexity and accuracy) and efficiency metrics (e.g., wall-clock time and memory). In total, we conduct six types of experiments, reported in twenty-one tables and seven figures (result tables and figures, not counting the tables marked as restated).
    • Experiment 1. Performance comparison against five baselines, showing the effectiveness of our method.
    • Experiment 2. Ablation study, showing the necessity of each module of our method.
    • Experiment 3. Comparison with the exact search and a heuristic search, showing that our method is near-optimal and much faster.
    • Experiment 4. Per-task evaluation, showing how our method performs on each downstream task.
    • Experiment 5. Sensitivity analysis on grids, solver tolerance, routing weights, and calibration resamples, showing the robustness of our method.
    • Experiment 6. Efficiency analysis on construction time, allocation time, memory, and throughput, showing the practicality of our method.

We understand that your main concern lies in the writing of the abstract and introduction, mainly because several terms are used before they are clearly explained. To make them easier to follow, we have thoroughly revised the abstract and introduction while keeping the overall logic unchanged, and we have also adjusted the rest of the paper accordingly for consistency. Please kindly refer to Pages 1 to 2 of the revised PDF (Lines 10–93). To make these changes easy to check, Table R1 lists the main term revisions, where the first column gives the terms used in the original paper, the second column gives their meaning, and the third column gives the unified term. In line 61 (line numbers of the reviewed version for previous wording, of the revised version otherwise), "That staging" referred to deciding which components survive before deciding how finely what survives is encoded (e.g., a shared-base decomposition fixes its base before it truncates the residuals against it); the revised Section 1 states this sequence explicitly (Lines 61–63).

Table R1: Main term revisions.

Original termsMeaningUnified term
Expert bank, parameter poolAll expert weights that an MoE model must storeExpert bank
Shared-base sparsity, base sparsity, base decisionFraction of the shared weights of all experts that are prunedShared-base sparsity
Expert-residual rank, residual rank, rank decisionRank kept for the difference between each expert and the shared weightsExpert-residual rank
Staging, in stages, That stagingDeciding which parts survive first, then how finely to encode themSequential allocation

Summarized W2: What is the novelty beyond established rate–distortion optimization?

A2: We respectfully clarify the significance and novelty of our work by addressing four arguments. In short, existing methods set the shared-base sparsity and the expert-residual rank separately, and our method prices the two jointly under one byte budget.

  • Argue 1: Is the allocation problem unimportant? The allocation problem is important, since under a fixed byte budget, compressing an MoE model is essentially deciding where each byte goes, namely to which layer, and to the shared base or the expert residuals. Therefore, the allocation directly decides the compression quality, and replacing our allocation with a greedy allocation on the same per-layer tables and the same budget raises perplexity by up to 0.318 at 60% compression.
  • Argue 2: Is handling the interaction no better than sequential pipelines? Handling the interaction between the shared-base sparsity and the expert-residual rank brings a clear gain over sequential decisions. Under identical grids, budget, and routing weights, separating the two decisions (scoring them on independent surfaces, Table 5, Lines 384–394) raises perplexity from 8.91 to 9.21, and the gap of the sequential pipeline D2-MoE grows from 0.20 to 0.88 perplexity as compression increases from 30% to 60%. We further compare with carefully tuned sequential and alternating pipelines in our response to your Q3.
  • Argue 3: Is there no novelty beyond existing methods (i.e., established rate–distortion optimization)? Rate–distortion optimization is the tool we use, and our novelty lies in overcoming three obstacles when applying it to this problem. First, standard rate–distortion allocation assumes independent options, while here the shared-base sparsity changes every expert residual, so we score the two decisions jointly in one per-layer table. Second, this coupling makes exact search grow multiplicatively with depth, so we price all layers with one multiplier, which solves the allocation 19.6× faster than exact dynamic programming on the bounded instance. Third, since the reconstruction objective cannot tell which tasks get worse, we report the gain or loss on each task separately.
  • Argue 4: Is the improvement marginal? The improvement is consistent and grows as compression tightens. Table R2 shows that our method reduces perplexity over the strongest baseline (D-Rank) in all eight settings by up to 8.4%, while keeping the aggregate score comparable or better.

Table R2: Perplexity reduction over the strongest baseline (D-Rank).

Model30%40%50%60%
Qwen1.5-MoE-A2.7B8.65 → 8.51 (−1.6%)9.16 → 8.91 (−2.7%)10.34 → 9.94 (−3.9%)12.74 → 12.05 (−5.4%)
Mixtral-8×7B5.68 → 5.33 (−6.2%)5.88 → 5.49 (−6.6%)6.33 → 5.86 (−7.4%)7.18 → 6.58 (−8.4%)

Following your suggestions, we have revised Section 1 (Lines 72–86) to state the significance and novelty more clearly.


Summarized Q2 (1): What evidence shows sequential allocation is a significant limitation?

A3 (1): We respectfully clarify that sequential allocation is a significant limitation, since fixing one decision before the other leaves quality behind even on identical surfaces. Two published pipelines that start from opposite halves of the decision (D2-MoE fixes the shared base first; D-Rank sets ranks with no shared base) disagree at every compression ratio, and their disagreement grows from 0.69% at 30% to 1.47% at 60% compression; on identical surfaces, a tuned sequential optimizer still trails our method by 0.23 and 0.52 perplexity at 40% and 60% (Table R4).

Following your suggestions, we have revised Section 2 (Lines 99–121) to make this evidence clearer.


Summarized Q2 (2): Quality and storage efficiency lost by sequential decisions

A3 (2): As you suggested, we quantify the quality and storage efficiency that the sequential pipeline D2-MoE loses against our method (i.e., higher perplexity and extra storage). Table R3 shows that the sequential pipeline D2-MoE loses up to 0.88 perplexity at the same budget and needs up to 5.0% extra storage at the same perplexity. The storage loss is the extra storage D2-MoE needs for the same perplexity, in percent of the uncompressed size; its 60% entry uses the perplexities of our method at 62%, 64%, and 66% compression (12.63, 13.33, and 14.06 perplexity). D2-MoE also differs from our method in its base and its factorization; the comparison on identical surfaces is in Table R4.

Table R3: Losses of the sequential pipeline D2-MoE.

Loss30%40%50%60%
Quality loss+0.20 PPL+0.36 PPL+0.52 PPL+0.88 PPL
Storage loss+5.0%+3.5%+2.5%+2.9%

Following your suggestions, we have revised Appendix A.15 (Lines 1654–1673) to make these losses clearer.

Summarized Q3 (1): Practical benefit over a carefully tuned sequential pipeline

A4 (1): Following your suggestion, we further compare our method with two strong optimizers (i.e., Sequential and Alternating) under identical grids, byte budget, and routing weights, as you recommended in W2. Table R4 shows that our method outperforms both optimizers, where even Alternating, the stronger one, remains 0.34 perplexity behind at 60% compression.

Table R4: Comparison with strong sequential and alternating optimizers.

Method40% PPL ↓40% Score ↑60% PPL ↓60% Score ↑
Sequential9.14 (+0.23)55.512.57 (+0.52)49.4
Alternating9.04 (+0.13)55.512.39 (+0.34)49.5
Our method8.9155.712.0549.7

Following your suggestions, we have revised the paper and add these results to Section 5.3 (Lines 413–415) and Appendix A.15 (Lines 1674–1694).


Summarized Q3 (2): Why is the benefit substantial rather than incremental?

A4 (2): We respectfully clarify that the benefit is substantial rather than incremental for two reasons. First, the benefit grows as compression tightens, which is exactly where compression is most needed, and the gap of Alternating grows from 0.13 perplexity at 40% to 0.34 at 60% compression, as shown in Table R4. Second, the benefit comes at almost no extra cost, since the allocation of our method takes less than 15 seconds, as shown in Table 8 (Lines 422–424).

Following your suggestions, we have revised Section 1 (Lines 91–92) to state this benefit more clearly.


Best Wishes,

The Authors of Paper 53226

Reviewer_Jg38OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the joint allocation of shared-base sparsity and expert-residual rank for MoE compression. It computes storage and reconstruction error for each sparsity-rank combination in each layer. It then searches over a single Lagrange multiplier to select per-layer configurations that satisfy a global storage budget. In the experiments, all experts within a layer share a common residual rank. The paper reports perplexity improvements across multiple compression ratios on Qwen1.5-MoE-A2.7B and Mixtral-8-7B. It also reports per-task score differences, showing that improvements in the aggregate score can coexist with regressions on individual tasks.

Critical strengths
  1. The problem formulation appropriately considers the interaction between the shared base and residual rank. Changing the shared base changes the residual matrices. Consequently, it can also change the error introduced by subsequent rank truncation. Evaluating both decisions within the same candidate configuration directly captures this interaction.

  2. The reported perplexity improvements extend beyond a single model or compression ratio. RDO-MoE achieves the lowest perplexity at every evaluated compression ratio on both models. The component analysis in Table 5 also partially separates the contributions of the joint objective and routing weights.

  3. The paper specifies the scope of the allocation algorithm’s guarantees. The proof of storage monotonicity in Proposition S2 is valid. The algorithm also explicitly checks whether the selected configuration satisfies the budget.

Critical weaknesses
  1. The approximation quality of the allocation algorithm is not evaluated in terms of its actual optimization objective.

Equation (1) minimizes the sum of per-layer reconstruction errors. However, the comparison with exact DP reports only perplexity differences. Solving Equation (1) optimally does not imply minimizing perplexity. Therefore, the relative perplexity gap of 0.0057 in Table 3 does not establish how closely the multiplier search approximates the optimum of Equation (1). The paper does not explicitly present this number as an optimality guarantee for reconstruction error. Nevertheless, evaluating the central allocation method requires comparing objective values. Reporting the relative reconstruction-error gap, the gap to the dual lower bound in Equation (6), and the unused budget would be useful. The bounded instance used for the exact-DP comparison also needs a precise definition. The stated cap on partial states is insufficient to reconstruct that instance.

  1. Key implementation details needed for reproducibility and baseline comparison are insufficiently specified.

Section 4.2 treats the shared base WlbW_{lb} as a fixed input. However, the paper does not sufficiently specify how this base was extracted in the experiments. The shared base affects the singular-value distributions of all residuals and therefore directly influences the results. It is also unclear how Equation (3) is applied to the multiple projection matrices within each expert. The paper should specify how projection-level errors are aggregated and at what granularity sparsity and rank are shared. Appendix A.8 states that all baselines were rerun under the paper’s common protocol. However, method-specific hyperparameters and the application of storage formats are insufficiently detailed. Expert pruning and low-rank decomposition use different representations. Consistent byte accounting and fair implementation conditions therefore require separate verification.

Questions for authors
  1. Regarding Weakness 1, what is the reconstruction-objective gap relative to exact DP? Could you precisely specify the layers, candidates, and budget of the bounded comparison instance? Please also report both methods’ objective values and the dual lower bound from Equation (6).

  2. Regarding Weakness 2, how was the shared base extracted, and how were errors and compression settings combined across projection matrices? Please also specify each baseline’s main hyperparameters and explain how the common storage-accounting convention was applied.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Jg38

Dear Reviewer Jg38:

Thank you for recognizing the well-motivated formulation, consistent improvements, and clearly scoped guarantees of our work. We have responded point by point to your concerns regarding the approximation quality on the actual objective, and the implementation details for reproducibility and fair comparison. In the marked copy appended to the revised PDF (pages 34–66), the revisions made in response to your comments are tagged with @Reviewer Jg38. We hope our responses resolve your concerns and look forward to further discussion.


Summarized W1 & Q1 (1): What layers, candidates, and budget define the bounded instance?

A1 (1): The bounded comparison instance is defined by its layers, candidates, and budget, namely the first 20 layers of Qwen1.5-MoE-A2.7B as the layers, 56 candidates per layer (i.e., 7 shared-base sparsities × 8 expert-residual ranks, as listed in Table 15 of the revised PDF, Lines 1247–1250) as the candidates, and a byte budget of 40% compression over these layers as the budget.

Following your suggestions, we have revised Appendix A.2.2 (Lines 822–827) to make this definition clearer.


Summarized W1 & Q1 (2): Reconstruction-objective gap to exact DP and dual lower bound

A1 (2): Following your suggestion, we further report the objective values of both methods (i.e., exact DP and our method) and the dual lower bound from Eq. (6) (Lines 232–236) on the bounded comparison instance, together with the reconstruction-objective gap (including the gap to exact DP and the gap to the dual lower bound) and the unused budget. Table R1 shows that our method closely approximates the optimum of Eq. (1) (Lines 145–147), with a reconstruction-objective gap of 0.35% to exact DP.

Table R1: Reconstruction-objective gap on the bounded instance.

MethodObjective valueGap to exact DPGap to dual lower boundUnused budget
Dual lower bound407.34–––
Exact DP408.4300.27%0.02%
Our method409.870.35%0.62%0.22%

Following your suggestions, we have revised the paper and add these results to Section 5 (Lines 365–367) and Appendix A.13 (Lines 1320–1338).


Summarized W2 & Q2 (1): How was the shared base extracted?

A2 (1): We respectfully clarify that the shared base of each layer is extracted as the routing-frequency-weighted average of all expert matrices in that layer, and it is shared by all experts in that layer. Since this shared base changes the singular-value distribution of every residual, each candidate configuration recomputes all residuals after pruning the shared base and scores them again, so the influence of the shared base on the results is directly taken into account by the allocation.

Following your suggestions, we have revised Section 4.2 (Lines 199–200) to make the extraction and its influence clearer.


Summarized W2 & Q2 (2): How are errors and settings combined across projection matrices?

A2 (2): We respectfully clarify how Eq. (3) (Lines 205–208) is applied to the multiple projection matrices in three parts. First, Eq. (3) is computed on each projection matrix of each expert (i.e., the gate, up, and down projections) separately. Second, the projection-level errors are aggregated by summing over the three projections, and then weighted by routing frequency across experts. Third, sparsity and rank are shared at the layer granularity, namely one shared-base sparsity and one expert-residual rank for all projections and all experts in a layer.

Following your suggestions, we have revised Section 4.2 (Lines 212–213) to make this granularity clearer.


Summarized W2 & Q2 (3): What are each baseline's main hyperparameters?

A2 (3): We respectfully clarify that the method-specific hyperparameters and storage formats of all baselines are listed in Table R2, where each baseline keeps its own representation and is taken at the same byte budget and on the same calibration data as our method.

Table R2: Hyperparameters and storage formats of baselines.

BaselineRepresentationStorage formatKey hyperparameters
Uniform+MagnitudeSparse weightsValues + 1-bit indexOne sparsity for all expert matrices, set by the byte budget
SAES-SVDLow-rank factorsDense factorsAdaptive coefficient in [0.25, 0.75]; no mixed rank
D-RankLow-rank factorsDense factorsEffective-rank allocation; group size 1
D2-MoEShared base + delta factorsSparse base + dense factorsFisher-weighted base, pruned 10%; truncation-aware SVD
EvoESAPPruned expertsDense retained expertsREAP order; population 32, 4 elites, 50 generations

Following your suggestions, we have revised the paper and add Table R2 to Appendix A.8 (Lines 1095–1106).

(Continued)

Summarized W2 & Q2 (4): How is the common storage-accounting convention applied to baselines?

A2 (4): We respectfully clarify that the common storage-accounting convention converts every representation into serialized bytes with the same rate, namely 16 bits per stored value, 1 bit of index per entry of a sparse matrix, and a 64-byte header per layer under the same rounding. In this way, expert pruning and low-rank decomposition are charged in the same bytes, and all methods are matched in bytes rather than in parameter count at each compression ratio.

Following your suggestions, we have revised Appendix A.7 (Lines 1073–1079) to make this convention clearer.


Best Wishes,

The Authors of Paper 53226

Thank you for the time and effort you put into responding to my review.

W1 & Q1. Comparison with exact DP

Table R1 addresses my concern about how closely RDO-MoE approximates the optimum of Equation (1). The original table reported a perplexity gap. The new table shows a 0.35% gap in the reconstruction objective itself on the specified 20-layer Qwen instance. I also appreciate the clarification of the comparison setup and the inclusion of the dual lower bound and unused budget. My concern on this point is resolved.

W2 & Q2. Implementation details and baseline comparison

I now understand how you construct the shared base, combine the reconstruction errors across the three projections, and share sparsity and rank within each layer. Table R2 and your explanation of the byte-accounting rule also clarify how you compared baselines that use different storage formats. These details address my concerns about reproducibility and the comparison setup.

Thank you for the additional analysis and clarifications. Please retain these results and implementation details in the final version of the paper.

Reviewer_qkVYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RDO-MoE, which jointly optimizes the shared base sparsity and the residual rank of experts under a unified storage byte budget, and efficiently coordinates the compression configurations of each layer through Lagrange multipliers. Experiments show that this method stably improves perplexity in two MoE models and multiple compression ratios, and simultaneously reveals the local performance degradation behind the average score improvement through task-by-task evaluation.

Critical strengths

S1: The research question holds practical value, clearly revealing the coupling relationship between the shared base and the residual rank, providing a reasonable approach for improving the staged MoE compression. S2: The experiment is comprehensive. Through cross-model, multi-budget comparisons and ablation experiments, the effectiveness of the joint allocation is verified, while also taking into account compression quality, solution efficiency and task-specific performance.

Critical weaknesses

W1: The routing weights are estimated from WikiText-2 and are fixed before compression. Therefore, their robustness under domain shift is not yet clear. Could the authors compare different calibrated domains and examine the impact of routing statistics changes on compression quality? W2: The multiplier-based distributor can skip unsupported discrete configurations, and its comparison with exact dynamic programming is limited to a bounded instance. Could the authors report the distortion gap and unused budget ratio in multiple handleable instances to more clearly illustrate when this approximation becomes no longer effective?

Questions for authors

see critical weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer qkVY

Dear Reviewer qkVY:

Thank you for recognizing the practical value, comprehensive experiments, and clear presentation of our work. We have responded point by point to your concerns regarding the robustness under domain shift and the approximation quality of our allocation. In the marked copy appended to the revised PDF (pages 34–66), the revisions made in response to your comments are tagged with @Reviewer qkVY. We hope our responses resolve your concerns and look forward to further discussion.


Summarized W1: Compression quality across different calibration domains

A1: Following your suggestion, we further examine the impact of routing statistics changes (i.e., routing shift) on compression quality (including PPL and Score) across three calibration domains (including WikiText-2 for encyclopedic text, C4 for web text, and The Stack for code) for our method on Qwen1.5-MoE-A2.7B at 40% compression. Table R1 shows that the compression quality changes only slightly as the routing shift grows, where the largest routing shift of 0.221 raises PPL from 8.91 to 9.05 and lowers Score from 55.71 to 55.47, so both domains stay within the response to uniform routing weights (9.13 PPL and 55.39 Score).

Table R1: Compression quality across calibration domains.

Calibration domainRouting shiftPPL ↓Score ↑
WikiText-208.9155.71
C40.0958.9755.66
The Stack0.2219.0555.47

Following your suggestions, we have added these results to Appendix A.15 (Lines 1609–1630), and updated the Limitations in Appendix A.0 (Lines 728–733).


Summarized W2: Distortion gap and unused budget ratio across multiple instances

A2: Following your suggestion, we further report the distortion gap and unused budget ratio in four handleable instances (i.e., the allocation problems over the first 8, 12, 18, and 20 layers of Qwen1.5-MoE-A2.7B at 40% compression, all solvable by exact dynamic programming; the 20-layer instance is the bounded instance of Table 3, Lines 354–360, and exact dynamic programming is not admitted on the full 24-layer model). Table R2 shows that the approximation remains effective in all handleable instances, with the distortion gap below 2%. The gap is largest on the smallest instance.

Table R2: Distortion gap and unused budget ratio in handleable instances.

InstanceDistortion gapUnused budget ratio
8 layers1.26%0.67%
12 layers1.03%0.46%
18 layers0.68%0.36%
20 layers0.35%0.22%

Following your suggestions, we have added these results to Appendix A.15 (Lines 1633–1651).


Best Wishes,

The Authors of Paper 53226

Reviewer_qkVYOpenReview ↗

Response to author

Thank you for your reply. For W1, new experiments have shown that even with significant changes in routing statistics, the compression performance of the model remains relatively stable. For W2, the authors supplement comparisons with exact dynamic programming on multiple solvable instances, further validating the effectiveness of the approximate allocation method with smaller distortion gaps and unused budget ratios.

I hope the authors keep these new experiments in the final version. Considering that I had already given a relatively high score initially, I have decided to maintain the current score.

AuthorsOpenReview ↗

Second-Round Response to Reviewer qkVY

Dear Reviewer qkVY,

Thank you for your timely response and for your positive assessment of our work.

We sincerely appreciate your recognition that the compression performance remains stable even under significant changes in routing statistics, and that the comparisons with exact dynamic programming further validate the effectiveness of our allocation method, with small distortion gaps and unused budget ratios. As suggested, we will keep these experiments in the final version.

Thank you again for your constructive feedback.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

General Response

We sincerely thank all reviewers for their time and constructive comments. Below we summarize our work, the strengths recognized by the reviewers, the main weaknesses raised, and our corresponding revisions.

Summary

Compressing a mixture-of-experts model requires choosing how sparse the weights shared by all experts are and how much rank each expert keeps, but existing methods set the two separately and miss their interaction. Our method prices the two jointly under one byte budget. We conduct six types of experiments (including main comparison, ablation, solver comparison, per-task evaluation, sensitivity analysis and cost analysis) on twelve datasets with two MoE models. Our method reduces perplexity by up to 8.4% over the previous SOTA.

Strengths

  • Sound design. The paper clearly reveals the coupling between the shared base and the residual rank (qkVY), directly captures this interaction (Jg38), and is a reasonable approach (peZt).
  • Solid experiments. The experiment is comprehensive across models and budgets (qkVY), with the lowest perplexity at every evaluated compression ratio (Jg38).
  • Important problem and clear presentation. The research question holds practical value, and the paper is clear and easy to follow (qkVY).
  • Sound theory. The scope of the guarantees is specified, and the proof of storage monotonicity is valid (Jg38).

Weaknesses

  1. Lack of experiments on different calibrated domains. (qkVY) After rebuttal: qkVY confirmed this concern is addressed.
  2. Lack of evaluation of approximation quality on multiple instances and on the reconstruction objective. (qkVY, Jg38) After rebuttal: Jg38 stated this concern is resolved, and qkVY confirmed it is addressed.
  3. Lack of comparison with strong sequential and alternating optimizers. (peZt) After rebuttal: peZt confirmed the new comparisons address this concern.
  4. Unclear significance and novelty beyond established rate–distortion optimization. (peZt)
  5. Unclear terms in the abstract and introduction. (peZt) After rebuttal: peZt confirmed this concern is addressed.
  6. Unclear details of the bounded instance, the implementation, and the baselines. (Jg38) After rebuttal: Jg38 stated this concern is resolved.

Revisions

  1. We added experiments on three calibrated domains, including web text and code (Table R1 for qkVY).
  2. We added the evaluation of approximation quality on four handleable instances and on the reconstruction objective, together with the dual lower bound (Table R2 for qkVY, Table R1 for Jg38).
  3. We added comparisons with strong sequential and alternating optimizers under identical conditions, and quantified the quality and storage losses of sequential decisions (Tables R3 and R4 for peZt).
  4. We clarified the significance and novelty beyond established rate–distortion optimization in four arguments, with the improvement over the previous SOTA in all eight settings (Table R2 for peZt). The gain is substantial rather than incremental: it widens as compression tightens, the sequential pipeline needs up to 5.0% extra storage for the same perplexity, and the allocation takes less than 15 seconds.
  5. We revised the abstract and introduction, and unified and defined the key terms (Table R1 for peZt).
  6. We explained the layers, candidates, and budget of the bounded instance, the shared-base extraction, the error aggregation, and the storage accounting, and listed the baseline hyperparameters (Table R2 for Jg38).

After the rebuttal

  • Calibrated domains and approximation quality: resolved, as confirmed by qkVY.
  • Approximation quality and implementation details: resolved, as confirmed by Jg38.
  • Comparison with sequential and alternating optimizers: resolved, as confirmed by peZt.

All revisions are marked in the revised PDF, and all experiments above will be kept in the final version.

Best Wishes,

The Authors of Paper 53226

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.