- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes BOLD-MoE, a post-training compression framework for Mixture-of-Experts (MoE) language models that exploits redundancy across experts while preserving expert-specific information. Building on shared-base residual compression, BOLD-MoE jointly optimizes the shared base and compressed expert residuals through alternating optimization, rather than treating them as independent stages. It further employs activation-aware reconstruction and orthogonal sparse dictionary learning to represent expert-specific residuals more effectively, along with model-wide compression budget allocation.
Critical strengths
-
The paper identifies a limitation of existing shared-base MoE compression methods: optimizing the shared base independently of residual compression may lead to suboptimal reconstruction quality.
-
The proposed base–residual optimization framework is compressor-agnostic, and the ablation results demonstrate improvements even when using conventional SVD-based residual compression.
Critical weaknesses
-
The paper builds on the shared-base and residual compression framework of D2-MoE, extending it with alternating base–residual optimization and orthogonal sparse dictionary learning. However, activation-aware reconstruction, block-coordinate descent, and orthogonal sparse dictionary learning have already been explored in prior work, including COMPOT. The authors should better clarify the methodological novelty beyond integrating these existing techniques.
-
The experimental evaluation and comparison are insufficient. First, some baseline results are taken from previously published papers rather than reproduced under a unified evaluation setting, potentially limiting the fairness of the comparisons. Second, the paper does not report end-to-end inference latency or throughput, making it unclear whether the proposed sparse dictionary representation provides practical inference benefits. Third, the evaluation omits recent state-of-the-art MoE compression methods, such as RFID-MoE (ICML 2026) and TD-MoE (ICLR 2026), both of which report strong compression results. Direct comparisons with these methods under matched settings would strengthen the experimental evaluation.
Questions for authors
See the weakness
3 comments
Response to Reviewer ick9. Part 2
W2.2. End-to-end decoding throughput and practical acceleration
We thank the reviewer for emphasizing the importance of evaluating practical acceleration rather than compression quality alone.
As discussed in Appendix A.2, Equations (39)–(43), the OSDL representation introduces sparsity in the coefficient matrices, enabling sparse matrix-vector multiplication instead of relying exclusively on dense operations. At our main operating point, , the coefficient matrices are approximately 70.6% sparse (Equation (42)).
To validate the practical implications of this representation, we conducted additional end-to-end autoregressive decoding experiments comparing BOLD-MoE with D²-MoE on Mixtral-8×7B at 20%, 40%, and 60% compression.
All experiments use FP16, batch size 1, and four NVIDIA A100-SXM4-40GB GPUs on a single node, without CPU or disk offloading. We evaluate prompt lengths of 128, 512, and 2048 tokens, each followed by 128 cached decoding steps. Throughput is derived from the measured decoding latencies, aggregated across these prompt lengths.
The principal difference in runtime execution is the incorporation of a MACKO-based sparse kernel for BOLD-MoE. D²-MoE executes its low-rank factors using optimized dense matrix multiplications, whereas OSDL additionally exposes coefficient sparsity that can be exploited by specialized sparse kernels. This distinction was already discussed in Appendix A.2, but had not been experimentally evaluated end-to-end in the original submission.
| CR | D²-MoE (tokens/s) ↑ | BOLD-MoE (tokens/s) ↑ | Speedup ↑ |
|---|---|---|---|
| 20% | 11.53 | 11.96 | 1.037× |
| 40% | 12.29 | 12.55 | 1.022× |
| 60% | 12.80 | 13.16 | 1.028× |
End-to-end autoregressive decoding throughput on Mixtral-8×7B (FP16, batch size 1, 4×A100-40GB).
BOLD-MoE consistently achieves higher decoding throughput across all three compression levels, with speedups of approximately 2–4%.
Although the improvement remains modest at the full-model level, these experiments confirm that the coefficient sparsity introduced by OSDL provides a practical acceleration opportunity beyond conventional dense low-rank execution. This complements the arithmetic-complexity analysis presented in Appendix A.2 and provides the end-to-end measurements missing from the original submission.
We will incorporate these results into the revised manuscript and update the corresponding limitation discussed in Section 5.
W2.3. Comparison with TD-MoE (ICLR 2026)
We thank the reviewer for suggesting TD-MoE as an additional baseline. We investigated its official implementation, the difference between nominal per-layer and full-model compression ratios, the all-layer diagnostic reproduction, and the measured relative changes in prefill throughput, decoding throughput, and peak GPU memory. The full table, calibration-path limitations, and deployment discussion are provided in General Response, §G2.3.
The separate inference measurements characterize TD-MoE relative to its own uncompressed reference and are not an absolute throughput comparison against BOLD-MoE's benchmark in W2.2. We also include RS-MoE following qTkM's suggestion (§G2.5).
W2.4. Comparison with RFID-MoE (ICML 2026)
We thank the reviewer for pointing out RFID-MoE. Using the supplementary implementation provided with its submission, we performed additional experiments on Mixtral-8×7B using our calibration and evaluation setup. The reproduced results for WikiText-2, PTB, C4, and seven zero-shot tasks, together with the limitation that this is not an independently validated official release, are reported in General Response, §G2.4.
Following other reviewers' suggestions, the same shared response additionally covers NAEE, MoE-I², and RS-MoE. We will extend Section 2 of our submission accordingly.
Dear, Reviewer ick9,
we sincerely appreciate your constructive review and suggestions. We would be grateful if you could share any follow-up feedback on our responses, particularly regarding the additional baseline comparisons and end-to-end throughput experiments.
If any concerns remain insufficiently addressed, we would be happy to provide further clarification.
Response to Reviewer ick9. Part 1
We sincerely thank the reviewer for the careful assessment of our work, for recognizing its strengths, and for providing constructive suggestions. We particularly appreciate the questions regarding methodological novelty, baseline comparability, and practical efficiency. Following the reviewer's suggestions, we conducted additional experiments, including independent baseline reproductions, comparisons with recent methods, and end-to-end decoding benchmarks. We address each concern below.
We refer to the consolidated General Response for overlapping baseline questions and retain the original numbering below.
W1. Methodological novelty beyond combining existing techniques
We agree that activation-aware reconstruction, block-coordinate optimization, and orthogonal sparse dictionary learning (OSDL) are established techniques. Our contribution is not their individual introduction, but a compressor-aware formulation of shared-base MoE compression.
As formulated in Section 3.1, Equation (5), BOLD-MoE jointly optimizes the shared base and compressed expert-specific residuals under an activation-aware reconstruction objective. Unlike D²-MoE, which constructs the shared base before residual compression and subsequently keeps it fixed, BOLD-MoE explicitly couples these components. The closed-form base update in Equation (8) accounts for the reconstruction errors introduced by residual compression, allowing the shared component to compensate for information discarded by the residual compressor.
Importantly, this formulation is independent of the residual representation, as established in Section 3.2. Our SVD-only ablation (Figure 2, Section 4.2) already demonstrates that alternating base–residual optimization improves over D²-MoE without introducing OSDL.
Furthermore, Table 3 isolates the contribution of alternating optimization under a fixed budget of 50 OSDL iterations. Increasing the frequency of base updates improves WikiText-2/C4 perplexity from 4.94/8.64 to 4.74/8.34, while improving average zero-shot accuracy from 58.9% to 60.1%.
Following the reviewer's concern, we additionally compared OSDL applied to residuals with a fixed shared base against BOLD-MoE with alternating base–residual optimization.
At 40% and 60% compression, BOLD-MoE improves average zero-shot accuracy by 1.4 and 1.0 percentage points, respectively, while consistently reducing perplexity.
Together with the controlled experiments already provided in Section 4.2, these results demonstrate that the improvements are not solely attributable to OSDL, but also to optimizing the shared representation in response to residual compression.
W2.1. Comparability of evaluation protocols and compression ratios
We thank the reviewer for highlighting the importance of consistent baseline comparisons. We address the NAEE reproduction and discrete expert-pruning budgets in General Response, §G2.1, and the unavailable MoE-I² implementation and resulting limitations in §G2.2. Both discussions retain the experimental results and the accounting references to Section 4.1 and Equation (21) of our submission.
In addition to the NAEE, MoE-I², TD-MoE, and RFID-MoE issues raised in this review, we also investigated RS-MoE following qTkM's suggestion (General Response, §G2.5). uf9y independently raised the NAEE expert-count and full-model storage-accounting issue, which is addressed jointly in §G2.1. We will explicitly distinguish reproduced baselines from externally reported reference results in the revised manuscript.