acceptodds
Under review as a conference paper at ICLR 2027

BOLD-MoE: Base-Optimized Learning of Orthogonal Dictionaries for Mixture-of-Experts Compression

Abstract

Recent Mixture-of-Experts (MoE) compression methods exploit inter-expert redundancy by representing each expert as a shared base plus a compressed expert-specific residual. However, the base is typically constructed before residual compression, ignoring that its optimal choice depends on how well the resulting residuals can be represented under a limited budget. We formulate shared-base MoE compression as a coupled optimization problem and solve it by alternating between residual compression and an activation-aware base update. This formulation is compressor-agnostic: even when retaining SVD residuals, alternating base–residual optimization improves over the one-shot decomposition. We further introduce orthogonal sparse dictionary residuals, which replace the single low-rank subspace imposed by SVD with a richer union-of-subspaces representation. Across three MoE architectures and 20-60% compression, BOLD-MoE consistently improves perplexity and zero-shot accuracy over the strongest shared-base baseline, with substantial gains persisting under strong compression.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
moe compression, alternating minimization, sparse dictionary learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ick9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BOLD-MoE, a post-training compression framework for Mixture-of-Experts (MoE) language models that exploits redundancy across experts while preserving expert-specific information. Building on shared-base residual compression, BOLD-MoE jointly optimizes the shared base and compressed expert residuals through alternating optimization, rather than treating them as independent stages. It further employs activation-aware reconstruction and orthogonal sparse dictionary learning to represent expert-specific residuals more effectively, along with model-wide compression budget allocation.

Critical strengths
  • The paper identifies a limitation of existing shared-base MoE compression methods: optimizing the shared base independently of residual compression may lead to suboptimal reconstruction quality.

  • The proposed base–residual optimization framework is compressor-agnostic, and the ablation results demonstrate improvements even when using conventional SVD-based residual compression.

Critical weaknesses
  • The paper builds on the shared-base and residual compression framework of D2-MoE, extending it with alternating base–residual optimization and orthogonal sparse dictionary learning. However, activation-aware reconstruction, block-coordinate descent, and orthogonal sparse dictionary learning have already been explored in prior work, including COMPOT. The authors should better clarify the methodological novelty beyond integrating these existing techniques.

  • The experimental evaluation and comparison are insufficient. First, some baseline results are taken from previously published papers rather than reproduced under a unified evaluation setting, potentially limiting the fairness of the comparisons. Second, the paper does not report end-to-end inference latency or throughput, making it unclear whether the proposed sparse dictionary representation provides practical inference benefits. Third, the evaluation omits recent state-of-the-art MoE compression methods, such as RFID-MoE (ICML 2026) and TD-MoE (ICLR 2026), both of which report strong compression results. Direct comparisons with these methods under matched settings would strengthen the experimental evaluation.

Questions for authors

See the weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer ick9. Part 1

We sincerely thank the reviewer for the careful assessment of our work, for recognizing its strengths, and for providing constructive suggestions. We particularly appreciate the questions regarding methodological novelty, baseline comparability, and practical efficiency. Following the reviewer's suggestions, we conducted additional experiments, including independent baseline reproductions, comparisons with recent methods, and end-to-end decoding benchmarks. We address each concern below.

We refer to the consolidated General Response for overlapping baseline questions and retain the original numbering below.

W1. Methodological novelty beyond combining existing techniques

We agree that activation-aware reconstruction, block-coordinate optimization, and orthogonal sparse dictionary learning (OSDL) are established techniques. Our contribution is not their individual introduction, but a compressor-aware formulation of shared-base MoE compression.

As formulated in Section 3.1, Equation (5), BOLD-MoE jointly optimizes the shared base and compressed expert-specific residuals under an activation-aware reconstruction objective. Unlike D²-MoE, which constructs the shared base before residual compression and subsequently keeps it fixed, BOLD-MoE explicitly couples these components. The closed-form base update in Equation (8) accounts for the reconstruction errors introduced by residual compression, allowing the shared component to compensate for information discarded by the residual compressor.

Importantly, this formulation is independent of the residual representation, as established in Section 3.2. Our SVD-only ablation (Figure 2, Section 4.2) already demonstrates that alternating base–residual optimization improves over D²-MoE without introducing OSDL.

Furthermore, Table 3 isolates the contribution of alternating optimization under a fixed budget of 50 OSDL iterations. Increasing the frequency of base updates improves WikiText-2/C4 perplexity from 4.94/8.64 to 4.74/8.34, while improving average zero-shot accuracy from 58.9% to 60.1%.

Following the reviewer's concern, we additionally compared OSDL applied to residuals with a fixed shared base against BOLD-MoE with alternating base–residual optimization.

CRMethodWikiText-2 ↓PTB ↓C4 ↓Avg. ↑
40%OSDL on Residuals (Fixed Base)4.9318.248.6559.0
BOLD-MoE4.6917.558.2760.40
60%OSDL on Residuals (Fixed Base)6.2125.0810.8754.20
BOLD-MoE5.8223.6010.2355.20

At 40% and 60% compression, BOLD-MoE improves average zero-shot accuracy by 1.4 and 1.0 percentage points, respectively, while consistently reducing perplexity.

Together with the controlled experiments already provided in Section 4.2, these results demonstrate that the improvements are not solely attributable to OSDL, but also to optimizing the shared representation in response to residual compression.

W2.1. Comparability of evaluation protocols and compression ratios

We thank the reviewer for highlighting the importance of consistent baseline comparisons. We address the NAEE reproduction and discrete expert-pruning budgets in General Response, §G2.1, and the unavailable MoE-I² implementation and resulting limitations in §G2.2. Both discussions retain the experimental results and the accounting references to Section 4.1 and Equation (21) of our submission.

In addition to the NAEE, MoE-I², TD-MoE, and RFID-MoE issues raised in this review, we also investigated RS-MoE following qTkM's suggestion (General Response, §G2.5). uf9y independently raised the NAEE expert-count and full-model storage-accounting issue, which is addressed jointly in §G2.1. We will explicitly distinguish reproduced baselines from externally reported reference results in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer ick9. Part 2

W2.2. End-to-end decoding throughput and practical acceleration

We thank the reviewer for emphasizing the importance of evaluating practical acceleration rather than compression quality alone.

As discussed in Appendix A.2, Equations (39)–(43), the OSDL representation introduces sparsity in the coefficient matrices, enabling sparse matrix-vector multiplication instead of relying exclusively on dense operations. At our main operating point, k/s=3.4k/s=3.4, the coefficient matrices are approximately 70.6% sparse (Equation (42)).

To validate the practical implications of this representation, we conducted additional end-to-end autoregressive decoding experiments comparing BOLD-MoE with D²-MoE on Mixtral-8×7B at 20%, 40%, and 60% compression.

All experiments use FP16, batch size 1, and four NVIDIA A100-SXM4-40GB GPUs on a single node, without CPU or disk offloading. We evaluate prompt lengths of 128, 512, and 2048 tokens, each followed by 128 cached decoding steps. Throughput is derived from the measured decoding latencies, aggregated across these prompt lengths.

The principal difference in runtime execution is the incorporation of a MACKO-based sparse kernel for BOLD-MoE. D²-MoE executes its low-rank factors using optimized dense matrix multiplications, whereas OSDL additionally exposes coefficient sparsity that can be exploited by specialized sparse kernels. This distinction was already discussed in Appendix A.2, but had not been experimentally evaluated end-to-end in the original submission.

CRD²-MoE (tokens/s) ↑BOLD-MoE (tokens/s) ↑Speedup ↑
20%11.5311.961.037×
40%12.2912.551.022×
60%12.8013.161.028×

End-to-end autoregressive decoding throughput on Mixtral-8×7B (FP16, batch size 1, 4×A100-40GB).

BOLD-MoE consistently achieves higher decoding throughput across all three compression levels, with speedups of approximately 2–4%.

Although the improvement remains modest at the full-model level, these experiments confirm that the coefficient sparsity introduced by OSDL provides a practical acceleration opportunity beyond conventional dense low-rank execution. This complements the arithmetic-complexity analysis presented in Appendix A.2 and provides the end-to-end measurements missing from the original submission.

We will incorporate these results into the revised manuscript and update the corresponding limitation discussed in Section 5.

W2.3. Comparison with TD-MoE (ICLR 2026)

We thank the reviewer for suggesting TD-MoE as an additional baseline. We investigated its official implementation, the difference between nominal per-layer and full-model compression ratios, the all-layer diagnostic reproduction, and the measured relative changes in prefill throughput, decoding throughput, and peak GPU memory. The full table, calibration-path limitations, and deployment discussion are provided in General Response, §G2.3.

The separate inference measurements characterize TD-MoE relative to its own uncompressed reference and are not an absolute throughput comparison against BOLD-MoE's benchmark in W2.2. We also include RS-MoE following qTkM's suggestion (§G2.5).

W2.4. Comparison with RFID-MoE (ICML 2026)

We thank the reviewer for pointing out RFID-MoE. Using the supplementary implementation provided with its submission, we performed additional experiments on Mixtral-8×7B using our calibration and evaluation setup. The reproduced results for WikiText-2, PTB, C4, and seven zero-shot tasks, together with the limitation that this is not an independently validated official release, are reported in General Response, §G2.4.

Following other reviewers' suggestions, the same shared response additionally covers NAEE, MoE-I², and RS-MoE. We will extend Section 2 of our submission accordingly.

Dear, Reviewer ick9,

we sincerely appreciate your constructive review and suggestions. We would be grateful if you could share any follow-up feedback on our responses, particularly regarding the additional baseline comparisons and end-to-end throughput experiments.

If any concerns remain insufficiently addressed, we would be happy to provide further clarification.

Reviewer_uf9yOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BOLD-MoE jointly optimizes a shared expert base and compressed expert-specific residuals under an activation-weighted objective. It alternates base updates with SVD or orthogonal sparse dictionary compression and adopts COMPOT's global budget allocation. Experiments on three MoE architectures at 20%, 40%, and 60% full-model storage reduction report improvements over a controlled D²-MoE baseline.

Critical strengths
  1. Coherent optimization with informative controls. The base and conditional residual updates support the stated objective-value convergence guarantee. Figure 2 isolates base re-optimization, while Table 1 isolates the benefit of sparse dictionary residuals.
  2. Consistent controlled perplexity gains. Table 4 reports improvements in all 27 paired perplexity comparisons across three architectures and compression levels. Storage accounting includes shared bases, sparse masks, and unchanged parameters.
Critical weaknesses
  1. Accuracy reporting is internally inconsistent. Table 5's task scores disagree with the reported averages for all three compressed DeepSeekMoE configurations and two original-model rows. For example, the 60% DeepSeekMoE BOLD-MoE scores average to 44.47%, not 42.8%. Recomputed gains remain positive, but the correct scores and configuration associations require verification.
  2. Baseline comparability is incompletely established. Although CR denotes full-model storage reduction, the NAEE entries lack the integer expert configurations and achieved storage needed to verify budget matching. The controlled D²-MoE also disables base pruning, limiting claims about superiority over the complete original method.
Questions for authors
  1. Please reconcile Tables 4 and 5, including the original-model averages, and identify the checkpoint and k/s configuration associated with each affected row.
  2. For NAEE at 20%, 40%, and 60% CR, what are the result sources, per-layer survivor counts, additional compression operations, and achieved full-model storage reductions? Are these percentages measured reductions or nominal targets?
  3. Is a budget-matched comparison with complete D²-MoE, including base pruning, available under the same evaluation protocol? Otherwise, please label the modified control explicitly and narrow the comparative claim.
6 comments
AuthorsOpenReview ↗

Response to Reviewer uf9y

We sincerely thank the reviewer for the careful and constructive assessment of our work, particularly for recognizing the motivation behind the coupled optimization formulation and the value of our controlled ablations. We also appreciate the detailed comments regarding reporting consistency, baseline comparability, and compression-ratio accounting. Following these suggestions, we revisited the experimental results, independently reproduced additional baselines, and conducted further ablations. We address each concern below.

For concerns overlapping with qTkM and ick9, we consolidate the detailed results in the General Response and retain the reviewer's W/Q labels below.

W1 / Q1. Inconsistent zero-shot accuracy reporting in Table 5

We thank the reviewer for identifying these inconsistencies. Regarding the uncompressed baselines, we initially retained the average accuracies reported in D²-MoE for consistency with prior work. Our independent evaluation yielded comparable results, and we will update the corresponding entries in the revised manuscript to ensure consistency between task-wise scores and reported averages.

Regarding the compressed DeepSeekMoE-16B-Base results, we identified a reporting error: the task-wise BOLD-MoE results in Table 5 correspond to k/s=2.2, whereas the reported averages correspond to k/s=3.4. The main BOLD-MoE results in Table 4 use k/s=3.4 and are correct. The full corrected task-wise table is in General Response, §G3, which also addresses qTkM's W2.

We apologize for the reporting inconsistency and will correct Table 5 and clarify the provenance of the original-model reference averages in the revised manuscript.

W2 / Q2. Verification of NAEE compression ratios and full-model storage accounting

Following the reviewer's suggestion, we independently reproduced NAEE using its official implementation and evaluated the resulting models using our evaluation pipeline. We uniformly removed two, three, and five of the eight experts per layer, retaining 6/8, 5/8, and 3/8 experts and obtaining 24.1%, 36.2%, and 60.4% full-model compression, respectively. The complete results, including published reference values and actual achieved ratios, are in General Response, §G2.1.

BOLD-MoE achieves higher average zero-shot accuracy at each target, improving over our NAEE reproduction by 2.12, 1.91, and 3.09 percentage points. We also explicitly acknowledge NAEE's lower PTB perplexity at the approximately 40% comparison, and distinguish target from achieved compression ratios.

In addition to this NAEE verification, the common baseline discussion includes MoE-I², TD-MoE, and RFID-MoE at ick9's suggestion, and RS-MoE at qTkM's suggestion (General Response, §§G2.2–G2.5).

W3 / Q3. Comparison with the complete D²-MoE method including shared-base pruning

We thank the reviewer for raising this important point. Our primary objective was to isolate the contribution of base–residual optimization from shared-base pruning. We reproduced the pruning ablation in D²-MoE Table 7 using its original released implementation, before the preprocessing and compression-accounting corrections documented in Appendix B. The full original-versus-reproduced pruning table is in General Response, §G1, also addressing qTkM's W1.

Within this initial reproduction, increasing shared-base pruning consistently worsened perplexity across all three datasets. We therefore used an unpruned base for the controlled D²-MoE comparison, while the final D²-MoE and BOLD-MoE results in Table 4 were evaluated with the common corrected pipeline described in Appendix B.

We will explicitly label this baseline “D²-MoE (reproduced, unpruned base)” in the revised manuscript, making the scope of the controlled comparison clear.

Thank you for your responses. I have read through all of them and truly appreciate the effort you have put in. However, your responses have not fully addressed my concerns. I intend to take the feedback from other reviewers into account; in the meantime, I have decided to maintain my current rating.

We sincerely appreciate the reviewer's continued engagement, constructive feedback, and positive assessment of our work.

We would be grateful if you could specify which concerns remain insufficiently addressed, and we will do our best to provide further clarification.

Reviewer_qTkmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies post-training compression of MoE models represented by a shared base and expert-specific residuals. Its main proposal is to jointly optimize the base and compressed residuals through block-coordinate descent, alternating residual compression with a closed-form, activation-aware base update. The formulation supports SVD residuals and adopts orthogonal sparse dictionary learning for a more flexible residual representation. Model-wide storage budgets are assigned using spectral allocation adapted from COMPOT. Experiments cover three MoE architectures at 20–60% compression and report improved perplexity and zero-shot accuracy over a controlled D²-MoE baseline. Ablations examine base updates, residual representations, allocation strategies, and update schedules.

Critical strengths
  1. The coupled formulation addresses a specific limitation of constructing the shared base before residual compression. The SVD-based experiment provides evidence that updating the base improves compression independently of changing the residual representation.

  2. The experiments cover three MoE architectures and multiple compression ratios, with reported gains persisting under strong compression. Separate evaluations of residual geometry, budget allocation, and update schedules help establish the effects of the individual designs.

Critical weaknesses
  1. The comparison with D²-MoE [1] is incomplete. The reproduced baseline disables shared-base pruning, which is part of the original method and affects how storage is distributed between the base and residuals. The authors should evaluate the complete D²-MoE under the same preprocessing, evaluation settings, and full-model storage budget. Appendix B describes several preprocessing and accounting changes, but does not adequately explain the substantial accuracy discrepancy on Phi-3.5-MoE.

  2. The DeepSeekMoE results need to be reconciled. In Table 5, the reported average accuracies do not match the arithmetic means of the individual task scores. Comparison with Table 8 suggests that the task scores and averages may correspond to different k/s settings. The authors should check whether the main results, task-wise scores, and sensitivity analysis consistently report the intended configurations and correct the discrepancies.

  3. The discussion and comparison with closely related methods remain incomplete. The uncited RS-MoE [2] learns a shared down-projection base after compressing the gate and up projections to compensate for reconstruction error. This directly relates to the paper’s motivation for making base construction account for compression, and the authors should clarify the additional contribution of repeated base–residual optimization. The OSDL compressor and spectral allocator are adapted from COMPOT [3]. A comparison with COMPOT applied directly to expert weight matrices under the same storage budget would help assess the benefit of the shared-base formulation.

[1] Delta Decompression for MoE-based LLMs Compression

[2] RS-MoE: Collaborative Compression for Mixture-of-Experts LLMs based on Low-Rank and Sparse Approximation

[3] COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compression

Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer qTkm. Part 1

We sincerely thank the reviewer for the thorough assessment of our work, for recognizing its strengths, and for providing constructive suggestions. We particularly appreciate the concerns regarding the completeness of the D²-MoE comparison, reporting consistency, and the distinction between BOLD-MoE and related compression methods. Following these suggestions, we conducted additional experiments and revisited the relevant baseline implementations. We address each concern below.

For questions shared with other reviewers, we provide the full tables and discussion in the General Response, while retaining the original W numbering here.

W1. Incomplete comparison with D²-MoE without shared-base pruning

We thank the reviewer for raising this important point. Our primary objective was to isolate the contribution of the proposed base–residual optimization from additional compression mechanisms, particularly shared-base pruning. We reproduced the D²-MoE pruning ablation from its Table 7 using the original released implementation, before the corrections documented in Appendix B. The published-versus-reproduced pruning table and the discussion of the Phi-3.5-MoE discrepancy are in General Response, §G1.

Within this initial reproduction, increasing shared-base pruning consistently worsened perplexity across all three datasets, with the unpruned configuration achieving the best results. This motivated our choice to retain an unpruned shared base. The final D²-MoE and BOLD-MoE comparisons in Table 4 use the common corrected pipeline described in Appendix B.

W2. Inconsistent DeepSeekMoE results in Table 5 and mismatched k/s configurations

We thank the reviewer for identifying this reporting inconsistency. The task-wise BOLD-MoE results for DeepSeekMoE-16B-Base in Table 5 correspond to the k/s=2.2 configuration, whereas the reported average accuracies correspond to k/s=3.4. We confirm that the main results in Table 4 use k/s=3.4 and are correct; the two configurations are discussed in Appendix A.6, Table 8.

We provide the full corrected task-wise results in General Response, §G3, also addressing the related reporting question from uf9y. We apologize for the reporting error and will correct Table 5 in the revised manuscript. The main compressed-model results and conclusions remain unchanged.

W3.1. Comparison with RS-MoE and clarification of methodological differences

We thank the reviewer for drawing our attention to RS-MoE. We attempted to reproduce its results using the supplementary implementation but encountered substantial issues with the Down-projection whitening/SVD path and expert-group processing. The reproduction difficulties, reported-result comparisons, and methodological distinctions are discussed in General Response, §G2.5.

Although both methods exploit shared components to mitigate compression errors, RS-MoE uses a one-time ridge-regression correction of the shared Down projection, whereas BOLD-MoE repeatedly updates the shared bases and compressed residuals of Up, Gate, and Down using Equations (5)–(8). Our SVD-only ablation (Figure 2) and optimization-schedule ablation (Table 3) isolate the benefit of this repeated optimization.

In addition to RS-MoE requested here, we evaluated NAEE, MoE-I², TD-MoE, and RFID-MoE following the baseline-comparability and additional-method requests of ick9 and uf9y. Their results and reproduction qualifications are provided in General Response, §§G2.1–G2.4.

AuthorsOpenReview ↗

Response to Reviewer qTkm. Part 2

W3.2. Benefit of joint base–residual optimization over OSDL-only compression

We thank the reviewer for suggesting this additional ablation, which directly helps isolate the contribution of the shared-base formulation.

As discussed in Section 3.3, BOLD-MoE builds on the orthogonal sparse dictionary learning formulation of COMPOT, while introducing it into the coupled shared-base objective of Equation (5).

The original submission already separates several relevant contributions: Table 1 compares SVD and OSDL with a fixed shared base, Figure 2 demonstrates the benefit of alternating optimization with SVD residuals, and Table 3 studies the frequency of shared-base updates under a fixed OSDL iteration budget.

To further isolate the contributions, we conducted additional experiments comparing three configurations:

  1. OSDL on Weights: COMPOT applied directly to the original expert parameters, without a shared base.
  2. OSDL on Residuals: COMPOT applied to expert-specific residuals after constructing a fixed shared base.
  3. BOLD-MoE: Alternating optimization of the shared base and compressed residuals.

All experiments were performed on Mixtral-8×7B using 50 optimization iterations, k/s=3.4k/s=3.4, and the indicated target compression ratios.

Perplexity comparison

CRMethodWikiText-2 ↓PTB ↓C4 ↓
40%OSDL on Weights5.3621.3011.20
OSDL on Residuals (Fixed Base)4.9318.248.65
BOLD-MoE4.6917.558.27
60%OSDL on Weights11.1893.4242.81
OSDL on Residuals (Fixed Base)6.2125.0810.87
BOLD-MoE5.8223.6010.23

Zero-shot accuracy comparison

CRMethodOpenB.ARC-eARC-cWinoG.HellaS.PIQAMathQAAvg. ↑
40%OSDL on Weights30.479.048.572.855.377.737.057.2
OSDL on Residuals32.079.649.574.159.880.436.759.0
BOLD-MoE33.880.951.475.761.180.939.060.4
60%OSDL on Weights19.660.229.162.037.564.427.943.0
OSDL on Residuals28.474.441.772.353.376.832.254.2
BOLD-MoE29.276.443.772.754.576.433.455.2

These experiments reveal two complementary sources of improvement.

First, introducing a shared-base representation substantially improves compression compared with applying COMPOT directly to expert parameters. This benefit becomes especially pronounced under aggressive compression: at 60% CR, average zero-shot accuracy increases from 43.0% to 54.2%.

Second, alternating optimization provides an additional improvement beyond fixed-base residual compression. At 40% and 60% CR, it increases average accuracy by another 1.4 and 1.0 percentage points, respectively, while consistently reducing perplexity.

These results complement the ablations already included in Section 4.2 and provide direct empirical evidence that BOLD-MoE is not merely an application of COMPOT to MoE parameters. Rather, its effectiveness arises from combining a shared-base representation with compressor-aware alternating optimization.

We again thank the reviewer for the constructive feedback. We believe that the additional shared-base pruning experiments, corrected reporting, comparison with RS-MoE, and controlled OSDL ablations substantially strengthen the experimental justification and clarify the methodological contributions of BOLD-MoE.

Dear, Reviewer qTkm,

we sincerely appreciate your constructive review and detailed suggestions. We would be grateful for any follow-up feedback on our responses, particularly regarding the D²-MoE comparison, additional RS-MoE experiments, and the ablation isolating the contribution of joint base–residual optimization.

We would be happy to clarify any remaining concerns.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response to Reviewers. Part 4

G2.5. RS-MoE (requested by qTkM)

We investigated the [RS-MoE OpenReview submission](https://openreview.net/forum?id=5SQbPbCU6P) and attempted to reproduce its results using the supplementary code provided with that submission.

However, we encountered substantial issues in the supplied implementation that prevented reliable reproduction. In particular, an isolated GPU test of the Down-projection factorization failed because of a tensor-shape mismatch in the whitening/SVD path. Code inspection also revealed potentially inconsistent processing of the Down projection outside the expert-group loop and repeated subtraction of the shared baseline in the medium-importance branch.

A revised version of RS-MoE was published in Knowledge-Based Systems (2026), on Sept. 5, 2026. However, its accompanying [public repository](https://github.com/ALittlething/RS-MoE) currently provides only a preliminary description and does not yet include the complete implementation. Consequently, we could not independently reproduce its published results using an official codebase.

Perplexity comparison on Mixtral-8×7B

CRMethodWikiText-2 ↓PTB ↓C4 ↓
20%RS-MoE (Reported)4.7016.498.52
BOLD-MoE4.1715.127.53
40%RS-MoE (Reported)5.8318.2312.54
BOLD-MoE4.6917.558.27
60%RS-MoE (Reported)7.7423.4315.36
BOLD-MoE5.8223.6010.23

Zero-shot accuracy comparison on Mixtral-8×7B

CRMethodOpenB.ARC-eWinoG.HellaS.PIQAMathQAAvg. ↑
20%RS-MoE33.081.075.062.080.039.061.67
BOLD-MoE33.882.976.663.682.341.563.45
40%RS-MoE30.078.070.056.078.033.057.50
BOLD-MoE33.880.975.761.180.939.061.90
60%RS-MoE26.071.067.051.071.031.052.83
BOLD-MoE29.276.472.754.576.433.457.10

Zero-shot averages are calculated over the six shared benchmarks, excluding ARC-Challenge, which is not reported by RS-MoE.

RS-MoE results are externally reported reference values, not a controlled reproduction; differences in calibration and evaluation protocols may remain.

G3. DeepSeekMoE Table 5 reporting (raised by qTkM W2 and uf9y W1/Q1)

Regarding the uncompressed baselines, we initially retained the average accuracies reported in D²-MoE for consistency with prior work. Our independent evaluation yielded comparable results, and we will update the corresponding entries in the revised manuscript to ensure consistency between task-wise scores and reported averages.

Regarding the compressed DeepSeekMoE-16B-Base results, we identified a reporting error: the task-wise BOLD-MoE results in Table 5 correspond to the k/s=2.2k/s=2.2 configuration, whereas the reported averages correspond to k/s=3.4k/s=3.4.

We confirm that the main BOLD-MoE results in Table 4 were obtained using k/s=3.4k/s=3.4 and are correct. The differences between these configurations are already analyzed in Appendix A.6, Table 8.

MethodCROpenB.ARC-eARC-cWinoG.HellaS.PIQAMathQAAvg. ↑
D²-MoE20%30.274.641.869.852.475.230.153.4
BOLD-MoE20%29.674.541.069.454.477.730.853.9
D²-MoE40%28.269.836.267.645.072.027.249.4
BOLD-MoE40%28.672.038.969.948.473.827.951.4
D²-MoE60%20.857.428.161.034.764.124.141.5
BOLD-MoE60%21.060.229.162.536.465.624.642.8

Corrected DeepSeekMoE-16B-Base task-wise results. The BOLD-MoE rows correspond to k/s = 3.4, as used in Table 4.

We apologize for the reporting inconsistency and will correct Table 5 and clarify the provenance of the original-model reference averages in the revised manuscript. These corrections do not affect the compressed-model results or the main conclusions of our paper.

AuthorsOpenReview ↗

General Response to Reviewers. Part 3

G2.3. TD-MoE (requested by ick9)

We investigated TD-MoE using its [official implementation](https://github.com/ust-xu/TD-MoE). An important distinction is that its released Mixtral configuration targeting 20% compression applies tensor decomposition to only 9 of the 32 MoE layers. With approximately 20% parameter reduction per selected layer, this corresponds to only about 5.6% reduction in total expert parameters before accounting for additional stored tensors.

To investigate the effect of extending compression beyond the selected layers, we additionally tested TD-MoE with all 32 MoE layers compressed using the same per-layer setting.

MethodCompressed LayersCompression settingWikiText-2 PPL ↓
Original Mixtral-8×7B (Table 4)0 / 32—3.98
TD-MoE (Official Configuration)9 / 3220% per selected layer4.6727
TD-MoE (All Layers, Diagnostic)32 / 3220% per selected layer598.4617
BOLD-MoE (Table 4)32 / 3220% full-model4.17

WikiText-2 perplexity on Mixtral-8×7B for TD-MOE reproduction

However, inspection of the released implementation revealed potential calibration issues. Specifically, `src/data_collection.py` propagates activations only through selected layers, potentially bypassing preceding transformer layers, while `src/run_covariances.py` reuses this procedure for uncached layers. These issues may produce incorrect calibration statistics, making the all-layer result inconclusive as an assessment of TD-MoE under broader compression. Resolving these external implementation issues is beyond the scope of our work.

Reference comparison with published TD-MoE results. Although the identified implementation issues prevent us from establishing a reliable matched-protocol reproduction of TD-MoE, we additionally provide a reference comparison using the results reported in Table 1 of the original TD-MoE paper (ICLR 2026).

Target CRMethodWikiText-2 ↓PTB ↓C4 ↓Avg. Zero-shot (%) ↑
20%TD-MoE (Output Whitening, Reported)4.4917.207.7362.00
BOLD-MoE4.1715.127.5362.10
40%TD-MoE (Output Whitening, Reported)5.7924.609.2157.00
BOLD-MoE4.6917.558.2760.40
60%TD-MoE (Input Whitening, Reported)13.6466.0919.6345.00
BOLD-MoE5.8223.6010.2355.20

TD-MoE values are taken from Table 1 of the original publication, selecting the best-performing whitening configuration at each compression level. BOLD-MoE results are taken from Table 4 of our submission. Compression ratios are nominal targets and are not independently verified as equivalent full-model storage reductions.

Practical inference efficiency. We additionally evaluated the released nine-layer TD-MoE implementation against the original Mixtral model under the same execution setup. The TD-MoE configuration reduced prefill throughput by 24.8% and decoding throughput by 16.2%, while increasing peak GPU memory by approximately 1.2%.

Our inspection further indicates that the implementation retains dense output-dewhitening matrices as buffers, which are not reflected in the reported parameter count and can offset the storage savings of tensor decomposition.

These observations characterize the available implementation rather than establishing an inherent limitation of TD-MoE.

G2.4. RFID-MoE (requested by ick9)

To the best of our knowledge, no official standalone implementation has been publicly released. However, the [ICML 2026 submission](https://proceedings.mlr.press/v306/mi26a.html) includes implementation materials in its supplementary files.

We used the supplementary code provided by the authors to reproduce RFID-MoE on Mixtral-8×7B.

CRMethodWikiText-2 ↓PTB ↓C4 ↓Avg. ↑
20%RFID-MoE (Our Reproduction)6.3831.849.2559.95
BOLD-MoE4.1715.127.5362.10
40%RFID-MoE (Our Reproduction)51.50307.3244.6945.97
BOLD-MoE4.6917.558.2760.40
60%RFID-MoE (Our Reproduction)16,216.8128,511.2718,194.8129.61
BOLD-MoE5.8223.6010.2355.20

Comparison with our RFID-MoE reproduction under the evaluation protocol of Section 4.1. BOLD-MoE values correspond to Table 4 of the submitted manuscript.

We will extend the related-work discussion in Section 2 accordingly.

AuthorsOpenReview ↗

General Response to Reviewers. Part 2

G2. Baseline selection, comparison protocols, and additional methods

As documented in Section 4.1 and Appendix B, our implementation is built directly on the D²-MoE codebase, with BOLD-MoE implemented on top of its compression and evaluation infrastructure. We also explicitly documented the preprocessing corrections and standardized compression accounting used for the controlled D²-MoE comparison. Consequently, we initially expected the additional baseline results reported in D²-MoE, including NAEE and MoE-I², to follow compatible experimental protocols and compression-ratio definitions.

All additional experiments reported below follow the calibration and evaluation protocol described in Section 4.1 of our submission. Specifically, we use 256 calibration sequences of 2,048 tokens sampled from the WikiText-103 training split, consistent with the configuration provided for SVD-scale collection in the released D²-MoE implementation. We evaluate perplexity on WikiText-2, PTB, and C4, together with average zero-shot accuracy across the same seven downstream benchmarks using the same benchmark definitions, evaluation harness, and evaluation settings.

G2.1. NAEE (requested by ick9 and uf9y)

NAEE. Following the suggestions of ick9 and uf9y, we independently reproduced NAEE using its [official implementation](https://github.com/Lucky-Lance/Expert_Sparsity) and evaluated the resulting models using our evaluation pipeline.

Unlike BOLD-MoE, NAEE removes entire experts, resulting in discrete compression levels determined by the number of retained experts. Consequently, the achievable full-model compression ratios do not necessarily coincide with the target budgets reported in D²-MoE.

To approximate the target compression ratios, we uniformly removed two, three, and five of the eight experts per MoE layer, obtaining 24.1%, 36.2%, and 60.4% full-model compression, respectively.

Target CRMethodRetained ExpertsCR (%)WikiText-2 ↓PTB ↓C4 ↓Avg. ↑
20%NAEE (Our Reproduction)6/824.15.0215.428.2859.98
NAEE (Reported in D²-MoE)Not verified20.04.7716.098.8958.00
BOLD-MoE8/820.04.1715.127.5362.10
40%NAEE (Our Reproduction)5/836.25.6816.489.0558.49
NAEE (Reported in D²-MoE)Not verified40.06.4422.1513.8648.00
BOLD-MoE8/840.04.6917.558.2760.40
60%NAEE (Our Reproduction)3/860.48.0232.3712.0852.11
NAEE (Reported in D²-MoE)Not verified60.011.4347.2831.1636.00
BOLD-MoE8/860.05.8223.6010.2355.20

Comparison on Mixtral-8×7B. Reproduced NAEE compression ratios account for removed expert parameters relative to the full model. Published NAEE compression ratios (marked ) are nominal values whose exact configurations could not be independently verified.

G2.2. MoE-I² (requested by ick9)

MoE-I². We also investigated independently reproducing MoE-I². Unfortunately, the repository referenced by the original publication is empty, and we could not identify supplementary implementation materials accompanying the submission.

Consequently, we were unable to reproduce MoE-I² under our calibration and evaluation protocol. Its results in Table 4 are therefore externally reported reference values adopted from D²-MoE rather than independently verified controlled comparisons. We will explicitly clarify this distinction in the revised manuscript.

For transparency, our full-model CR definition is already provided in Section 4.1, and the corresponding OSDL storage accounting, including sparse coefficient masks, is specified in Equation (21), Section 3.4. We will make the provenance and compression-ratio accounting of external baselines more explicit.

AuthorsOpenReview ↗

General Response to Reviewers. Part 1

We sincerely thank ick9, qTkM, and uf9y for their careful assessments and constructive suggestions. We consolidate overlapping responses here and refer to them in our individual replies.

G1. D²-MoE shared-base pruning (raised by qTkM W1 and uf9y W3/Q3)

Our primary objective was to isolate the contribution of the proposed base–residual optimization from additional compression mechanisms, particularly shared-base pruning. As described in Section 4.1 and Appendix B, BOLD-MoE was implemented directly on top of the released D²-MoE codebase, retaining its shared-base construction and evaluation infrastructure.

To verify whether excluding base pruning was justified, we reproduced the shared-base pruning ablation from Table 7 of D²-MoE, using its original released implementation before introducing the preprocessing and compression-accounting corrections documented in Appendix B.

Base PruningResidual CRWikiText-2 Paper ↓WikiText-2 Ours ↓PTB Paper ↓PTB Ours ↓C4 Paper ↓C4 Ours ↓
0%0.5391—5.47—22.69—9.69
10%0.52665.635.6327.4022.8612.659.98
20%0.51415.315.8320.4323.9710.7510.38
30%0.50165.286.1920.5426.4210.1011.14
40%0.48905.746.7420.8328.1011.4812.35
50%0.47676.227.7122.5431.1910.7214.55
60%0.46425.289.4020.5436.4510.1018.39

D²-MoE shared-base pruning ablation on Mixtral-8×7B targeting 40% full-model compression. "Paper" denotes results reported in Table 7 of D²-MoE; "Ours" denotes our initial reproduction using the original released codebase, prior to the corrections described in Appendix B. CR denotes compression ratio.

Regarding discrepancies with the original D²-MoE results, we acknowledge that these extend beyond the pruning ablation. As documented in Appendix B, we invested substantial effort in reproducing the original method and correcting several issues in its released implementation, including:

  • Causal-language-model preprocessing for Fisher estimation.

  • Handling of padded tokens when estimating expert routing frequencies.

  • Residual-retention conventions and full-model compression accounting.

Despite these corrections, some discrepancies remain. In particular, for Phi-3.5-MoE at 40% CR, Table 9 reports 53.3% average zero-shot accuracy for our reproduced D²-MoE, compared with 60.0% in the original publication.

These discrepancies may reflect differences between the published experimental protocols and the released code; identifying their precise causes requires additional information about the original experiments.

We consistently follow the corrected released implementation rather than combine results obtained under potentially different protocols.

As detailed in Appendix B, D²-MoE and BOLD-MoE use identical corrected calibration statistics, shared-base initialization, compression accounting, and evaluation settings, providing a controlled comparison of the compression procedures.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.