acceptodds
Under review as a conference paper at ICLR 2027

AdaUpMoE: Adaptive Fine-grained Compression of Upcycled MoE Models

Abstract

Mixture-of-Experts (MoE) has become a promising architecture for scaling large models with sparse activation. Recently, upcycling dense models into MoE models has emerged as an effective approach for constructing powerful sparse models, motivating the need for effective compression techniques for upcycled MoE models. In this work, we present the first systematic study of upcycled MoE compression and reveal two key observations: (1) existing methods typically apply uniform compression strategies across the entire model, overlooking the heterogeneous compression requirements among different MoE components; (2) compression effectiveness is highly sensitive to fine-grained compression policies, including sparsity ratios and rescaling factors. Based on these findings, we propose AdaUpMoE, an adaptive fine-grained compression framework for upcycled MoE models. Specifically, AdaUpMoE formulates compression as a component-wise policy learning problem, leveraging a small amount of unlabeled data and a two-stage distillation-based learning paradigm to automatically learn compression policies for different layers, experts, and model modules. By adaptively allocating compression budgets according to component characteristics, AdaUpMoE effectively exploits the heterogeneous redundancy of upcycled MoE models while preserving model capability. Extensive experiments on 5 upcycled MoE approaches, 8 models with diverse architectures and scales, and 30 datasets covering various tasks and modalities demonstrate that AdaUpMoE consistently outperforms existing compression approaches, achieving particularly significant improvements under extreme (up to 100× on expert-specific deltas) compression regimes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture of Experts, Model Compression, Sparse Upcycling, Adaptive Sparsification, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_3rq7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AdaUpMoE, a method for adaptively assigning sparsity ratios and rescaling factors to different components of upcycled Mixture-of-Experts models. The motivation comes from an analysis and previous works showing that different modules in MoE architecture have different sensitivity to compression ratios. The proposed method then learns compression policies through two-stage distillation using calibration datasets. Experiments then cover several MoE architectures, model variants, and tasks, showing promising performance, especially at extreme compression ratios compared with existing baselines.

Critical strengths
  1. The problem formulation and proposed method are interesting. The idea of applying adaptive sparsity ratios to different modules is natural and has been successfully applied in other areas of pruning and efficient training. Therefore, extending this idea to upcycled MoE compression is well motivated and makes the proposed approach easy to follow.

  2. The experimental results are promising The method shows strong improvements over the baselines across different models and tasks, , especially at extreme compression ratios. The additional experiments and analyses in the appendix are also quite comprehensive, covering many different settings.

Critical weaknesses
  1. The method introduces many loss terms and hyperparameters, which may make it difficult to tune and apply. The training objective combines several distillation and regularization terms, each with its own tunable coefficient. These coefficients also vary heavily across models and compression ratios. Although the paper provides some ablation studies and a short appendix for this, I find the hyperparameter analysis insufficient to show how sensitive the method is to these choices. This may make the method very expensive when training with other models.

  2. The ablation comparisons do not fully separate the benefits of adaptive policies from the additional calibration and distillation. Furthermore, the limitations of proposed method are not fully discussed, especially those related to using the straight-through estimator (STE) for sparsification.

Questions for authors
  1. Could the authors provide more evidence about the convergence speed and detailed analysis of the proposed method, such as training loss curves and performance across optimization steps? Since the objective contains several loss terms, this would help clarify the optimization behavior.

  2. Could the authors provide a more detailed sensitivity analysis of the coefficients of the proposed loss terms? In particular, how sensitive is the performance to these coefficients across different models and compression ratios?

  3. Could the authors discuss the limitations and possible failure modes of the proposed method? Also, why do the authors choose STE for learning sparsity ratios instead of other differentiable sparsification methods? Are there any stability or optimization issues with this choice?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 3rq7 (1/5)

[W1] Many loss terms and hyperparameters

Thank you for raising this concern. In practice, hyperparameter selection is lightweight across three levels: (i) for a given model and ratio, one set of coefficients serves all benchmarks; (ii) across ratios of the same model, we mainly adjust the sparsity and rescaling weights; and (iii) across models, all runs follow one balancing rule (Section 2.4 of the supplementary) that approximately balances the initial weighted loss shares. Additionally, as detailed in our reply to [Q2] (Tables R11 and R12), across 15 single-coefficient perturbations spanning 0.33× to 7× the defaults, Avg changes by at most 0.42 percent, and the coefficients tuned on one Med-MoE backbone can transfer directly to the other without retuning.

Table R11. Sensitivity to the loss coefficients on MoE-LLaVA-StableLM at 100×, where each row changes the coefficient of one loss term and keeps the others at their defaults. The defaults of the paper are λt=7\lambda_{\mathrm{t}}{=}7, λct=3\lambda_{\mathrm{ct}}{=}3, λmod=10−3\lambda_{\mathrm{mod}}{=}10^{-3}, λsp=1\lambda_{\mathrm{sp}}{=}1 and λrs=5×10−5\lambda_{\mathrm{rs}}{=}5{\times}10^{-5}. ΔAvg is relative to the AdaUpMoE default row.

TermCoefficientSQA-IVQA-TGQAPOPEMM-VetMMBenchVQAv2AvgΔAvg
DeRS-SP–58.8847.8459.0686.8228.2052.7375.0958.37-1.37
AdaUpMoEDefault63.2149.8259.2187.1927.6055.2275.9659.74–
KL original λt\lambda_{\mathrm{t}}362.3749.8459.1087.2428.5955.4375.8459.77+0.03
562.4250.2459.1387.3227.7655.3275.8559.72-0.02
962.3749.8358.9286.9929.2455.7975.8959.86+0.12
KL compressed λct\lambda_{\mathrm{ct}}162.0749.6958.9986.8227.5155.8475.9359.55-0.19
562.2249.8659.0486.8228.0655.8475.9459.68-0.06
762.4250.0259.1486.8628.9155.6475.9559.85+0.11
Module λmod\lambda_{\mathrm{mod}}2×10−32{\times}10^{-3}63.1149.8758.4887.3027.9555.1275.7559.65-0.09
4×10−34{\times}10^{-3}62.3249.9858.0287.3329.0755.0175.5359.61-0.13
7×10−37{\times}10^{-3}62.0749.8357.3087.1428.8854.8175.2359.32-0.42
Sparsity λsp\lambda_{\mathrm{sp}}262.9250.0158.9387.2828.7255.2275.8259.84+0.10
462.7749.7058.7487.1728.6155.5875.8159.77+0.03
762.5749.6158.9386.9228.0655.6475.8559.65-0.09
Rescaling λrs\lambda_{\mathrm{rs}}2×10−52{\times}10^{-5}62.6249.6258.3487.1428.2955.5875.6459.61-0.13
4×10−54{\times}10^{-5}62.6749.7158.8887.0728.5055.9075.8159.79+0.05
7×10−57{\times}10^{-5}62.0249.5259.1486.8727.7455.9075.9159.59-0.15
AuthorsOpenReview ↗

Response to Reviewer 3rq7 (2/5)

Table R12. Coefficient transfer on Med-MoE, where Tuned uses the coefficients of the paper for each backbone and Transferred reuses the coefficients tuned for the other backbone without retuning.

BackboneRatioMethodVQA-RAD OpenVQA-RAD ClosedSLAKE OpenSLAKE ClosedPathVQA OpenPathVQA ClosedAvgΔAvg
StableLM10×DeRS-SP47.2582.3583.5984.3833.2990.9870.31–
Tuned51.3982.7283.8484.1333.2590.7471.01+0.70
Transferred51.8681.9983.7184.6233.2290.8071.03+0.72
20×DeRS-SP46.5782.7282.8884.8633.1290.9570.18–
Tuned51.0281.9983.8584.8633.2691.0471.00+0.82
Transferred50.6381.9983.0483.8933.2990.8670.62+0.44
100×DeRS-SP44.6080.8882.7585.1032.7190.6869.45–
Tuned51.9682.3583.2585.3433.4890.8371.20+1.75
Transferred50.9482.3583.1085.1033.3590.7170.92+1.47
Phi10×DeRS-SP55.6183.4683.8286.3034.2291.2772.45–
Tuned56.8383.8283.7586.3034.5391.2472.74+0.29
Transferred57.7683.4683.3086.3034.4991.2772.76+0.31
20×DeRS-SP55.5283.4683.5086.3034.3491.2772.40–
Tuned57.7683.3283.5086.3034.5791.0672.75+0.35
Transferred56.6483.4683.5986.0634.2691.2772.55+0.15
100×DeRS-SP54.0382.3583.0285.8234.1390.8971.71–
Tuned56.7383.8284.0886.0634.5991.3672.77+1.06
Transferred56.4583.8284.0885.8234.4691.2472.65+0.94

[W2] Ablation does not isolate the adaptive policy; STE limitations

Because all network weights remain frozen throughout calibration and distillation, optimization updates only the two policy scalars per projection (its sparsity ratio and rescaling factor), so calibration and distillation cannot improve accuracy by themselves and act solely through the learned policy.

To isolate the two components of this policy along our default schedule, we evaluate the checkpoint at the end of Stage 1, which learns only where to prune while retaining DeRS-SP's analytic 1/k1/k rescaling, so that its only difference from DeRS-SP is the learned sparsity; Stage 2 then learns rescaling. Tables R13 to R15 compare DeRS-SP, Stage 1 only and full AdaUpMoE on MoE-LLaVA, Med-MoE and X\mathcal{X}FT (whereas the θ\theta-only variant in Tables S4 and S5 of the supplementary trains sparsity for all steps).

Stage 1. The learned sparsity alone outperforms DeRS-SP in all nine settings, with gains growing alongside the compression ratio: whereas retaining a tenth of every delta at 10× is tolerable for most components, uniform dropping at 100× starves sensitive ones, so reallocating budget adds 0.6 to 1.5 points on MoE-LLaVA and 1.4 points on Med-MoE. On X\mathcal{X}FT at 100×, where DeRS-SP drops from the uncompressed 60.12 to 53.23, Stage 1 alone recovers Avg to 59.55. These improvements concentrate on knowledge-intensive tasks (SQA, TextVQA, open-ended VQA-RAD and HumanEval).

Stage 2. Learned rescaling adds a further 0.3 to 1.2 points in every setting. While analytic 1/k1/k preserves each delta's expected value, it does not preserve the delta's effect on the output once most entries are dropped, and Stage 2 corrects this residual error by aligning each component with the compressed teacher. On MoE-LLaVA-Qwen, Stage 2 recovers 1.2 of the 1.8-point gap remaining after Stage 1; on X\mathcal{X}FT at 10× and 20×, where Stage 1 changes Avg by at most 0.6, Stage 2 accounts for most of the improvement, adding 1.0 to 1.1 points mainly on HumanEval and HumanEval+.

STE. STE [1] keeps an exact hard 0/1 mask in the forward pass, so training and inference can use the same sparse model. Under the same objective, it achieves the best Avg in 7 of 9 settings against Gumbel-sigmoid and Hard Concrete and stays within 0.21 Avg in the rest (Tables R16–R17); besides, its biased gradient and other limitations are discussed in our reply to [Q3].

AuthorsOpenReview ↗

Response to Reviewer 3rq7 (3/5)

Table R13. Contribution of each stage on MoE-LLaVA at 100×. Stage 1 only uses 30 steps to learn the sparsity ratios and keeps the analytic 1/k1/k rescaling of DeRS-SP. AdaUpMoE adds 70 Stage 2 steps to learn the rescaling.

BackboneMethodSQA-IVQA-TGQAPOPEMM-VetMMBenchVQAv2AvgΔAvg
StableLMDeRS-SP58.8847.8459.0686.8228.2052.7375.0958.37–
Stage 1 only61.3848.5959.3786.7727.7954.4975.5259.13+0.76
AdaUpMoE63.2149.8259.2187.1927.6055.2275.9659.74+1.37
QwenDeRS-SP58.3543.3258.4886.9624.4048.4273.1956.16–
Stage 1 only59.0545.0159.7687.5326.5651.4374.4257.68+1.52
AdaUpMoE61.6347.1860.2287.6327.4053.4074.8958.91+2.75
PhiDeRS-SP66.9350.4360.6987.6431.3062.4476.8162.32–
Stage 1 only68.4251.2761.2287.4532.2962.2977.3462.90+0.58
AdaUpMoE68.2250.9161.2787.2733.7063.3877.4063.16+0.84

Table R14. Contribution of each stage on Med-MoE-StableLM, with stage definitions as in Table R13.

RatioMethodVQA-RAD OpenVQA-RAD ClosedSLAKE OpenSLAKE ClosedPathVQA OpenPathVQA ClosedAvgΔAvg
10×DeRS-SP47.2582.3583.5984.3833.2990.9870.31–
Stage 1 only48.6282.3582.9784.1333.2991.0470.40+0.09
AdaUpMoE51.3982.7283.8484.1333.2590.7471.01+0.70
20×DeRS-SP46.5782.7282.8884.8633.1290.9570.18–
Stage 1 only50.0882.3583.2284.1333.4691.0670.72+0.54
AdaUpMoE51.0281.9983.8584.8633.2691.0471.00+0.82
100×DeRS-SP44.6080.8882.7585.1032.7190.6869.45–
Stage 1 only50.7483.0982.8484.3833.4590.8670.89+1.44
AdaUpMoE51.9682.3583.2585.3433.4890.8371.20+1.75

Table R15. Contribution of each stage on X\mathcal{X}FT, with stage definitions as in Table R13.

RatioMethodHumanEvalHumanEval+MBPPMBPP+AvgΔAvg
10×DeRS-SP63.4059.1062.2052.9059.40–
Stage 1 only63.4059.1563.1652.8859.64+0.24
AdaUpMoE65.2061.0063.4053.1060.68+1.28
20×DeRS-SP61.0057.3062.7053.1058.52–
Stage 1 only62.2058.5462.9152.8859.13+0.61
AdaUpMoE64.6059.8062.9053.6060.22+1.70
100×DeRS-SP57.9053.0055.1046.9053.23–
Stage 1 only66.4660.9859.6551.1359.55+6.32
AdaUpMoE64.0060.4063.9053.6060.48+7.25

Table R16. Gradient estimators for the sparsity masks on Med-MoE under the same objective and coefficients.

BackboneRatioEstimatorVQA-RAD OpenVQA-RAD ClosedSLAKE OpenSLAKE ClosedPathVQA OpenPathVQA ClosedAvg
Phi10×STE56.8383.8283.7586.3034.5391.2472.74
Gumbel-sigmoid57.7683.8283.6986.3034.6791.2772.92
Hard Concrete58.5984.5683.5485.3434.4291.2772.95
20×STE57.7683.3283.5086.3034.5791.0672.75
Gumbel-sigmoid55.8083.0983.9886.3034.7091.3972.54
Hard Concrete59.7484.1983.1485.1033.2991.2172.78
100×STE56.7383.8284.0886.0634.5991.3672.77
Gumbel-sigmoid56.1783.4684.1285.8234.1791.2472.50
Hard Concrete57.8384.5682.8484.6233.2891.0672.36
StableLM10×STE51.3982.7283.8484.1333.2590.7471.01
Gumbel-sigmoid51.3081.9983.6984.3833.2390.7770.89
Hard Concrete52.5781.6282.0082.9332.5090.3370.32
20×STE51.0281.9983.8584.8633.2691.0471.00
Gumbel-sigmoid50.9482.3583.2383.4133.3690.8670.69
Hard Concrete52.9481.6282.3983.8932.5690.3970.63
100×STE51.9682.3583.2585.3433.4890.8371.20
Gumbel-sigmoid49.8081.9983.1084.8633.5990.7770.68
Hard Concrete51.7880.8882.0283.4131.7390.2470.01
Reviewer_5wQ6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks how to compress "upcycled" MoE models — dense checkpoints converted into MoE via expert deltas. AdaUpMoE learns a sparsity ratio and an output rescaling factor for every (layer, expert, module) triple by distilling from the model itself on ~100 unlabeled calibration inputs: first learn sigmoid-gated masks under a global ratio budget against the original teacher, then freeze the masks and learn rescalings against the compressed teacher. They run this on 8 upcycled variants from 5 families, 30 benchmarks and 5–100× expert-delta ratios against DeRS-QT/DeRS-SP, backed by granularity/two-stage/objective ablations, leakage and calibration-size controls, seed and hardware sensitivity, and ~20 pages of compressibility probes.

Critical strengths
  • Sweeping sparsity and rescaling per layer, per expert and per module gives the first systematic map of where upcycled-MoE capacity actually sits, and it earns the fine-grained policy.
  • Leakage checks show held-out calibration moves results by less than a point, the calibration-size curve is flat from 50 to 1000 inputs, seed and hardware sensitivity are reported, and the method ordering is consistent across all eight variants
Critical weaknesses
  • Quality is normalized to the uncompressed upcycled MoE, hiding the dense→upcycled hop; and for a storage-only method the decision-relevant rival — the same dense checkpoint quantized or sparsified to matched bytes — is absent, even though at nominal 100× the model still stores ~19% of upcycled bytes (5.23× whole-model), i.e. dense-weight scale.
  • The comparator set is only two variants of one prior method (DeRS): no pruning, merging, other quantizer.
Questions for authors
  1. please report per-variant dense-checkpoint scores and a bytes-matched dense-compression baseline (INT4/INT8 PTQ or structured sparsity on the same dense checkpoint), and decompose the two-hop quality loss; does the upcycle→compress pipeline still dominate at equal storage?
  2. if I built a dumb greedy allocator from your own probe sensitivities (or allocated by router frequency), how much of the gap to the learned policy would it close at matched ratios?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 5wQ6 (1/5)

[W1] Missing dense and bytes-matched dense baselines

We agree that the dense$\to$Upcycled hop should be visible. Our reply to [Q1] now reports dense baselines for mainly seven models: Upcycling gains 0.82 to 2.7 Avg points over the data-matched dense model, and AdaUpMoE at 100×, which stores slightly more than the dense checkpoint, outperforms the uncompressed data-matched dense model by 0.98 to 3.20 Avg points on six models, and the pre-Upcycling dense checkpoint by 21 points on Uni-MoE.

Since AdaUpMoE preserves the shared dense base, its parameter count is lower-bounded by that of the dense checkpoint. Building on this base, Upcycling followed by compression converts a modest parameter overhead into a substantial performance gain. Moreover, because the number of experts is necessarily an integer, directly Upcycled MoEs are restricted to a discrete set of model sizes, which in general do not coincide with the compressed size of AdaUpMoE. By contrast, the sparsity ratio of AdaUpMoE can be set continuously, allowing the compressed model to be tailored precisely to a given memory budget.

[W2] Only DeRS variants as comparators

To broaden the comparator set beyond DeRS, we evaluate REAP [1] (whole-expert pruning) and MCMoE [2] (mixed-precision expert quantization) on four of the five model families (Tables R19 to R22). Neither matches AdaUpMoE at high compression ratios: REAP cannot exceed 4× on four-expert models and collapses on X\mathcal{X}FT at 4×, while MCMoE, theoretically capped at 9.14× on four-expert models and 10.3× on MultiPL-MoE, collapses on the latter. AdaUpMoE at 10× outperforms every evaluated REAP and MCMoE setting on MoE-LLaVA, Med-MoE and X\mathcal{X}FT and is comparable to REAP on MultiPL-MoE (within 0.63 pass@1), with each ratio following its method's own definition.

Table R19. Comparison with REAP and MCMoE on MoE-LLaVA. The ratio is defined per method: REAP reduces the expert count, with 2× keeping 2 of the 4 experts and 4× keeping 1, MCMoE uses 16 bits over the average bit width.

BackboneMethodRatioSQA-IVQA-TGQAPOPEMM-VetMMBenchVQAv2Avg
StableLMVanilla–59.3750.1460.4986.7527.4058.2976.6359.87
REAP2×61.9750.4260.2287.0728.4757.1476.3660.24
REAP4×62.2748.5159.5586.6828.3255.4375.4259.45
MCMoE8×52.9019.6845.2584.5017.2642.7555.3545.38
MCMoE9.14×39.869.7622.5577.3313.3420.6837.4531.57
AdaUpMoE10×62.7250.1460.1486.9231.5058.1376.4360.85
AdaUpMoE100×63.2149.8259.2187.1927.6055.2275.9659.74
QwenVanilla–63.1647.7461.5087.8125.5054.7076.1859.51
REAP2×62.9645.2860.3987.5125.3454.7074.5158.67
REAP4×61.5344.2059.6787.5226.7154.4973.7458.27
MCMoE8×57.9143.6960.0286.8624.6447.4374.3056.41
MCMoE9.14×54.3938.2455.9986.2523.2739.0671.1452.62
AdaUpMoE10×62.5747.5161.1787.6727.4054.3975.9659.52
AdaUpMoE100×61.6347.1860.2287.6327.4053.4074.8958.91
PhiVanilla–68.9651.1661.4287.2332.1063.7477.5863.17
REAP2×69.2650.9061.2587.0033.2463.8477.5763.29
REAP4×68.4249.5060.4185.7130.8562.1877.0862.02
MCMoE8×62.7748.4560.0386.5131.6455.5376.0460.14
MCMoE9.14×58.5045.3558.1385.8729.5451.9573.9857.62
AdaUpMoE10×69.0151.4861.3787.2334.2063.4377.5863.47
AdaUpMoE100×68.2250.9161.2787.2733.7063.3877.4063.16
AuthorsOpenReview ↗

Response to Reviewer 5wQ6 (2/5)

Table R20. Comparison with REAP and MCMoE on Med-MoE.

BackboneMethodRatioVQA-RAD OpenVQA-RAD ClosedSLAKE OpenSLAKE ClosedPathVQA OpenPathVQA ClosedAvg
StableLMVanilla–47.6481.6283.5485.1033.3890.8070.35
REAP2×45.8078.6882.0883.6531.8590.3668.74
REAP4×36.3468.7574.4480.7727.6484.3162.04
MCMoE8×43.9482.3578.7383.4131.5290.8968.48
MCMoE9.14×36.0780.5172.0183.4130.0090.5965.43
AdaUpMoE10×51.3982.7283.8484.1333.2590.7471.01
AdaUpMoE100×51.9682.3583.2585.3433.4890.8371.20
PhiVanilla–56.4583.4683.6986.3034.4391.3372.61
REAP2×50.1384.9379.7783.8932.0390.3370.18
REAP4×41.8882.3571.6881.7330.9390.1866.46
MCMoE8×51.8083.4685.0184.3833.7090.0071.39
MCMoE9.14×51.5982.3584.7384.1333.4689.7771.01
AdaUpMoE10×56.8383.8283.7586.3034.5391.2472.74
AdaUpMoE100×56.7383.8284.0886.0634.5991.3672.77

Table R21. Comparison with REAP and MCMoE on X\mathcal{X}FT.

MethodRatioHumanEvalHumanEval+MBPPMBPP+Avg
Vanilla–65.2061.0062.2052.1060.12
REAP2×55.4951.8354.6446.8752.21
REAP4×14.6313.4122.8118.0517.23
MCMoE8×60.3757.3262.1651.3857.80
MCMoE9.14×59.7655.4959.1549.3755.94
AdaUpMoE10×65.2061.0063.4053.1060.68
AdaUpMoE100×64.0060.4063.9053.6060.48

Table R22. Comparison with REAP and MCMoE on MultiPL-MoE, with ratio definitions as in Table R19 except that REAP 1.33× keeps 9 of the 12 experts and 6× keeps 2. Avg is the six-language mean.

MethodRatioHumanEval PyHumanEval JavaHumanEval C++HumanEval RustHumanEval GoHumanEval RubyHumanEval AvgMBPP PyMBPP JavaMBPP C++MBPP RustMBPP GoMBPP RubyMBPP Avg
Vanilla–16.5010.769.328.336.496.839.7125.4018.3918.8912.9915.7817.8818.22
REAP1.33×16.509.4911.807.057.146.219.7022.6017.6216.6212.1514.9714.8616.47
REAP6×16.5012.038.703.856.496.218.9618.8018.6517.389.8916.8415.6216.20
MCMoE8×0.600.000.620.640.000.000.310.000.000.250.281.340.000.31
MCMoE10.3×0.000.000.000.000.000.000.000.000.000.000.000.000.000.00
AdaUpMoE10×15.9010.769.945.775.196.839.0726.2013.9917.9312.4315.7815.8717.03
AdaUpMoE100×17.708.8610.564.494.554.358.4223.5016.0617.888.7611.234.7913.70

[Q1] Dense scores, bytes-matched dense compression, two-hop loss

We now report dense baselines for mainly seven models in Table R23 (MoE-LLaVA) and Tables R24–R26 (Med-MoE, Uni-MoE and X\mathcal{X}FT). Because AdaUpMoE at 100× stores the full dense model plus compressed expert deltas (slightly above the dense checkpoint alone), its natural comparator is the uncompressed dense model. AdaUpMoE outperforms the data-matched dense model by 0.98 to 3.20 Avg points on six models (MoE-LLaVA, Med-MoE and X\mathcal{X}FT) and the pre-Upcycling dense checkpoint by 21 points on Uni-MoE.

Since AdaUpMoE preserves the shared dense base, its parameter count is lower-bounded by that of the dense checkpoint. Building on this base, Upcycling followed by compression converts a modest parameter overhead into a substantial performance gain. Moreover, because the number of experts is necessarily an integer, directly Upcycled MoEs are restricted to a discrete set of model sizes, which in general do not coincide with the compressed size of AdaUpMoE. By contrast, the sparsity ratio of AdaUpMoE can be set continuously, allowing the compressed model to be tailored precisely to a given memory budget.

AuthorsOpenReview ↗

Response to Reviewer 5wQ6 (3/5)

MoE-LLaVA. The dense$\to$Upcycled hop gains 1.99 to 2.71 Avg points over Dense-Stage3 (while the larger gap to Dense-Stage2 stems mainly from Stage-3 instruction data), and compressing to AdaUpMoE at 100× costs at most 0.60 points, yielding a net gain of 1.86 to 2.15 points over uncompressed Dense-Stage3 across all three backbones. Tighter budgets instead degrade the dense checkpoint: whereas 8-bit RTN is lossless, 4-bit loses 1.67 to 6.85 Avg points and 2-bit collapses.

Med-MoE. Med-MoE exhibits the same two-hop pattern: Upcycling gains 0.82 to 2.35 Avg points over Dense-Stage3 (with the much larger gap to Dense-Stage2 driven by MoE-stage training data), and AdaUpMoE at 100× surpasses Dense-Stage3 by 0.98 to 3.20 points. MoE-2E-Top1 exceeds Dense-Stage3 by 1.76 points on StableLM while trailing by 0.61 on Phi; for dense RTN, 8-bit is nearly lossless, 4-bit loses ∼0.6\sim0.6 points, and 2-bit loses 5.0 to 17.4 points.

Uni-MoE. On Uni-MoE, the pre-Upcycling dense checkpoint trails AdaUpMoE at 100× by 21 Avg points (primarily on speech benchmarks) because, like Dense-Stage2, it has not been trained on LLaVA-665K or speech RACE; 8-bit RTN is lossless, 4-bit loses 2.25 points, and 2-bit collapses; in the revised version, we will add a data-matched dense model fine-tuned on the same data alongside MoE-4E-Top2 and MoE-2E-Top1 variants.

X\mathcal{X}FT. A consistent trend holds on X\mathcal{X}FT: relative to Dense-SFT (the dense checkpoint fine-tuned on the same instruction data), Upcycling gains 2.54 Avg points and 100× compression incurs no loss, so AdaUpMoE at 100× outperforms Dense-SFT by 2.90 points; for RTN on Dense-Base, 8-bit is lossless while 4-bit and 2-bit collapse to zero.

Table R23. Dense baselines and bytes-matched dense compression on MoE-LLaVA, where MoE-4E-Top2 is the uncompressed Upcycled model of the paper, MoE-2E-Top1 is Upcycled with 2 experts and top-1 routing, Dense-Stage2 is the checkpoint they were Upcycled from, and Dense-Stage3 is Dense-Stage2 trained on the Stage-3 data without Upcycling.

BackboneConfigSQA-IVQA-TGQAPOPEMM-VetMMBenchVQAv2Avg
StableLMMoE-4E-Top259.3750.1460.4986.7527.4058.2976.6359.87
MoE-2E-Top160.8849.2360.1485.9428.7359.4376.1260.07
Dense-Stage358.8845.6359.1186.1128.3453.0974.0257.88
Dense-Stage254.6427.0616.7455.7826.4134.6023.8934.16
8-bit54.8327.4317.0355.8026.9735.1224.5034.53
4-bit49.7320.1014.5566.4223.3924.2618.5130.99
2-bit0.050.000.0050.560.250.000.007.27
QwenMoE-4E-Top263.1647.7461.5087.8125.5054.7076.1859.51
MoE-2E-Top162.4247.1360.0487.0627.1155.2275.8459.26
Dense-Stage360.7345.1057.6988.0722.7250.1373.1456.80
Dense-Stage258.8518.1414.2271.5625.9140.7321.0535.78
8-bit58.6018.2614.3770.6326.7041.0421.2035.83
4-bit54.3915.3614.0082.8522.6029.8719.7334.11
2-bit0.050.000.0050.460.920.000.007.35
PhiMoE-4E-Top268.9651.1661.4287.2332.1063.7477.5863.17
MoE-2E-Top167.1350.3459.9086.6630.9164.3177.1662.34
Dense-Stage366.9848.4059.1185.0330.8660.8375.8461.01
Dense-Stage269.0131.7332.7580.6228.4652.8842.2948.25
8-bit68.5231.8832.6380.3630.1852.7841.9948.33
4-bit65.9923.9019.6480.4525.6950.1324.0341.40
2-bit0.000.030.0050.030.000.000.007.15

Comments on the paper

12 comments
AuthorsOpenReview ↗

Response to Reviewer 1iF9 (1/8)

[W1] Narrow baseline set

Thank you for suggesting a broader comparison. We now add whole-expert pruning with REAP [1], mixed-precision expert quantization with MCMoE [2], and two additional allocation rules, delta norm and OWL.

REAP. Whereas REAP prunes whole experts, AdaUpMoE retains every expert and sparsifies only inside each delta, and the results (Tables R1 to R4, with each ratio following its method's own definition) consistently favor the latter. On four-expert models, REAP is capped at 4×, where only a single expert remains. AdaUpMoE at 10× outperforms REAP at 2× on MoE-LLaVA, Med-MoE and X\mathcal{X}FT, and REAP at 6× on MultiPL-MoE; at 100×, AdaUpMoE outperforms REAP at 4× on every four-expert model. On X\mathcal{X}FT, REAP already loses 7.91 Avg points at 2× and collapses at 4×. This confirms our core finding that the redundancy of Upcycled MoEs lies within the deltas rather than across experts, so pruning whole experts discards useful parameters alongside redundant ones.

MCMoE. MCMoE assigns 1, 2 or 3 bits per expert while requiring at least one 3-bit and one 2-bit expert in every layer; with four experts, the lowest allocation is 3, 2, 1 and 1 bits (1.75 bits on average, or 16/1.75=9.14×16/1.75 = 9.14\times), making 9.14× its theoretical upper limit on our four-expert backbones. Even at a milder 2-bit average (8×), MCMoE loses 1.2 to 14.5 Avg points on MoE-LLaVA, Med-MoE and X\mathcal{X}FT, and collapses to nearly zero on MultiPL-MoE. By contrast, AdaUpMoE at 100× stays within 0.60 Avg points of Vanilla on MoE-LLaVA and exceeds it on Med-MoE and X\mathcal{X}FT (Tables R1 to R3), and on MultiPL-MoE, where MCMoE drops to 0.31% already at 8×, it achieves 8.42% and 13.70% average pass@1 on HumanEval and MBPP, respectively (Table R4). Because quantization must spend bits on every weight, whereas AdaUpMoE drops redundant delta entries and retains the rest at full precision, extreme delta sparsification scales to much higher ratios.

Additional allocation baselines. We add two additional layer-wise rules on X\mathcal{X}FT that match the target average keep ratio and reuse DeRS-SP's random masks and 1/k1/k rescaling, differing only in how per-layer keep ratios are assigned: (1) delta norm, proportional to each layer's delta norm; (2) OWL [3], giving layers whose activation-weighted deltas contain more outliers larger budgets (λ=0.25 and 0.5). Table R5 shows that AdaUpMoE exceeds the better of the two rules by 0.81, 0.31 and 5.15 Avg points at 10×, 20× and 100×, respectively, as fixed heuristic scores coupled with analytic rescaling do not jointly optimize the compressed network. On both Med-MoE backbones (Table R6), AdaUpMoE again achieves the highest Avg in every setting while OWL falls below DeRS-SP. In addition, MCMoE in Tables R1 to R4 allocates bit widths per expert and thus covers HAWQ-style [4] mixed-precision allocation.

AuthorsOpenReview ↗

Response to Reviewer 1iF9 (2/8)

Table R1. Comparison with REAP and MCMoE on MoE-LLaVA. The ratio is defined per method: REAP reduces the expert count, with 2× keeping 2 of the 4 experts and 4× keeping 1, MCMoE uses 16 bits over the average bit width, and AdaUpMoE uses the nominal delta ratio of the paper. MM-Vet value is taken from the mean of five judge runs.

BackboneMethodRatioSQA-IVQA-TGQAPOPEMM-VetMMBenchVQAv2Avg
StableLMVanilla–59.3750.1460.4986.7527.4058.2976.6359.87
REAP2×61.9750.4260.2287.0728.4757.1476.3660.24
REAP4×62.2748.5159.5586.6828.3255.4375.4259.45
MCMoE8×52.9019.6845.2584.5017.2642.7555.3545.38
MCMoE9.14×39.869.7622.5577.3313.3420.6837.4531.57
AdaUpMoE10×62.7250.1460.1486.9231.5058.1376.4360.85
AdaUpMoE100×63.2149.8259.2187.1927.6055.2275.9659.74
QwenVanilla–63.1647.7461.5087.8125.5054.7076.1859.51
REAP2×62.9645.2860.3987.5125.3454.7074.5158.67
REAP4×61.5344.2059.6787.5226.7154.4973.7458.27
MCMoE8×57.9143.6960.0286.8624.6447.4374.3056.41
MCMoE9.14×54.3938.2455.9986.2523.2739.0671.1452.62
AdaUpMoE10×62.5747.5161.1787.6727.4054.3975.9659.52
AdaUpMoE100×61.6347.1860.2287.6327.4053.4074.8958.91
PhiVanilla–68.9651.1661.4287.2332.1063.7477.5863.17
REAP2×69.2650.9061.2587.0033.2463.8477.5763.29
REAP4×68.4249.5060.4185.7130.8562.1877.0862.02
MCMoE8×62.7748.4560.0386.5131.6455.5376.0460.14
MCMoE9.14×58.5045.3558.1385.8729.5451.9573.9857.62
AdaUpMoE10×69.0151.4861.3787.2334.2063.4377.5863.47
AdaUpMoE100×68.2250.9161.2787.2733.7063.3877.4063.16

Table R2. Comparison with REAP and MCMoE on Med-MoE, with ratio definitions as in Table R1 except that REAP prunes routed experts.

BackboneMethodRatioVQA-RAD OpenVQA-RAD ClosedSLAKE OpenSLAKE ClosedPathVQA OpenPathVQA ClosedAvg
StableLMVanilla–47.6481.6283.5485.1033.3890.8070.35
REAP2×45.8078.6882.0883.6531.8590.3668.74
REAP4×36.3468.7574.4480.7727.6484.3162.04
MCMoE8×43.9482.3578.7383.4131.5290.8968.48
MCMoE9.14×36.0780.5172.0183.4130.0090.5965.43
AdaUpMoE10×51.3982.7283.8484.1333.2590.7471.01
AdaUpMoE100×51.9682.3583.2585.3433.4890.8371.20
PhiVanilla–56.4583.4683.6986.3034.4391.3372.61
REAP2×50.1384.9379.7783.8932.0390.3370.18
REAP4×41.8882.3571.6881.7330.9390.1866.46
MCMoE8×51.8083.4685.0184.3833.7090.0071.39
MCMoE9.14×51.5982.3584.7384.1333.4689.7771.01
AdaUpMoE10×56.8383.8283.7586.3034.5391.2472.74
AdaUpMoE100×56.7383.8284.0886.0634.5991.3672.77

Table R3. Comparison with REAP and MCMoE on X\mathcal{X}FT, with ratio definitions as in Table R1 except that REAP prunes routed experts.

MethodRatioHumanEvalHumanEval+MBPPMBPP+Avg
Vanilla–65.2061.0062.2052.1060.12
REAP2×55.4951.8354.6446.8752.21
REAP4×14.6313.4122.8118.0517.23
MCMoE8×60.3757.3262.1651.3857.80
MCMoE9.14×59.7655.4959.1549.3755.94
AdaUpMoE10×65.2061.0063.4053.1060.68
AdaUpMoE100×64.0060.4063.9053.6060.48
AuthorsOpenReview ↗

Response to Reviewer 1iF9 (3/8)

Table R4. Comparison with REAP and MCMoE on MultiPL-MoE, with ratio definitions as in Table R1 except that REAP 1.33× keeps 9 of the 12 experts and 6× keeps 2. Avg is the six-language mean.

MethodRatioHumanEval PyHumanEval JavaHumanEval C++HumanEval RustHumanEval GoHumanEval RubyHumanEval AvgMBPP PyMBPP JavaMBPP C++MBPP RustMBPP GoMBPP RubyMBPP Avg
Vanilla–16.5010.769.328.336.496.839.7125.4018.3918.8912.9915.7817.8818.22
REAP1.33×16.509.4911.807.057.146.219.7022.6017.6216.6212.1514.9714.8616.47
REAP6×16.5012.038.703.856.496.218.9618.8018.6517.389.8916.8415.6216.20
MCMoE8×0.600.000.620.640.000.000.310.000.000.250.281.340.000.31
MCMoE10.3×0.000.000.000.000.000.000.000.000.000.000.000.000.000.00
AdaUpMoE10×15.9010.769.945.775.196.839.0726.2013.9917.9312.4315.7815.8717.03
AdaUpMoE100×17.708.8610.564.494.554.358.4223.5016.0617.888.7611.234.7913.70

Table R5. Non-learned layer-wise allocation on X\mathcal{X}FT. DeRS-SP uses one ratio for all layers, while delta norm and OWL set per-layer ratios at the same average and keep the DeRS-SP masks and rescaling. The best Avg per ratio is in bold.

RatioAllocationHumanEvalHumanEval+MBPPMBPP+AvgΔAvg
10×DeRS-SP63.4059.1062.2052.9059.40–
Delta norm62.2059.1560.1551.6358.28-1.12
OWL λ=0.2565.8561.5960.4051.6359.87+0.47
OWL λ=0.563.4157.9360.6551.1358.28-1.12
AdaUpMoE65.2061.0063.4053.1060.68+1.28
20×DeRS-SP61.0057.3062.7053.1058.52–
Delta norm62.2057.3260.9050.8857.82-0.70
OWL λ=0.2564.6359.7660.4050.1358.73+0.21
OWL λ=0.565.2460.3762.4151.6359.91+1.39
AdaUpMoE64.6059.8062.9053.6060.22+1.70
100×DeRS-SP57.9053.0055.1046.9053.23–
Delta norm54.8851.8354.6446.3751.93-1.30
OWL λ=0.2559.7653.0558.4050.1355.33+2.10
OWL λ=0.555.4950.0059.1549.8753.63+0.40
AdaUpMoE64.0060.4063.9053.6060.48+7.25

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.