- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes AdaUpMoE, a method for adaptively assigning sparsity ratios and rescaling factors to different components of upcycled Mixture-of-Experts models. The motivation comes from an analysis and previous works showing that different modules in MoE architecture have different sensitivity to compression ratios. The proposed method then learns compression policies through two-stage distillation using calibration datasets. Experiments then cover several MoE architectures, model variants, and tasks, showing promising performance, especially at extreme compression ratios compared with existing baselines.
Critical strengths
-
The problem formulation and proposed method are interesting. The idea of applying adaptive sparsity ratios to different modules is natural and has been successfully applied in other areas of pruning and efficient training. Therefore, extending this idea to upcycled MoE compression is well motivated and makes the proposed approach easy to follow.
-
The experimental results are promising The method shows strong improvements over the baselines across different models and tasks, , especially at extreme compression ratios. The additional experiments and analyses in the appendix are also quite comprehensive, covering many different settings.
Critical weaknesses
-
The method introduces many loss terms and hyperparameters, which may make it difficult to tune and apply. The training objective combines several distillation and regularization terms, each with its own tunable coefficient. These coefficients also vary heavily across models and compression ratios. Although the paper provides some ablation studies and a short appendix for this, I find the hyperparameter analysis insufficient to show how sensitive the method is to these choices. This may make the method very expensive when training with other models.
-
The ablation comparisons do not fully separate the benefits of adaptive policies from the additional calibration and distillation. Furthermore, the limitations of proposed method are not fully discussed, especially those related to using the straight-through estimator (STE) for sparsification.
Questions for authors
-
Could the authors provide more evidence about the convergence speed and detailed analysis of the proposed method, such as training loss curves and performance across optimization steps? Since the objective contains several loss terms, this would help clarify the optimization behavior.
-
Could the authors provide a more detailed sensitivity analysis of the coefficients of the proposed loss terms? In particular, how sensitive is the performance to these coefficients across different models and compression ratios?
-
Could the authors discuss the limitations and possible failure modes of the proposed method? Also, why do the authors choose STE for learning sparsity ratios instead of other differentiable sparsification methods? Are there any stability or optimization issues with this choice?
7 comments
Response to Reviewer 3rq7 (2/5)
Table R12. Coefficient transfer on Med-MoE, where Tuned uses the coefficients of the paper for each backbone and Transferred reuses the coefficients tuned for the other backbone without retuning.
| Backbone | Ratio | Method | VQA-RAD Open | VQA-RAD Closed | SLAKE Open | SLAKE Closed | PathVQA Open | PathVQA Closed | Avg | ΔAvg |
|---|---|---|---|---|---|---|---|---|---|---|
| StableLM | 10× | DeRS-SP | 47.25 | 82.35 | 83.59 | 84.38 | 33.29 | 90.98 | 70.31 | – |
| Tuned | 51.39 | 82.72 | 83.84 | 84.13 | 33.25 | 90.74 | 71.01 | +0.70 | ||
| Transferred | 51.86 | 81.99 | 83.71 | 84.62 | 33.22 | 90.80 | 71.03 | +0.72 | ||
| 20× | DeRS-SP | 46.57 | 82.72 | 82.88 | 84.86 | 33.12 | 90.95 | 70.18 | – | |
| Tuned | 51.02 | 81.99 | 83.85 | 84.86 | 33.26 | 91.04 | 71.00 | +0.82 | ||
| Transferred | 50.63 | 81.99 | 83.04 | 83.89 | 33.29 | 90.86 | 70.62 | +0.44 | ||
| 100× | DeRS-SP | 44.60 | 80.88 | 82.75 | 85.10 | 32.71 | 90.68 | 69.45 | – | |
| Tuned | 51.96 | 82.35 | 83.25 | 85.34 | 33.48 | 90.83 | 71.20 | +1.75 | ||
| Transferred | 50.94 | 82.35 | 83.10 | 85.10 | 33.35 | 90.71 | 70.92 | +1.47 | ||
| Phi | 10× | DeRS-SP | 55.61 | 83.46 | 83.82 | 86.30 | 34.22 | 91.27 | 72.45 | – |
| Tuned | 56.83 | 83.82 | 83.75 | 86.30 | 34.53 | 91.24 | 72.74 | +0.29 | ||
| Transferred | 57.76 | 83.46 | 83.30 | 86.30 | 34.49 | 91.27 | 72.76 | +0.31 | ||
| 20× | DeRS-SP | 55.52 | 83.46 | 83.50 | 86.30 | 34.34 | 91.27 | 72.40 | – | |
| Tuned | 57.76 | 83.32 | 83.50 | 86.30 | 34.57 | 91.06 | 72.75 | +0.35 | ||
| Transferred | 56.64 | 83.46 | 83.59 | 86.06 | 34.26 | 91.27 | 72.55 | +0.15 | ||
| 100× | DeRS-SP | 54.03 | 82.35 | 83.02 | 85.82 | 34.13 | 90.89 | 71.71 | – | |
| Tuned | 56.73 | 83.82 | 84.08 | 86.06 | 34.59 | 91.36 | 72.77 | +1.06 | ||
| Transferred | 56.45 | 83.82 | 84.08 | 85.82 | 34.46 | 91.24 | 72.65 | +0.94 |
[W2] Ablation does not isolate the adaptive policy; STE limitations
Because all network weights remain frozen throughout calibration and distillation, optimization updates only the two policy scalars per projection (its sparsity ratio and rescaling factor), so calibration and distillation cannot improve accuracy by themselves and act solely through the learned policy.
To isolate the two components of this policy along our default schedule, we evaluate the checkpoint at the end of Stage 1, which learns only where to prune while retaining DeRS-SP's analytic rescaling, so that its only difference from DeRS-SP is the learned sparsity; Stage 2 then learns rescaling. Tables R13 to R15 compare DeRS-SP, Stage 1 only and full AdaUpMoE on MoE-LLaVA, Med-MoE and FT (whereas the -only variant in Tables S4 and S5 of the supplementary trains sparsity for all steps).
Stage 1. The learned sparsity alone outperforms DeRS-SP in all nine settings, with gains growing alongside the compression ratio: whereas retaining a tenth of every delta at 10× is tolerable for most components, uniform dropping at 100× starves sensitive ones, so reallocating budget adds 0.6 to 1.5 points on MoE-LLaVA and 1.4 points on Med-MoE. On FT at 100×, where DeRS-SP drops from the uncompressed 60.12 to 53.23, Stage 1 alone recovers Avg to 59.55. These improvements concentrate on knowledge-intensive tasks (SQA, TextVQA, open-ended VQA-RAD and HumanEval).
Stage 2. Learned rescaling adds a further 0.3 to 1.2 points in every setting. While analytic preserves each delta's expected value, it does not preserve the delta's effect on the output once most entries are dropped, and Stage 2 corrects this residual error by aligning each component with the compressed teacher. On MoE-LLaVA-Qwen, Stage 2 recovers 1.2 of the 1.8-point gap remaining after Stage 1; on FT at 10× and 20×, where Stage 1 changes Avg by at most 0.6, Stage 2 accounts for most of the improvement, adding 1.0 to 1.1 points mainly on HumanEval and HumanEval+.
STE. STE [1] keeps an exact hard 0/1 mask in the forward pass, so training and inference can use the same sparse model. Under the same objective, it achieves the best Avg in 7 of 9 settings against Gumbel-sigmoid and Hard Concrete and stays within 0.21 Avg in the rest (Tables R16–R17); besides, its biased gradient and other limitations are discussed in our reply to [Q3].
Response to Reviewer 3rq7 (3/5)
Table R13. Contribution of each stage on MoE-LLaVA at 100×. Stage 1 only uses 30 steps to learn the sparsity ratios and keeps the analytic rescaling of DeRS-SP. AdaUpMoE adds 70 Stage 2 steps to learn the rescaling.
| Backbone | Method | SQA-I | VQA-T | GQA | POPE | MM-Vet | MMBench | VQAv2 | Avg | ΔAvg |
|---|---|---|---|---|---|---|---|---|---|---|
| StableLM | DeRS-SP | 58.88 | 47.84 | 59.06 | 86.82 | 28.20 | 52.73 | 75.09 | 58.37 | – |
| Stage 1 only | 61.38 | 48.59 | 59.37 | 86.77 | 27.79 | 54.49 | 75.52 | 59.13 | +0.76 | |
| AdaUpMoE | 63.21 | 49.82 | 59.21 | 87.19 | 27.60 | 55.22 | 75.96 | 59.74 | +1.37 | |
| Qwen | DeRS-SP | 58.35 | 43.32 | 58.48 | 86.96 | 24.40 | 48.42 | 73.19 | 56.16 | – |
| Stage 1 only | 59.05 | 45.01 | 59.76 | 87.53 | 26.56 | 51.43 | 74.42 | 57.68 | +1.52 | |
| AdaUpMoE | 61.63 | 47.18 | 60.22 | 87.63 | 27.40 | 53.40 | 74.89 | 58.91 | +2.75 | |
| Phi | DeRS-SP | 66.93 | 50.43 | 60.69 | 87.64 | 31.30 | 62.44 | 76.81 | 62.32 | – |
| Stage 1 only | 68.42 | 51.27 | 61.22 | 87.45 | 32.29 | 62.29 | 77.34 | 62.90 | +0.58 | |
| AdaUpMoE | 68.22 | 50.91 | 61.27 | 87.27 | 33.70 | 63.38 | 77.40 | 63.16 | +0.84 |
Table R14. Contribution of each stage on Med-MoE-StableLM, with stage definitions as in Table R13.
| Ratio | Method | VQA-RAD Open | VQA-RAD Closed | SLAKE Open | SLAKE Closed | PathVQA Open | PathVQA Closed | Avg | ΔAvg |
|---|---|---|---|---|---|---|---|---|---|
| 10× | DeRS-SP | 47.25 | 82.35 | 83.59 | 84.38 | 33.29 | 90.98 | 70.31 | – |
| Stage 1 only | 48.62 | 82.35 | 82.97 | 84.13 | 33.29 | 91.04 | 70.40 | +0.09 | |
| AdaUpMoE | 51.39 | 82.72 | 83.84 | 84.13 | 33.25 | 90.74 | 71.01 | +0.70 | |
| 20× | DeRS-SP | 46.57 | 82.72 | 82.88 | 84.86 | 33.12 | 90.95 | 70.18 | – |
| Stage 1 only | 50.08 | 82.35 | 83.22 | 84.13 | 33.46 | 91.06 | 70.72 | +0.54 | |
| AdaUpMoE | 51.02 | 81.99 | 83.85 | 84.86 | 33.26 | 91.04 | 71.00 | +0.82 | |
| 100× | DeRS-SP | 44.60 | 80.88 | 82.75 | 85.10 | 32.71 | 90.68 | 69.45 | – |
| Stage 1 only | 50.74 | 83.09 | 82.84 | 84.38 | 33.45 | 90.86 | 70.89 | +1.44 | |
| AdaUpMoE | 51.96 | 82.35 | 83.25 | 85.34 | 33.48 | 90.83 | 71.20 | +1.75 |
Table R15. Contribution of each stage on FT, with stage definitions as in Table R13.
| Ratio | Method | HumanEval | HumanEval+ | MBPP | MBPP+ | Avg | ΔAvg |
|---|---|---|---|---|---|---|---|
| 10× | DeRS-SP | 63.40 | 59.10 | 62.20 | 52.90 | 59.40 | – |
| Stage 1 only | 63.40 | 59.15 | 63.16 | 52.88 | 59.64 | +0.24 | |
| AdaUpMoE | 65.20 | 61.00 | 63.40 | 53.10 | 60.68 | +1.28 | |
| 20× | DeRS-SP | 61.00 | 57.30 | 62.70 | 53.10 | 58.52 | – |
| Stage 1 only | 62.20 | 58.54 | 62.91 | 52.88 | 59.13 | +0.61 | |
| AdaUpMoE | 64.60 | 59.80 | 62.90 | 53.60 | 60.22 | +1.70 | |
| 100× | DeRS-SP | 57.90 | 53.00 | 55.10 | 46.90 | 53.23 | – |
| Stage 1 only | 66.46 | 60.98 | 59.65 | 51.13 | 59.55 | +6.32 | |
| AdaUpMoE | 64.00 | 60.40 | 63.90 | 53.60 | 60.48 | +7.25 |
Table R16. Gradient estimators for the sparsity masks on Med-MoE under the same objective and coefficients.
| Backbone | Ratio | Estimator | VQA-RAD Open | VQA-RAD Closed | SLAKE Open | SLAKE Closed | PathVQA Open | PathVQA Closed | Avg |
|---|---|---|---|---|---|---|---|---|---|
| Phi | 10× | STE | 56.83 | 83.82 | 83.75 | 86.30 | 34.53 | 91.24 | 72.74 |
| Gumbel-sigmoid | 57.76 | 83.82 | 83.69 | 86.30 | 34.67 | 91.27 | 72.92 | ||
| Hard Concrete | 58.59 | 84.56 | 83.54 | 85.34 | 34.42 | 91.27 | 72.95 | ||
| 20× | STE | 57.76 | 83.32 | 83.50 | 86.30 | 34.57 | 91.06 | 72.75 | |
| Gumbel-sigmoid | 55.80 | 83.09 | 83.98 | 86.30 | 34.70 | 91.39 | 72.54 | ||
| Hard Concrete | 59.74 | 84.19 | 83.14 | 85.10 | 33.29 | 91.21 | 72.78 | ||
| 100× | STE | 56.73 | 83.82 | 84.08 | 86.06 | 34.59 | 91.36 | 72.77 | |
| Gumbel-sigmoid | 56.17 | 83.46 | 84.12 | 85.82 | 34.17 | 91.24 | 72.50 | ||
| Hard Concrete | 57.83 | 84.56 | 82.84 | 84.62 | 33.28 | 91.06 | 72.36 | ||
| StableLM | 10× | STE | 51.39 | 82.72 | 83.84 | 84.13 | 33.25 | 90.74 | 71.01 |
| Gumbel-sigmoid | 51.30 | 81.99 | 83.69 | 84.38 | 33.23 | 90.77 | 70.89 | ||
| Hard Concrete | 52.57 | 81.62 | 82.00 | 82.93 | 32.50 | 90.33 | 70.32 | ||
| 20× | STE | 51.02 | 81.99 | 83.85 | 84.86 | 33.26 | 91.04 | 71.00 | |
| Gumbel-sigmoid | 50.94 | 82.35 | 83.23 | 83.41 | 33.36 | 90.86 | 70.69 | ||
| Hard Concrete | 52.94 | 81.62 | 82.39 | 83.89 | 32.56 | 90.39 | 70.63 | ||
| 100× | STE | 51.96 | 82.35 | 83.25 | 85.34 | 33.48 | 90.83 | 71.20 | |
| Gumbel-sigmoid | 49.80 | 81.99 | 83.10 | 84.86 | 33.59 | 90.77 | 70.68 | ||
| Hard Concrete | 51.78 | 80.88 | 82.02 | 83.41 | 31.73 | 90.24 | 70.01 |
Response to Reviewer 3rq7 (1/5)
[W1] Many loss terms and hyperparameters
Thank you for raising this concern. In practice, hyperparameter selection is lightweight across three levels: (i) for a given model and ratio, one set of coefficients serves all benchmarks; (ii) across ratios of the same model, we mainly adjust the sparsity and rescaling weights; and (iii) across models, all runs follow one balancing rule (Section 2.4 of the supplementary) that approximately balances the initial weighted loss shares. Additionally, as detailed in our reply to [Q2] (Tables R11 and R12), across 15 single-coefficient perturbations spanning 0.33× to 7× the defaults, Avg changes by at most 0.42 percent, and the coefficients tuned on one Med-MoE backbone can transfer directly to the other without retuning.
Table R11. Sensitivity to the loss coefficients on MoE-LLaVA-StableLM at 100×, where each row changes the coefficient of one loss term and keeps the others at their defaults. The defaults of the paper are λt=7, λct=3, λmod=10−3, λsp=1 and λrs=5×10−5. ΔAvg is relative to the AdaUpMoE default row.