- Rating
- 3
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies extreme compression of expert-specific delta weights in upcycled MoE models and argues that both weight distribution and knowledge distribution become important under very high sparsity. It proposes UltraSparse, combining Weight Distribution Preservation (WDP) with Knowledge Distribution-Guided Hybrid Sparsification, and evaluates it across multiple upcycled MoE families, tasks, and modalities at compression ratios from 20× to 1000×. The reported results show that UltraSparse generally outperforms the compared DeRS-based compression methods, particularly under extreme compression.
Critical strengths
The experimental scope is a clear strength. The benchmark covers 5 upcycling approaches, 7 models, and 30 datasets across 5 task/modalities, including multimodal, medical, and code-generation settings.
The method is also tested over a wide compression range up to 1000×, and the tables show that UltraSparse remains considerably more stable than DeRS-SP in several extreme-sparsity settings where the baseline performance collapses.
Critical weaknesses
IF DONE
The reported 20×–1000× compression ratios apply to the expert-specific parameters/delta weights, not necessarily to the complete MoE model. The paper would be stronger if it also reported the resulting end-to-end model-size reduction, GPU memory consumption, inference latency, and throughput, so that the practical deployment benefit of the claimed extreme compression is clear.
Performance preservation at the most extreme compression levels is not uniform across models. For example, on Uni-MoE the average score decreases from 70.13 for the vanilla model to 55.50 at 1000×, even though UltraSparse remains much better than DeRS-SP. A deeper analysis of when 1000× compression remains reliable, and when a lower compression ratio should be preferred, would make the conclusions more precise.
Questions for authors
Plesae follow strengths and weaknesses.
4 comments
Response to Reviewer rr53 (2/2)
[W2] When is 1000× reliable?
On the model side, the reliability of 1000× compression depends on where the task competence resides. On the method side, UltraSparse ensures that 1000× degrades gracefully rather than collapsing. Table R20 summarizes the two upcycling regimes.
Table R20. Upcycling regime, relative delta norm , share of the uncompressed score retained at 1000×, and the recommended ratio.
| Model | Upcycling regime | r | Retained at 1000× | Recommended |
|---|---|---|---|---|
| Med-MoE-Phi, Med-MoE-StableLM | Joint backbone–expert | 0.019–0.020 | 98%–99% | 1000× |
| MoE-LLaVA-Phi | Joint backbone–expert | 0.027 | 98.5% | 1000× |
| MoE-LLaVA-Qwen | Joint backbone–expert | 0.046 | 95.4% | 1000× |
| XFT | Joint, uncompressed shared expert | 0.070 | 98.6% | 1000× |
| MultiPL-MoE | Frozen backbone, new experts | 0.045 | 79.1%–86.7% | 20×–100× |
| Uni-MoE | Independently tuned experts | 0.211 | 79.1% | 20×–100× |
1. On the model side, 1000× is reliable when the shared parameters carry the task competence (Regime I). In joint backbone–expert upcycling (Med-MoE, MoE-LLaVA) and in architectures with an always-active shared expert (XFT), the shared parameters already carry the main task competence, and the expert deltas apply fine-grained corrections. Within jointly upcycled models, the relative delta norm r orders the sensitivity to compression. At 1000×, UltraSparse retains 98%–99% of the uncompressed score on Med-MoE (r = 0.019–0.020), 98.5% on MoE-LLaVA-Phi (r = 0.027) and 95.4% on MoE-LLaVA-Qwen (r = 0.046). XFT retains 98.6% despite r = 0.070, because its uncompressed shared expert preserves the general capability.
2. A moderate ratio is preferable when domain-specific capability resides in the expert deltas (Regime II). In frozen-backbone upcycling (MultiPL-MoE) and in assemblies of independently tuned experts (Uni-MoE), the shared base lacks the target-domain skills. Replacing every Uni-MoE expert with the dense Vicuna FFN [1] lowers the average from 70.13 to 40.97 and RACE-Audio to 4.95, which shows that this capability is carried by large expert deltas (r = 0.211). The regime, not the delta norm alone, is decisive. MultiPL-MoE has r = 0.045, close to MoE-LLaVA-Qwen, yet its new experts alone carry the multilingual adaptation that the frozen backbone lacks. When accuracy is the priority, a moderate ratio is therefore preferable, and Uni-MoE retains 91% at 20×. Even at 1000×, UltraSparse degrades gracefully. From 0.1% of the delta entries it keeps 55.50 on Uni-MoE, recovering half of the gap to the uncompressed model and restoring RACE-Audio to 43.39, and it keeps 86.7% on HumanEval and 79.1% on MBPP for MultiPL-MoE. DeRS-SP, in contrast, drops to zero from 100× on MultiPL-MoE and from 200× on Uni-MoE.
3. On the method side, Top-dominant allocation keeps 1000× stable in both regimes. The variance of rescaled Coverage grows as , so at 1000× a 50/50 split falls below 61 on MoE-LLaVA-Phi and collapses on MoE-LLaVA-Qwen. UltraSparse switches to a Top-dominant split with λ ≥ 0.9 at this ratio and remains stable on every evaluated model. Beyond 100×, Uni-MoE thus declines only from 57.36 to 55.50.
Operating rule. We recommend 1000× for Regime I and 20×–100× for Regime II when accuracy is the priority. The revision states this rule together with Table R20, the delta statistics and the Uni-MoE dense-expert control.
References
[1] Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023.
Response to Reviewer rr53
Dear Reviewer rr53,
Thank you again for your positive assessment and helpful suggestions. We would like to kindly follow up on our response. We reported the checkpoint size, memory, latency and throughput of the full model in Tables R18 and R19, and stated when 1000× compression is reliable with an operating rule in Table R20.
Is there anything you would like us to discuss or clarify further? We would be happy to respond promptly.
Best regards,
The Authors
Response to Reviewer rr53
Dear Reviewer rr53,
As the discussion period is drawing to a close, we wanted to politely follow up to see if our previous response has addressed all of your concerns. Please let us know if you have any further questions or points you would like to discuss, we would be more than happy to clarify.
Best regards,
The Authors
Response to Reviewer rr53 (1/2)
We sincerely thank the reviewer for the positive assessment and the helpful suggestions. We address each point below.
[W1] End-to-end size, memory, latency, throughput
The 20×–1000× ratios target exactly the part that upcycling adds, the expert-specific deltas, and this saving carries through to all four requested metrics. At 1000×, a four-expert MoE-LLaVA comes within 0.3% of the storage of its dense backbone, and our hybrid runtime serves it with higher throughput, lower latency and less GPU memory.
1. Model size approaches that of the dense backbone. An upcycled MoE checkpoint consists of two parts. The shared dense weights, namely the non-expert modules and one shared base per MoE layer, together match the dense backbone in size. The expert-specific deltas account for the rest, 44.9% of MoE-LLaVA-Phi and 36.2% of MoE-LLaVA-Qwen. Counting every CSR index byte, UltraSparse at 1000× compresses the deltas of all four experts from 4.69 GiB to 16 MiB on MoE-LLaVA-Phi and from 2.27 GiB to 8.6 MiB on MoE-LLaVA-Qwen. The full checkpoint therefore shrinks by 44.7% (1.81×) and 36.0% (1.56×), to within 0.3% of the shared dense weights (Table R18).
Table R18. Checkpoint size of MoE-LLaVA on disk, including all CSR index bytes.
2. Memory, latency and throughput improve simultaneously. Our hybrid runtime keeps only the shared bases and the sparse deltas resident. During prefill, which is compute-bound, it temporarily materializes one dense projection at a time for a standard GEMM. During decoding, which is memory-bound, it reads the shared base once per step and adds the sparse deltas with Triton CSR kernels, instead of reading one dense matrix per expert. Compared with the original DeepSpeed execution of the same weights, whose cost equals that of the uncompressed model, the runtime reduces peak GPU memory by 32%–42% and latency by 13%–19% and increases throughput by 15%–24% at every ratio from 100× to 1000× (Table R19). Reconstruction is bit-exact, and for all ten prompts, both models and all three ratios, the sparse path predicts the same next token as dense execution.
Table R19. End-to-end runtime of the same compressed weights on one RTX 4090 with batch size 1 and 32 new tokens per prompt. Dense uses the original DeepSpeed MoE layers. Latency is the mean time per prompt, prefill included.
3. The shared dense weights compose with quantization. With INT4 shared dense weights and 1-bit sparse deltas, MoE-LLaVA-Phi scores 66.12, matching whole-model INT4 at 66.19 while storing all expert deltas in 71 MiB, so the remaining dense part can be compressed further with standard quantization. The revision reports full-model size, memory, latency and throughput alongside the delta ratios.