acceptodds
Under review as a conference paper at ICLR 2027

UltraSparse: Extreme Compression of Upcycled MoE Models

Abstract

Mixture-of-Experts (MoE) has become a promising architecture for scaling large models through sparse activation. Recently, upcycling dense models into MoE models has emerged as an effective approach for constructing powerful sparse models, motivating the need for effective compression techniques for upcycled MoE models. Recent studies reveal that upcycled MoE models exhibit an expert-shared base weights and expert-specific delta weights structure, where the delta weights contain substantial redundancy and compression potential. However, how to achieve extreme compression of expert-specific delta weights while preserving model capability remains largely unexplored. In this work, we conduct the first systematic study on the extreme sparsification of expert-specific delta weights and reveal that both weight distribution and knowledge distribution play critical roles in governing model performance under ultra-high sparsity regimes. Building upon these insights, we present UltraSparse, the first framework for extreme compression of upcycled MoE models. A Weight Distribution Preservation module is proposed to maintain consistency between the weight distributions before and after sparsification. A Knowledge Distribution-Guided Hybrid Sparsification module is proposed to preserve critical knowledge associated with high-magnitude weights while retaining complementary knowledge contributed by low-magnitude weights. To comprehensively evaluate compression methods for upcycled MoE models, we establish a large-scale benchmark covering diverse upcycling approaches, model architectures of different scales, and datasets spanning different tasks and modalities. Extensive experiments demonstrate that UltraSparse consistently outperforms existing compression methods across various settings, and achieves unprecedented compression ratios, reaching up to 1000× compression while maintaining strong model performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture of Experts, Model Compression, Sparse Upcycling, Extreme Sparsification, Delta Compression, Weight Distribution Preservation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_rr53OpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies extreme compression of expert-specific delta weights in upcycled MoE models and argues that both weight distribution and knowledge distribution become important under very high sparsity. It proposes UltraSparse, combining Weight Distribution Preservation (WDP) with Knowledge Distribution-Guided Hybrid Sparsification, and evaluates it across multiple upcycled MoE families, tasks, and modalities at compression ratios from 20× to 1000×. The reported results show that UltraSparse generally outperforms the compared DeRS-based compression methods, particularly under extreme compression.

Critical strengths

The experimental scope is a clear strength. The benchmark covers 5 upcycling approaches, 7 models, and 30 datasets across 5 task/modalities, including multimodal, medical, and code-generation settings.

The method is also tested over a wide compression range up to 1000×, and the tables show that UltraSparse remains considerably more stable than DeRS-SP in several extreme-sparsity settings where the baseline performance collapses.

Critical weaknesses

IF DONE

The reported 20×–1000× compression ratios apply to the expert-specific parameters/delta weights, not necessarily to the complete MoE model. The paper would be stronger if it also reported the resulting end-to-end model-size reduction, GPU memory consumption, inference latency, and throughput, so that the practical deployment benefit of the claimed extreme compression is clear.

Performance preservation at the most extreme compression levels is not uniform across models. For example, on Uni-MoE the average score decreases from 70.13 for the vanilla model to 55.50 at 1000×, even though UltraSparse remains much better than DeRS-SP. A deeper analysis of when 1000× compression remains reliable, and when a lower compression ratio should be preferred, would make the conclusions more precise.

Questions for authors

Plesae follow strengths and weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer rr53 (1/2)

We sincerely thank the reviewer for the positive assessment and the helpful suggestions. We address each point below.

[W1] End-to-end size, memory, latency, throughput

The 20×–1000× ratios target exactly the part that upcycling adds, the expert-specific deltas, and this saving carries through to all four requested metrics. At 1000×, a four-expert MoE-LLaVA comes within 0.3% of the storage of its dense backbone, and our hybrid runtime serves it with higher throughput, lower latency and less GPU memory.

1. Model size approaches that of the dense backbone. An upcycled MoE checkpoint consists of two parts. The shared dense weights, namely the non-expert modules and one shared base per MoE layer, together match the dense backbone in size. The expert-specific deltas account for the rest, 44.9% of MoE-LLaVA-Phi and 36.2% of MoE-LLaVA-Qwen. Counting every CSR index byte, UltraSparse at 1000× compresses the deltas of all four experts from 4.69 GiB to 16 MiB on MoE-LLaVA-Phi and from 2.27 GiB to 8.6 MiB on MoE-LLaVA-Qwen. The full checkpoint therefore shrinks by 44.7% (1.81×) and 36.0% (1.56×), to within 0.3% of the shared dense weights (Table R18).

Table R18. Checkpoint size of MoE-LLaVA on disk, including all CSR index bytes.

MoE-LLaVA-PhiMoE-LLaVA-Qwen
Uncompressed100×1000×Uncompressed100×1000×
Checkpoint size10.45 GiB5.89 GiB5.78 GiB6.27 GiB4.06 GiB4.01 GiB
Shared dense weights5.76 GiB4.00 GiB
Expert-specific deltas4.69 GiB131 MiB16 MiB2.27 GiB64 MiB8.6 MiB

2. Memory, latency and throughput improve simultaneously. Our hybrid runtime keeps only the shared bases and the sparse deltas resident. During prefill, which is compute-bound, it temporarily materializes one dense projection at a time for a standard GEMM. During decoding, which is memory-bound, it reads the shared base once per step and adds the sparse deltas with Triton CSR kernels, instead of reading one dense matrix per expert. Compared with the original DeepSpeed execution of the same weights, whose cost equals that of the uncompressed model, the runtime reduces peak GPU memory by 32%–42% and latency by 13%–19% and increases throughput by 15%–24% at every ratio from 100× to 1000× (Table R19). Reconstruction is bit-exact, and for all ten prompts, both models and all three ratios, the sparse path predicts the same next token as dense execution.

Table R19. End-to-end runtime of the same compressed weights on one RTX 4090 with batch size 1 and 32 new tokens per prompt. Dense uses the original DeepSpeed MoE layers. Latency is the mean time per prompt, prefill included.

Peak memory (GiB)Latency (s)Throughput (tokens/s)
ModelRatioDenseOursDenseOursReductionDenseOursSpeedup
MoE-LLaVA-Qwen100×6.944.701.1811.0071.172×27.0931.761.172×
200×6.954.651.1660.9981.168×27.4532.051.168×
1000×6.954.611.1841.0291.150×27.0331.101.150×
MoE-LLaVA-Phi100×11.016.511.6201.3101.237×19.7624.431.237×
200×11.016.411.6641.3531.230×19.2423.651.230×
1000×11.016.341.6341.3351.224×19.5923.961.224×

3. The shared dense weights compose with quantization. With INT4 shared dense weights and 1-bit sparse deltas, MoE-LLaVA-Phi scores 66.12, matching whole-model INT4 at 66.19 while storing all expert deltas in 71 MiB, so the remaining dense part can be compressed further with standard quantization. The revision reports full-model size, memory, latency and throughput alongside the delta ratios.

AuthorsOpenReview ↗

Response to Reviewer rr53 (2/2)

[W2] When is 1000× reliable?

On the model side, the reliability of 1000× compression depends on where the task competence resides. On the method side, UltraSparse ensures that 1000× degrades gracefully rather than collapsing. Table R20 summarizes the two upcycling regimes.

Table R20. Upcycling regime, relative delta norm r=∥Δ∥F/∥Wbase∥Fr = \lVert\boldsymbol{\Delta}\rVert_F / \lVert\mathbf{W}^{\text{base}}\rVert_F, share of the uncompressed score retained at 1000×, and the recommended ratio.

ModelUpcycling regimerRetained at 1000×Recommended
Med-MoE-Phi, Med-MoE-StableLMJoint backbone–expert0.019–0.02098%–99%1000×
MoE-LLaVA-PhiJoint backbone–expert0.02798.5%1000×
MoE-LLaVA-QwenJoint backbone–expert0.04695.4%1000×
XFTJoint, uncompressed shared expert0.07098.6%1000×
MultiPL-MoEFrozen backbone, new experts0.04579.1%–86.7%20×–100×
Uni-MoEIndependently tuned experts0.21179.1%20×–100×

1. On the model side, 1000× is reliable when the shared parameters carry the task competence (Regime I). In joint backbone–expert upcycling (Med-MoE, MoE-LLaVA) and in architectures with an always-active shared expert (XFT), the shared parameters already carry the main task competence, and the expert deltas apply fine-grained corrections. Within jointly upcycled models, the relative delta norm r orders the sensitivity to compression. At 1000×, UltraSparse retains 98%–99% of the uncompressed score on Med-MoE (r = 0.019–0.020), 98.5% on MoE-LLaVA-Phi (r = 0.027) and 95.4% on MoE-LLaVA-Qwen (r = 0.046). XFT retains 98.6% despite r = 0.070, because its uncompressed shared expert preserves the general capability.

2. A moderate ratio is preferable when domain-specific capability resides in the expert deltas (Regime II). In frozen-backbone upcycling (MultiPL-MoE) and in assemblies of independently tuned experts (Uni-MoE), the shared base lacks the target-domain skills. Replacing every Uni-MoE expert with the dense Vicuna FFN [1] lowers the average from 70.13 to 40.97 and RACE-Audio to 4.95, which shows that this capability is carried by large expert deltas (r = 0.211). The regime, not the delta norm alone, is decisive. MultiPL-MoE has r = 0.045, close to MoE-LLaVA-Qwen, yet its new experts alone carry the multilingual adaptation that the frozen backbone lacks. When accuracy is the priority, a moderate ratio is therefore preferable, and Uni-MoE retains 91% at 20×. Even at 1000×, UltraSparse degrades gracefully. From 0.1% of the delta entries it keeps 55.50 on Uni-MoE, recovering half of the gap to the uncompressed model and restoring RACE-Audio to 43.39, and it keeps 86.7% on HumanEval and 79.1% on MBPP for MultiPL-MoE. DeRS-SP, in contrast, drops to zero from 100× on MultiPL-MoE and from 200× on Uni-MoE.

3. On the method side, Top-dominant allocation keeps 1000× stable in both regimes. The variance of rescaled Coverage grows as O(1/k∗)\mathcal{O}(1/k^{\ast}), so at 1000× a 50/50 split falls below 61 on MoE-LLaVA-Phi and collapses on MoE-LLaVA-Qwen. UltraSparse switches to a Top-dominant split with λ ≥ 0.9 at this ratio and remains stable on every evaluated model. Beyond 100×, Uni-MoE thus declines only from 57.36 to 55.50.

Operating rule. We recommend 1000× for Regime I and 20×–100× for Regime II when accuracy is the priority. The revision states this rule together with Table R20, the delta statistics and the Uni-MoE dense-expert control.

References

[1] Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023.

AuthorsOpenReview ↗

Response to Reviewer rr53

Dear Reviewer rr53,

Thank you again for your positive assessment and helpful suggestions. We would like to kindly follow up on our response. We reported the checkpoint size, memory, latency and throughput of the full model in Tables R18 and R19, and stated when 1000× compression is reliable with an operating rule in Table R20.

Is there anything you would like us to discuss or clarify further? We would be happy to respond promptly.

Best regards,

The Authors

AuthorsOpenReview ↗

Response to Reviewer rr53

Dear Reviewer rr53,

As the discussion period is drawing to a close, we wanted to politely follow up to see if our previous response has addressed all of your concerns. Please let us know if you have any further questions or points you would like to discuss, we would be more than happy to clarify.

Best regards,
The Authors

Reviewer_g5epOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Upcycled MoEs decompose trained expert weights into expert-shared base weights and expert-specific delta weights. Prior delta sparsification methods suffer severe performance degradation or total collapse under extreme sparsity regimes (e.g., 100x to 1000x delta compression). The authors identify weight distribution preservation and knowledge distribution preservation to resolve this. The primary contributions include the formulation of the distribution preservation paradigm, rigorous theoretical bias-variance analysis proving that global scaling shrinkage strictly reduces MSE over unbiased estimators, and a comprehensive benchmark across multimodal, vision-language, medical VQA, and multilingual code generation tasks.

Critical strengths
  • UltraSparse maintains competitive performance at up to 1000x delta compression (0.999 sparsity ratio) across complex reasoning benchmarks. While prior state-of-the-art baselines like DeRS completely collapse to near-zero scores on Uni-MoE and MultiPL-MoE beyond 100x compression, UltraSparse retains over 79% to 86% of the dense vanilla capacity.
  • The paper provides a mathematical analysis of the biased global scale factor 1/k*, proving via entry-wise variance and Frobenius MSE derivations that its shrinkage properties yield strictly lower total MSE than Coverage-unbiased scaling 1/e. It further establishes that allocation residual errors vanish as total retained parameters grow.
  • The framework is extensively tested across 5 distinct upcycling mechanisms, 7 base backbones (e.g., Qwen, Phi, Vicuna, StableLM, DeepSeek-Coder), and 30 datasets spanning VLM, audio-visual, medical, and multi-language code generation under various routing mechanisms (top-1, top-2, and segment routing).
Critical weaknesses
  • The method outputs unstructured, element-wise non-zero masks, which offer substantial theoretical parameter and storage savings but cannot directly translate into linear inference latency speedups on modern GPU architectures without specialized sparse kernels.
  • The optimal Top vs. Coverage budget split (lambda) fluctuates across models and modalities (e.g., 30%/70% for Qwen vs. 80%/20% for XFT) without an automatic, data-driven calibration mechanism.
  • The current pipeline is post-hoc and data-free. The paper lacks exploration of whether lightweight parameter fine-tuning (e.g., LoRA or router recalibration) could completely eliminate the remaining performance gap at 1000x compression.
Questions for authors
  • How does the reconstruction runtime overhead of WDP+ scale when dynamically materializing experts on device memory during generation?
  • Why does the optimal Top/Coverage split vary so drastically between language-heavy and vision-heavy tasks, and can this ratio be derived analytically from the delta matrix singular spectrum?
  • Have the authors benchmarked UltraSparse in conjunction with post-training weight quantization (e.g., FP4/INT4 base weights with 1-bit sparse deltas)?
5 comments
AuthorsOpenReview ↗

Response to Reviewer g5ep (1/3)

We sincerely thank the reviewer for the constructive review and for recognizing UltraSparse's results under extreme compression. We address each point below.

[W1Q1] Unstructured sparsity gives no direct speedup

Our Triton sparse kernels translate UltraSparse's unstructured masks into measured end-to-end gains. The resulting hybrid runtime increases generation throughput by 15.0%–17.2% on MoE-LLaVA-Qwen and 22.4%–23.7% on MoE-LLaVA-Phi and reduces peak GPU memory by 32%–42% across 100×–1000×. Expert reconstruction (Q1) occurs once per prompt during prefill and accounts for at most 1.2% of end-to-end latency.

1. [W1] Hybrid execution achieves higher throughput with a smaller memory footprint. The runtime keeps only the shared dense bases and the sparse CSR deltas resident and treats the two inference phases differently. Prefill processes many tokens at once and is compute-bound, so the runtime temporarily materializes one dense expert projection at a time, runs a standard GEMM and immediately releases the buffer, leaving no dense expert resident in memory. Decoding processes one token per step and is memory-bound, so each expert projection is evaluated as a shared dense product plus a sparse CSR delta product. Because the experts of a layer share one base, each decoding step reads it once instead of once per expert, which reduces the weight traffic that bounds single-token generation. Table R11 compares this runtime with the original DeepSpeed MoE layers executing the same weights. Reconstruction is bit-exact, and for all ten prompts, both models and all three ratios, the sparse path predicts the same next token as dense execution (mean KL divergence at most 5.4×10−55.4\times10^{-5}). On disk, counting every index byte, the expert-specific storage of MoE-LLaVA-Phi falls from 4.69 GiB to 16 MiB at 1000×.

Table R11. End-to-end generation on one RTX 4090 with batch size 1 and 32 new tokens per prompt, prefill included. Dense executes the same compressed weights with the original DeepSpeed MoE layers, at the cost of the uncompressed model.

Throughput (tokens/s)Peak memory (GiB)
ModelRatioDenseOursSpeedupDenseOursReduction
MoE-LLaVA-Qwen100×27.0931.761.172×6.944.701.48×
200×27.4532.051.168×6.954.651.49×
1000×27.0331.101.150×6.954.611.51×
MoE-LLaVA-Phi100×19.7624.431.237×11.016.511.69×
200×19.2423.651.230×11.016.411.72×
1000×19.5923.961.224×11.016.341.74×

2. [Q1] Reconstruction cost scales with model size but not with generation length. Materializing an expert writes its dense entries and scatters the retained deltas into them. Our runtime materializes every expert exactly once per prompt, during prefill, so the per-prompt cost is O(LMoE E ∣W∣)\mathcal{O}(L_{\mathrm{MoE}}\,E\,|\mathbf{W}|), where LMoEL_{\mathrm{MoE}} is the number of MoE layers, E the number of experts per layer and ∣W∣|\mathbf{W}| the number of parameters per expert. The cost decreases slightly with sparsity as the scatter work shrinks, and it is independent of the number of generated tokens because decoding never materializes weights. Measured with CUDA events during generation, reconstruction takes 6.0–6.2 ms per prompt on MoE-LLaVA-Qwen and 13.8–15.5 ms on MoE-LLaVA-Phi, at most 1.2% of end-to-end latency, and Table R11 already includes it. Materializing the two routed experts at every decoding step would instead cost 62 ms on MoE-LLaVA-Qwen and 215–226 ms on MoE-LLaVA-Phi over the 31 decoding steps, 10×–16× more, and this cost would grow linearly with generation length (Table R12).

Table R12. Reconstruction cost on one RTX 4090. Ours materializes all experts once per prompt, timed with CUDA events during generation, with its share of end-to-end latency. Per-step rebuilding materializes the two routed experts of every MoE layer at each of the 31 decoding steps.

Ours, once per promptPer-step rebuilding
ModelRatiomsShare of latencyms per stepms over 31 steps
MoE-LLaVA-Qwen100×6.210.6%2.0262.6
200×6.100.6%2.0062.0
1000×5.990.6%2.0162.2
MoE-LLaVA-Phi100×15.541.2%7.29226.1
200×14.131.1%6.92214.7
1000×13.781.1%7.15221.7

The revision adds the kernels, the timing protocol and Tables R11 and R12.

AuthorsOpenReview ↗

Response to Reviewer g5ep (2/3)

[W2Q2] Top/Coverage split needs manual tuning

The optimal split is governed by the shape of the expert deltas, which reflects how deeply the experts were fine-tuned, rather than by the task modality. A weight-only guideline distilled from the paper recovers near-optimal settings without search, and 100 unlabeled prompts suffice to calibrate the split automatically.

1. The tail of ∣Δ∣|\boldsymbol{\Delta}|, not the modality, determines the split. We measure the tail of the delta magnitudes by the dispersion ratio Std(∣Δ∣)/Mean(∣Δ∣)\mathrm{Std}(|\boldsymbol{\Delta}|)/\mathrm{Mean}(|\boldsymbol{\Delta}|). Fine-tuning depth and task difficulty determine how strongly the delta entries separate. Under joint upcycling with mild expert specialization, the updates spread evenly and ∣Δ∣|\boldsymbol{\Delta}| stays close to a folded Gaussian, so the largest entries are barely distinguishable from the rest. Top then has little to single out, and Coverage is needed to retain the broadly distributed, complementary adaptation. This is why MoE-LLaVA and Med-MoE, with dispersion ratios of 0.75–0.84, assign 50%–70% of the budget to Coverage. Deep, specialized fine-tuning instead concentrates the update in a few large entries. XFT exhibits such a heavy-tailed distribution (dispersion ratio 0.92) and assigns 80% to Top, which preserves these critical entries exactly. Uni-MoE provides the decisive control. It serves vision and audio tasks like the vision-language models, yet its independently and deeply tuned experts reach a dispersion ratio of 0.97, and its optimum is 90% Top (Table R13). The apparent language–vision gap therefore reflects fine-tuning depth and delta shape rather than modality.

Table R13. Expert-delta statistics and the Top share selected at 100×. Std/Mean is the dispersion ratio of ∣Δ∣|\boldsymbol{\Delta}| and Max/Mean its peak ratio.

ModelTasksStd/MeanMax/MeanTop share
Med-MoE-StableLMMedical VQA0.7510.640%
MoE-LLaVA-QwenVision-language0.7710.430%
MoE-LLaVA-PhiVision-language0.7819.250%
Med-MoE-PhiMedical VQA0.8418.840%
XFTCode generation0.92144.380%
Uni-MoEImage, audio, speech0.9770.390%

2. A setting guideline distilled from the paper. The dispersion ratio equals π/2−1≈0.76\sqrt{\pi/2-1}\approx0.76 for a folded Gaussian and 1.00 for a Laplacian, so their midpoint 0.88 separates near-Gaussian from heavy-tailed deltas. This yields three rules for the Top share λ and the Coverage bin count B. For compression up to 200×, (i) use λ=0.5 and B=16 when the dispersion ratio is below 0.88, and (ii) use λ=0.8 and B=8 when it is 0.88 or higher. (iii) At 1000×, use λ ≥ 0.9 regardless of the dispersion ratio, because the variance of rescaled Coverage grows as O(1/k∗)\mathcal{O}(1/k^{\ast}) (Appendix D). Without any search, the matching rule stays within 1.1 points of the per-ratio grid optimum on both MoE-LLaVA and XFT. It outperforms DeRS-SP by up to 4.39 points on MoE-LLaVA, and on XFT by 4.65 points at 100× and 10.98 points at 200× (Tables R14 and R15). Even rule (i), set on MoE-LLaVA without any XFT data, outperforms DeRS-SP on XFT by 2.13 and 10.21 points at the same two ratios. DeRS-SP has no allocation to tune, so these fixed settings give a like-for-like comparison. Rule (iii) agrees with the 1000× grid, where pure Top is optimal for all three MoE-LLaVA backbones.

Table R14. Rule (i) without search against DeRS-SP and the per-ratio grid optimum on MoE-LLaVA, in 4Avg.

MoE-LLaVA-PhiMoE-LLaVA-Qwen
RatioDeRS-SPRule (i)Grid optimumDeRS-SPRule (i)Grid optimum
100×66.8267.2167.2161.8063.5763.59
200×65.9366.8767.2457.1861.5762.00

Table R15. Fixed settings on XFT, applied unchanged at 100× and 200×.

RatioConfigurationHEvalHEval+MBPPMBPP+Avg
100×DeRS-SP57.3254.2758.6550.6355.21
UltraSparse, per-ratio setting (Paper)61.5957.3265.6654.8959.86
UltraSparse, rule (ii) (λ=0.8, B=8)61.5957.3265.6654.8959.86
UltraSparse, rule (i) (λ=0.5, B=16)60.8557.4460.8550.2357.34
200×DeRS-SP49.3944.5149.8742.8646.66
UltraSparse, per-ratio setting (Paper)62.8057.3262.1652.3858.66
UltraSparse, rule (ii) (λ=0.8, B=8)61.5958.2960.6050.0857.64
UltraSparse, rule (i) (λ=0.5, B=16)62.3257.9358.1049.1256.87
AuthorsOpenReview ↗

Response to Reviewer g5ep (3/3)

[W2Q2] Top/Coverage split needs manual tuning (continued)

3. The singular spectrum cannot determine the split, whereas unlabeled data can calibrate it. The singular spectrum is invariant under orthogonal transformations Δ↦UΔVT\boldsymbol{\Delta} \mapsto \mathbf{U}\boldsymbol{\Delta}\mathbf{V}^{\mathsf T}, yet such a rotation can spread a few large entries uniformly over the whole matrix and thus change the entry-magnitude distribution completely. Because Top and Coverage act on individual entries, the spectrum alone cannot determine λ, and entry-wise statistics such as the dispersion ratio are the matching data-free quantity. When unlabeled data are available, λ can also be calibrated automatically by minimizing the next-token KL divergence of the compressed model to the uncompressed model. With 100 unlabeled prompts, this selects λ=0.9 on MoE-LLaVA-Phi and λ=1 on MoE-LLaVA-Qwen at 1000×, recovering rule (iii) without labels or manual tuning (Table R16).

Table R16. Next-token KL divergence to the uncompressed model at 1000× on 100 unlabeled prompts (lower is better). λ=0 is pure Coverage and λ=1 is pure Top.

Modelλ=0λ=0.5λ=0.9λ=1
MoE-LLaVA-Phi10.650.350.170.21
MoE-LLaVA-Qwen12.258.160.300.23

The revision adds the setting guideline and Tables R13–R16.

[W3] No lightweight fine-tuning explored

We agree that lightweight fine-tuning such as LoRA [1] or router recalibration is a valuable complement. This paper deliberately targets the strictest setting, post-hoc compression that is both training-free and data-free, which applies when the original multimodal training data or training compute are unavailable. UltraSparse reads only the weights, and no sample, label or gradient enters compression. Under this constraint, it already retains at least 95% of the uncompressed score at 1000× on MoE-LLaVA-Phi, MoE-LLaVA-Qwen and Med-MoE, and 98.6% on XFT.

UltraSparse also provides a natural starting point for such fine-tuning. Its fixed sparse support defines a compact set of trainable entries, and LoRA or router recalibration can be added on top without changing the storage format. The revision discusses this extension as future work.

[Q3] Combination with weight quantization

Yes. Base quantization and UltraSparse act on disjoint parameters, the shared dense weights and the expert deltas, so they compose as orthogonal compression axes. We quantize every dense parameter (attention, dense FFNs, the multimodal projector and the shared expert bases) with calibration-free INT4 round-to-nearest at group size 128 [2]. The expert deltas are compressed with UltraSparse at 100×, and the retained values are kept in BF16 or binarized to 1 bit as sign(Δ)⋅μ∣Δ∣\mathrm{sign}(\Delta)\cdot \mu_{|\Delta|} [3] (Table R17).

Table R17. 4Avg of INT4 base quantization combined with UltraSparse 100×. An INT4 base with dense deltas serves as the whole-model INT4 reference.

Dense deltas (no sparsity)UltraSparse 100×UltraSparse 100× + 1-bit values
BaseMoE-LLaVA-PhiMoE-LLaVA-QwenMoE-LLaVA-PhiMoE-LLaVA-QwenMoE-LLaVA-PhiMoE-LLaVA-Qwen
BF1667.1965.0567.2163.5967.0061.95
INT466.1963.5466.0961.0266.1261.01

On MoE-LLaVA-Phi, the full stack of an INT4 base and 1-bit sparse deltas scores 66.12, matching the whole-model INT4 checkpoint at 66.19 while storing all expert deltas in 71 MiB. On MoE-LLaVA-Qwen, whose experts carry larger relative deltas (relative delta norm ∥Δ∥F/∥Wbase∥F=0.046\lVert\boldsymbol{\Delta}\rVert_F / \lVert\mathbf{W}^{\text{base}}\rVert_F = 0.046 versus 0.027 on MoE-LLaVA-Phi), the same stack retains 96.0% of the whole-model INT4 score. Once the base is INT4, binarizing the retained values leaves the score unchanged on both models, 66.12 versus 66.09 on MoE-LLaVA-Phi and 61.01 versus 61.02 on MoE-LLaVA-Qwen. The revision adds Table R17.

References

[1] Hu et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022.

[2] Frantar et al. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323, 2022.

[3] Liu et al. BitDelta: Your Fine-Tune May Only Be Worth One Bit. NeurIPS 2024.

Comments on the paper

11 comments
AuthorsOpenReview ↗

Response to Reviewer 1gd2 (1/6)

We sincerely thank the reviewer for the thorough and constructive review, including the careful cross-check of our supplementary tables. We address each point below.

[W1Q1] Unclear incremental mechanism

Starting from DeRS-SP, we first introduce KDHS (Knowledge Distribution-Guided Hybrid Sparsification, our hybrid Top–Coverage allocation) with fixed-size uniform tail sampling. We then replace uniform tail sampling with our magnitude-stratified sampling while matching the Top budget, the scaling factor 1/k* and the total non-zero count, which isolates stratification as the reviewer requested (Tables R1–R3). Both components improve performance in all six settings.

1. KDHS outperforms DeRS-SP in all six settings, with larger gains at higher sparsity. From 100× to 200×, its gain over DeRS-SP grows from +4.07 to +10.50 on XFT, from +0.12 to +0.66 on MoE-LLaVA-Phi, and from +1.12 to +4.35 on MoE-LLaVA-Qwen. The smaller differences on MoE-LLaVA-Phi at 100× reflect its limited headroom. There DeRS-SP is only 0.31 points below the uncompressed model (62.86 versus 63.17), and the full UltraSparse even exceeds the uncompressed score at 63.30.

2. Stratified sampling adds a further gain over uniform tail sampling in all six settings. Comparing the last two rows at each ratio, the gain is largest at 200×, with +1.50 on XFT, +0.51 on MoE-LLaVA-Phi and +0.88 on MoE-LLaVA-Qwen, which brings the total gains over DeRS-SP to +12.00, +1.17 and +5.23.

3. The stratification gain is statistically robust across random masks. We repeat the 200× comparison with three independent mask seeds on XFT, the model with the largest run-to-run fluctuation (Table R4). Stratified sampling scores 58.55 ± 0.27 versus 57.01 ± 0.25 for uniform sampling, a gain of +1.54 points with Welch's t-test giving p < 0.01, and every stratified run outperforms every uniform run.

The revision includes these ablations.

Table R1. Cumulative ablation on XFT. KDHS is the hybrid Top–Coverage allocation. The last two rows of each ratio share the same Top budget, scaling factor 1/k* and total non-zero count. Gain is over DeRS-SP.

RatioConfigurationHEvalHEval+MBPPMBPP+AvgGain
1×Uncompressed65.2061.0062.2052.1060.12–
100×DeRS-SP57.3254.2758.6550.6355.21–
+ Our KDHS + Uniform Sampling65.2461.5960.1550.1359.28+4.07
+ Our KDHS + Our Stratified Sampling61.5957.3265.6654.8959.86+4.65
200×DeRS-SP49.3944.5149.8742.8646.66–
+ Our KDHS + Uniform Sampling59.7656.1060.6552.1357.16+10.50
+ Our KDHS + Our Stratified Sampling62.8057.3262.1652.3858.66+12.00

Table R2. Cumulative ablation on MoE-LLaVA-Phi, set up as in Table R1.

RatioConfigurationSQA$^{\mathrm{I}}$VQA$^{\mathrm{T}}$GQAPOPEMM-VMMB$^{\mathrm{EN}}$VQA$^{\mathrm{v2}}$AvgGain
1×Uncompressed68.9651.1661.4287.2332.1063.7477.5863.17–
100×DeRS-SP68.3250.3761.1887.4134.4061.1977.1562.86–
+ Our KDHS + Uniform Sampling68.8650.7160.4287.3534.5261.8777.1562.98+0.12
+ Our KDHS + Our Stratified Sampling70.0050.5460.8387.4934.9062.1377.1963.30+0.44
200×DeRS-SP67.0849.5860.2486.8232.1059.9076.5361.75–
+ Our KDHS + Uniform Sampling68.2250.0360.4487.7031.7661.9276.8262.41+0.66
+ Our KDHS + Our Stratified Sampling69.2651.1060.9587.6532.8061.9276.7362.92+1.17

Table R3. Cumulative ablation on MoE-LLaVA-Qwen, set up as in Table R1.

RatioConfigurationSQA$^{\mathrm{I}}$VQA$^{\mathrm{T}}$GQAPOPEMM-VMMB$^{\mathrm{EN}}$VQA$^{\mathrm{v2}}$AvgGain
1×Uncompressed63.1647.7461.5087.8125.5054.7076.1859.51–
100×DeRS-SP57.9643.1358.6387.4624.8049.3073.7956.44–
+ Our KDHS + Uniform Sampling61.2843.5758.8587.1827.1150.8674.0857.56+1.12
+ Our KDHS + Our Stratified Sampling62.3745.1258.9687.8925.8048.7374.4957.62+1.18
200×DeRS-SP53.0035.2254.5285.9924.6035.0169.3651.10–
+ Our KDHS + Uniform Sampling57.7142.1557.3987.4226.4744.0572.9755.45+4.35
+ Our KDHS + Our Stratified Sampling59.9942.6757.6987.6425.8048.1672.3356.33+5.23
AuthorsOpenReview ↗

Response to Reviewer 1gd2 (2/6)

[W1Q1] Unclear incremental mechanism (continued)

Table R4. Three independent mask seeds on XFT at 200×, with the configurations of Table R1. Mean rows give mean$_{\pm\mathrm{std}}$.

ConfigurationSeedHEvalHEval+MBPPMBPP+Avg
Our KDHS + Uniform Sampling059.7656.1060.6552.1357.16
159.7656.1060.6550.3856.72
260.3757.9360.4049.8757.14
Mean59.96±0.3559.96_{\pm 0.35}56.71±1.0656.71_{\pm 1.06}60.57±0.1460.57_{\pm 0.14}50.79±1.1950.79_{\pm 1.19}57.01±0.2557.01_{\pm 0.25}
Our KDHS + Our Stratified Sampling061.5957.9362.1652.3858.51
162.8059.7661.6551.1358.84
259.7657.9363.4152.1358.31
Mean61.38±1.5361.38_{\pm 1.53}58.54±1.0658.54_{\pm 1.06}62.41±0.9162.41_{\pm 0.91}51.88±0.6651.88_{\pm 0.66}58.55±0.2758.55_{\pm 0.27}
Gain from stratified sampling+1.42+1.83+1.84+1.09+1.54

[W2Q2] Insufficient functional explanation

UltraSparse preserves complementary expert capabilities at the same 100× compression budget. On both MoE-LLaVA backbones, it reaches the highest mean over science-question answering and compositional visual reasoning among the three retention strategies (Table R5, from the allocation sweep in Tables S1 and S2).

Table R5. Capabilities kept by each branch at 100×, from Tables S1 and S2. SQA$^{\mathrm{I}}$ tests science knowledge, GQA tests compositional visual reasoning, and Mean averages the two. Pure Coverage uses its best bin count.

MoE-LLaVA-QwenMoE-LLaVA-Phi
ConfigurationSQA$^{\mathrm{I}}$GQAMeanSQA^{\mathrm{I}}\GQAMean
Pure Coverage59.7459.3159.5368.4261.5464.98
Pure Top61.9256.8259.3769.7159.1264.42
UltraSparse (Top + Coverage)62.3758.9660.6770.0060.8365.42

1. UltraSparse retains the prominent updates without attenuating the expert deltas. Expert deltas encode the task-specific adaptation learned in fine-tuning [1]. On MoE-LLaVA, the largest 1% of entries hold under 9% of the delta energy ∥Δ∥F2\lVert\boldsymbol{\Delta}\rVert_F^2. Pure Top minimizes the element-wise reconstruction error, yet at 100× it discards the remaining entries without rescaling, removes over 91% of this energy and thereby shrinks every expert update toward the shared base. UltraSparse keeps the largest updates exactly and adds rescaled Coverage samples across the remaining magnitude range, which preserves the overall scale of the delta. Figure 2 shows the resulting alignment of the magnitude CDF and normalized spectrum before and after compression.

2. The two pure branches are complementary while the hybrid method improves performance. Pure Top leads pure Coverage on SQA$^{\mathrm{I}}$ by +2.18 on MoE-LLaVA-Qwen and +1.29 on MoE-LLaVA-Phi, while pure Coverage leads pure Top on GQA by +2.49 and +2.42. Combining the branches raises GQA over pure Top by +2.14 and +1.71, while also raising SQA$^{\mathrm{I}}$ by +0.45 and +0.29.

3. Delta differentiation guides the balance between the two branches. We interpret the separation between prominent and background delta entries, measured by the dispersion ratio Std(∣Δ∣)/Mean(∣Δ∣)\mathrm{Std}(|\boldsymbol{\Delta}|)/\mathrm{Mean}(|\boldsymbol{\Delta}|), as a weight-space signature of fine-tuning depth, since longer or harder fine-tuning accumulates sustained updates in task-relevant directions. A less differentiated delta spreads adaptation broadly and favors Coverage, while a more differentiated delta concentrates prominent updates that Top preserves exactly. The selected splits of the evaluated models follow this pattern (Table R6 in our response to Q3).

The revision adds Table R5 and this analysis.

[W3Q3] Underexplained expert heterogeneity

Two characteristics predict compression behavior on the evaluated models. Delta magnitude, absolute and relative to the shared base, predicts compression sensitivity, and the differentiation of delta magnitudes predicts the Top/Coverage split. Within a model, the admissible common rate is set by the expert matrix whose output is most perturbed on its routed inputs. Here, a common rate means the same retained fraction for every expert matrix in a model, while the Top set and Coverage bins are still built separately for each matrix. Table R6 profiles six evaluated models, 48.7 billion delta entries in total.

AuthorsOpenReview ↗

Response to Reviewer 1gd2 (3/6)

[W3Q3] Underexplained expert heterogeneity (continued)

Table R6. Expert-delta statistics and the share of the uncompressed score retained at 100×, sorted by the mean entry magnitude. Std/Mean is the dispersion ratio of ∣Δ∣|\boldsymbol{\Delta}|, which equals 0.76 for a folded Gaussian and 1.00 for a Laplacian, and Top share is the Top/Coverage split selected at 100×.

Delta magnitude $\vert \boldsymbol{\Delta}\vert $Retained at 100×
ModelMean (10−410^{-4})Std/MeanMax/MeanTop shareDeRS-SPUltraSparse
Med-MoE-StableLM4.150.7510.640%100.2%100.9%
Med-MoE-Phi4.150.8418.840%99.5%101.3%
MoE-LLaVA-Phi6.100.7819.250%99.5%100.2%
MoE-LLaVA-Qwen6.880.7710.430%94.8%96.8%
XFT15.590.92144.380%91.8%99.6%
Uni-MoE30.860.9770.390%9.7%81.8%

1. Delta magnitude sets compression sensitivity, and UltraSparse widens the usable common rate. The variance of drop-and-rescale grows quadratically with the entry magnitude. As the mean entry magnitude increases across the models in Table R6, DeRS-SP retention at 100× drops from 100.2% to 9.7%. At the same rate, UltraSparse retains 96.8%–101.3% on five models and 81.8% on Uni-MoE, over 8× the DeRS-SP score. Relative to the shared base, the Med-MoE and MoE-LLaVA deltas have relative delta norms r=∥Δ∥F/∥Wbase∥Fr = \lVert\boldsymbol{\Delta}\rVert_F/\lVert\mathbf{W}^{\text{base}}\rVert_F of 0.019–0.046, and UltraSparse keeps 95.4%–99% of their scores at 1000× (Tables 1 and 3). XFT keeps its always-active universal expert intact and retains 98.6% (Table 4), and Uni-MoE retains 79% where DeRS-SP scores zero (Table 2). Aggressive common rates therefore suit models whose intact shared component carries substantial capability, and the usable rate depends on both the model and the compression operator.

2. Delta differentiation sets the Top/Coverage split. As discussed under Q2, deeper fine-tuning separates prominent entries from the background more sharply. MoE-LLaVA and Med-MoE have dispersion ratios of 0.75–0.84 and maximum-to-mean ratios of 10.4–19.2, with 50%–70% of the budget assigned to Coverage for broadly distributed adaptation. XFT and Uni-MoE have dispersion ratios of 0.92–0.97 and maximum-to-mean ratios of 70.3–144.3, with 80%–90% assigned to Top for exact retention of the more prominent updates.

3. Within a model, the output perturbation on routed inputs gives the condition for a common rate. For a linear projection in expert e, let We=Wbase+Δe\mathbf{W}_e=\mathbf{W}^{\text{base}}+\boldsymbol{\Delta}_e and Ee(k)=Δ^e(k)−Δe\mathbf{E}_e(k)=\widehat{\boldsymbol{\Delta}}_e(k)-\boldsymbol{\Delta}_e, where k is the common retained fraction. On the same inputs x\mathbf{x} routed to that expert in the uncompressed model, let Σe=E[xxT∣e is routed]\boldsymbol{\Sigma}_e=\mathbb{E}[\mathbf{x}\mathbf{x}^{\mathsf T}\mid e\text{ is routed}] be the input second-moment matrix. The relative mean-squared output perturbation is

De(k)=E[∥Ee(k)x∥22∣e is routed]E[∥Wex∥22∣e is routed]=tr⁡(Ee(k)ΣeEe(k)T)tr⁡(WeΣeWeT).\mathcal{D}_e(k) =\frac{\mathbb{E}[\lVert \mathbf{E}_e(k)\mathbf{x}\rVert_2^2\mid e\text{ is routed}]}{\mathbb{E}[\lVert \mathbf{W}_e\mathbf{x}\rVert_2^2\mid e\text{ is routed}]} =\frac{\operatorname{tr}(\mathbf{E}_e(k)\boldsymbol{\Sigma}_e \mathbf{E}_e(k)^{\mathsf T})}{\operatorname{tr}(\mathbf{W}_e\boldsymbol{\Sigma}_e \mathbf{W}_e^{\mathsf T})}.

For a chosen local fidelity tolerance τe\tau_e, a common k preserves each expert when De(k)≤τe2\mathcal{D}_e(k)\leq\tau_e^2 across the expert matrices, so the most restrictive expert or module sets the admissible common budget. UltraSparse builds the Top set and bins from each matrix's own magnitudes, which makes its relative reconstruction error insensitive to the matrix's overall scale. Within MultiPL-MoE and Uni-MoE, delta norms differ by up to 3.6× while relative weight-reconstruction errors at 100× differ by at most 1.1%. At subsequent routers, a sufficient condition for preserving the top-m selected set is that every routing logit changes by less than half the original gap between the mth and (m+1)th logits. The end-to-end task results evaluate the complete compressed model, including these routing changes.

We thank the reviewer for suggesting expert-aware budgets. Such budgets plug directly into the per-matrix operator, and tying them to expert contribution, delta energy or routed inputs could raise these results further, which we will explore in future work. The revision adds Table R6 and this analysis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.