- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates retraining-free compression of vision-language Mixture-of-Experts (VLM-MoE) models and proposes SPARE, a framework that preserves the functional expressiveness of the expert routing space. The authors observe that visual tokens exhibit more diffuse expert routing patterns than text tokens, making conventional frequency-based pruning and parameter merging less effective. To address this issue, SPARE constructs a routing-weighted Gram matrix from expert outputs collected on a calibration dataset. It then selects a well-conditioned subset of experts through greedy pivoting on the principal eigenvectors and derives a closed-form routing remapping matrix to redistribute the contributions of discarded experts. The method does not require additional training or modifications to the retained expert parameters. Experiments on Kimi-VL-A3B and Qwen3-VL-30B-A3B compare SPARE with several expert pruning and merging methods across multiple compression budgets and multimodal benchmarks. The results show that SPARE achieves the highest overall performance among the evaluated methods, with more noticeable advantages under aggressive compression. Additional ablation studies and calibration robustness experiments provide further support for the proposed framework.
Critical strengths
The main strength of this paper is its routing-space-based compression framework, which considers the relationships among experts rather than relying solely on individual importance scores. SPARE combines expert subset selection with closed-form routing reconstruction without requiring retraining. The experiments demonstrate promising performance, particularly under aggressive compression, and the additional ablation and robustness analyses further support the effectiveness of the proposed method.
Critical weaknesses
Although SPARE achieves strong overall performance, its advantages over existing methods are relatively limited at moderate compression ratios. For example, in Table 1, at 75% expert retention on Kimi-VL, SPARE and Sub-MoE achieve the same average performance of 97.27%. In addition, the theoretical reconstruction is based on a linear subspace formulation, while actual inference involves Top-k selection and normalization, which may affect reconstruction accuracy. The method also requires evaluating all experts during calibration, introducing additional offline computational costs that deserve further discussion.
Questions for authors
-
SPARE achieves strong overall performance, but its improvements over existing methods are relatively small at moderate compression ratios. Could the authors clarify under which compression ratios and model architectures SPARE provides the most consistent advantages? How stable are these improvements across different calibration samples?
-
The routing reconstruction is derived from a linear subspace formulation, while actual inference involves Top-k selection and normalization. Could the authors clarify how these operations affect reconstruction accuracy and whether the theoretical properties remain valid during inference?
-
Since constructing the Gram matrix requires evaluating all experts during calibration, could the authors provide a more detailed comparison of the total offline compression costs against existing methods and discuss the practical trade-offs?
3 comments
Response to Reviewer Gq5R (Part 2/2)
Since constructing the Gram matrix requires evaluating all experts during calibration, could the authors provide a more detailed comparison of the total offline compression costs against existing methods and discuss the practical trade-offs?
We measured uncached compression with 128 calibration examples and 50% expert retention, using one A100 80GB per run. REAP computes routed-token saliency without constructing SPARE's Gram matrix.
| Model | Method | Profiling/grouping (s) | Compression (s) | Total (s) | Peak allocated (GiB) |
|---|---|---|---|---|---|
| Kimi-VL | SPARE | 165.02 | 0.58 | 205.59 | 33.42 |
| Kimi-VL | REAP | 139.56 | 0.02 | 179.03 | 31.66 |
| Kimi-VL | Hierarchical merge | 251.70 | 0.40 | 289.80 | 32.93 |
| Qwen3-VL | SPARE | 679.02 | 1.80 | 732.46 | 59.74 |
| Qwen3-VL | REAP | 348.20 | 0.05 | 396.57 | 59.39 |
| Qwen3-VL | Hierarchical merge | 2036.21 | 0.91 | 2086.37 | 59.76 |
Totals additionally include model loading and data preparation, and exclude generation and checkpoint serialization. For merging, grouping is included in the profiling/grouping column. These are single measurements of the implemented pipelines; the merge implementation includes Python clustering overhead, and concurrent work on other GPUs may introduce shared-host I/O variation.
SPARE takes approximately 3.4 minutes on Kimi and 12.2 minutes on Qwen, or 1.15× and 1.85× REAP's total time. All-expert profiling dominates; the spectral selection and remapping stage takes under two seconds. The practical trade-off is therefore a larger one-time calibration cost for improved aggregate retention at tight budgets. Once collected, the Gram matrices can be reused across compression budgets. We have added the full offline-cost comparison and the cross-budget reuse discussion to Appendix C.
We thank the reviewer again for the constructive feedback and hope these responses address the remaining concerns. We are happy to answer any further questions.
Thank you again for your time and thoughtful review. As the discussion period is approaching its end, we would like to follow up on our responses and the revised manuscript. We would appreciate hearing whether the clarifications and additional results address your concerns. If any questions remain, we would be happy to discuss them further.
Response to Reviewer Gq5R (Part 1/2)
We thank Reviewer Gq5R for the careful review and constructive feedback. We address the concerns regarding gains across compression regimes, sparse routing, and offline calibration cost below.
SPARE's advantage is most consistent in aggregate retention under tight budgets and persists with further compression. The following values are Table 1's aggregate retained performance, expressed as a percentage of each dense model's aggregate score.
Kimi-VL ties the strongest baseline at 75% retention, while Qwen3-VL already leads by 2.00 points. At 50%, SPARE has the highest aggregate score on both architectures, with margins of 1.65 and 3.61 points, respectively.
The advantage persists at 37.5% retention. In Section 5.3, SPARE retains 80.36% of dense performance on Kimi-VL versus REAP's 78.29%, and 85.04% on Qwen3-VL versus 82.66%—margins of 2.07 and 2.38 points. Thus, the benefit holds across both tested architectures as the expert budget tightens beyond 50% retention.
SPARE is robust to calibration-set size and source. At 50% retention, varying the calibration size over 32, 128, and 512 examples changes the aggregate score by only 0.88 points on Kimi and 0.43 on Qwen. A separate corpus sweep at 75% retention, using GQA, COCO, and TextVQA, spans just 0.33 and 1.68 points, respectively (Appendix D.3). These results demonstrate stable performance across calibration sizes and data sources.
We thank the reviewer for raising this distinction. The linear subspace formulation describes an internal reconstruction, not a linear model of the entire inference function. Let ΦI(x,a) denote the retained-expert inference function, including sparse selection and output aggregation, with input x and routing coefficients a. SPARE's deployed computation can then be written as
y~(x)=ΦI(x,M⊤g(x)),
where g(x) denotes the routing weights and M is the subspace-derived reconstruction matrix. The linear remapping is therefore embedded within the inference function; the nonlinear routing operations are neither omitted nor approximated as linear.
Reconstructing the internal linear representation allows retained experts to approximate contributions from discarded experts, rather than simply losing those contributions through pruning. The resulting closed-form remapping compensates for expert removal without linearizing the full inference function or retraining expert parameters. It is used within ΦI while preserving the original number of active experts per token. Our end-to-end results and the controlled comparison below support the practical effectiveness of this reconstruction.
To isolate the effect of top-k, we tested the output-matched form with and without truncation at 50% retention, keeping the expert subset, calibration data, and evaluation fixed. Local errors use 32 held-out GQA examples at identical original-model layer inputs; task scores are paired on fixed 500-example subsets and are not directly comparable with Table 1.
Top-k leaves the pooled layer error essentially unchanged on Kimi-VL (0.072) and raises it from 0.27 to 0.32 on Qwen3-VL, with no consistent change in task scores. These results provide empirical support for the reconstruction's usefulness under sparse inference, beyond the linear-mixing setting.