acceptodds
Under review as a conference paper at ICLR 2027

Reconstructing the Routing Space: Retraining-Free Compression of Vision-Language Mixture-of-Experts

Abstract

Vision-language Mixture-of-Experts models (VLM-MoEs) hold most of their parameters in expert weights, yet their expert pools are hard to compress because visual tokens use experts far more diffusely than text tokens. Our experiments show that covering of visual routing mass takes up to as many experts as for text, and that at expert retention frequency-based pruning keeps only – of dense performance. This exposes a limitation of existing compressors: they rank experts alone or fuse parameters, ignoring the function expressiveness of the routing space. We propose \method, which selects a well-conditioned expert subset by pivoting on the eigenbasis of a routing-aware expert-output Gram matrix and, from the same basis, reconstructs the discarded routing coordinates in closed form. The procedure is one-shot and retraining-free, with no learned parameters. On Kimi-VL-A3B and Qwen3-VL-30B-A3B, \method delivers the strongest overall performance across compression budgets, and its advantage widens as compression becomes more aggressive, reaching up to points over the strongest baseline at expert retention.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Mixture-of-Experts, Vision-Language Models, Model Compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Gq5ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates retraining-free compression of vision-language Mixture-of-Experts (VLM-MoE) models and proposes SPARE, a framework that preserves the functional expressiveness of the expert routing space. The authors observe that visual tokens exhibit more diffuse expert routing patterns than text tokens, making conventional frequency-based pruning and parameter merging less effective. To address this issue, SPARE constructs a routing-weighted Gram matrix from expert outputs collected on a calibration dataset. It then selects a well-conditioned subset of experts through greedy pivoting on the principal eigenvectors and derives a closed-form routing remapping matrix to redistribute the contributions of discarded experts. The method does not require additional training or modifications to the retained expert parameters. Experiments on Kimi-VL-A3B and Qwen3-VL-30B-A3B compare SPARE with several expert pruning and merging methods across multiple compression budgets and multimodal benchmarks. The results show that SPARE achieves the highest overall performance among the evaluated methods, with more noticeable advantages under aggressive compression. Additional ablation studies and calibration robustness experiments provide further support for the proposed framework.

Critical strengths

The main strength of this paper is its routing-space-based compression framework, which considers the relationships among experts rather than relying solely on individual importance scores. SPARE combines expert subset selection with closed-form routing reconstruction without requiring retraining. The experiments demonstrate promising performance, particularly under aggressive compression, and the additional ablation and robustness analyses further support the effectiveness of the proposed method.

Critical weaknesses

Although SPARE achieves strong overall performance, its advantages over existing methods are relatively limited at moderate compression ratios. For example, in Table 1, at 75% expert retention on Kimi-VL, SPARE and Sub-MoE achieve the same average performance of 97.27%. In addition, the theoretical reconstruction is based on a linear subspace formulation, while actual inference involves Top-k selection and normalization, which may affect reconstruction accuracy. The method also requires evaluating all experts during calibration, introducing additional offline computational costs that deserve further discussion.

Questions for authors
  1. SPARE achieves strong overall performance, but its improvements over existing methods are relatively small at moderate compression ratios. Could the authors clarify under which compression ratios and model architectures SPARE provides the most consistent advantages? How stable are these improvements across different calibration samples?

  2. The routing reconstruction is derived from a linear subspace formulation, while actual inference involves Top-k selection and normalization. Could the authors clarify how these operations affect reconstruction accuracy and whether the theoretical properties remain valid during inference?

  3. Since constructing the Gram matrix requires evaluating all experts during calibration, could the authors provide a more detailed comparison of the total offline compression costs against existing methods and discuss the practical trade-offs?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Gq5R (Part 1/2)

We thank Reviewer Gq5R for the careful review and constructive feedback. We address the concerns regarding gains across compression regimes, sparse routing, and offline calibration cost below.

SPARE achieves strong overall performance, but its improvements over existing methods are relatively small at moderate compression ratios. Could the authors clarify under which compression ratios and model architectures SPARE provides the most consistent advantages? How stable are these improvements across different calibration samples?

SPARE's advantage is most consistent in aggregate retention under tight budgets and persists with further compression. The following values are Table 1's aggregate retained performance, expressed as a percentage of each dense model's aggregate score.

ModelExperts retainedStrongest baselineSPAREMargin (points)
Kimi-VL75%97.2797.270.00
Qwen3-VL75%96.5398.53+2.00
Kimi-VL50%87.6489.29+1.65
Qwen3-VL50%87.6391.24+3.61

Kimi-VL ties the strongest baseline at 75% retention, while Qwen3-VL already leads by 2.00 points. At 50%, SPARE has the highest aggregate score on both architectures, with margins of 1.65 and 3.61 points, respectively.

The advantage persists at 37.5% retention. In Section 5.3, SPARE retains 80.36% of dense performance on Kimi-VL versus REAP's 78.29%, and 85.04% on Qwen3-VL versus 82.66%—margins of 2.07 and 2.38 points. Thus, the benefit holds across both tested architectures as the expert budget tightens beyond 50% retention.

SPARE is robust to calibration-set size and source. At 50% retention, varying the calibration size over 32, 128, and 512 examples changes the aggregate score by only 0.88 points on Kimi and 0.43 on Qwen. A separate corpus sweep at 75% retention, using GQA, COCO, and TextVQA, spans just 0.33 and 1.68 points, respectively (Appendix D.3). These results demonstrate stable performance across calibration sizes and data sources.

The routing reconstruction is derived from a linear subspace formulation, while actual inference involves Top-k selection and normalization. Could the authors clarify how these operations affect reconstruction accuracy and whether the theoretical properties remain valid during inference?

We thank the reviewer for raising this distinction. The linear subspace formulation describes an internal reconstruction, not a linear model of the entire inference function. Let ΦI(x,a)\Phi_I(x,a) denote the retained-expert inference function, including sparse selection and output aggregation, with input xx and routing coefficients aa. SPARE's deployed computation can then be written as

y~(x)=ΦI ⁣(x,M⊤g(x)),\tilde y(x)=\Phi_I\!\left(x,M^\top g(x)\right),

where g(x)g(x) denotes the routing weights and MM is the subspace-derived reconstruction matrix. The linear remapping is therefore embedded within the inference function; the nonlinear routing operations are neither omitted nor approximated as linear.

Reconstructing the internal linear representation allows retained experts to approximate contributions from discarded experts, rather than simply losing those contributions through pruning. The resulting closed-form remapping compensates for expert removal without linearizing the full inference function or retraining expert parameters. It is used within ΦI\Phi_I while preserving the original number of active experts per token. Our end-to-end results and the controlled comparison below support the practical effectiveness of this reconstruction.

To isolate the effect of top-k, we tested the output-matched form with and without truncation at 50% retention, keeping the expert subset, calibration data, and evaluation fixed. Local errors use 32 held-out GQA examples at identical original-model layer inputs; task scores are paired on fixed 500-example subsets and are not directly comparable with Table 1.

ModelOutput-matched routingPooled relative L2Mean layer relative L2TextVQAChartQA
Kimi-VLLinear mixing0.07190.306169.0869.00
Kimi-VL+ Top-k0.07170.306068.9469.40
Qwen3-VLLinear mixing0.27110.396282.1270.60
Qwen3-VL+ Top-k0.31590.395181.9072.20

Top-k leaves the pooled layer error essentially unchanged on Kimi-VL (0.072) and raises it from 0.27 to 0.32 on Qwen3-VL, with no consistent change in task scores. These results provide empirical support for the reconstruction's usefulness under sparse inference, beyond the linear-mixing setting.

AuthorsOpenReview ↗

Response to Reviewer Gq5R (Part 2/2)

Since constructing the Gram matrix requires evaluating all experts during calibration, could the authors provide a more detailed comparison of the total offline compression costs against existing methods and discuss the practical trade-offs?

We measured uncached compression with 128 calibration examples and 50% expert retention, using one A100 80GB per run. REAP computes routed-token saliency without constructing SPARE's Gram matrix.

ModelMethodProfiling/grouping (s)Compression (s)Total (s)Peak allocated (GiB)
Kimi-VLSPARE165.020.58205.5933.42
Kimi-VLREAP139.560.02179.0331.66
Kimi-VLHierarchical merge251.700.40289.8032.93
Qwen3-VLSPARE679.021.80732.4659.74
Qwen3-VLREAP348.200.05396.5759.39
Qwen3-VLHierarchical merge2036.210.912086.3759.76

Totals additionally include model loading and data preparation, and exclude generation and checkpoint serialization. For merging, grouping is included in the profiling/grouping column. These are single measurements of the implemented pipelines; the merge implementation includes Python clustering overhead, and concurrent work on other GPUs may introduce shared-host I/O variation.

SPARE takes approximately 3.4 minutes on Kimi and 12.2 minutes on Qwen, or 1.15× and 1.85× REAP's total time. All-expert profiling dominates; the spectral selection and remapping stage takes under two seconds. The practical trade-off is therefore a larger one-time calibration cost for improved aggregate retention at tight budgets. Once collected, the Gram matrices can be reused across compression budgets. We have added the full offline-cost comparison and the cross-budget reuse discussion to Appendix C.

We thank the reviewer again for the constructive feedback and hope these responses address the remaining concerns. We are happy to answer any further questions.

Thank you again for your time and thoughtful review. As the discussion period is approaching its end, we would like to follow up on our responses and the revised manuscript. We would appreciate hearing whether the clarifications and additional results address your concerns. If any questions remain, we would be happy to discuss them further.

Reviewer_Ne8ROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper study the routing differences between vision tokens and text tokens in VLM-MoE, proposing a novel retraining-free compression techniques. Experiment results demonstrate the effectiveness of the insights and validate author's findings.

Critical strengths

This work is well motivated with concise evidences showing routing of vision tokens are more even than text tokens. Insights and designs are clear. Overall, this paper is well presented and convincing. It contributes novel knowledge to the compression of VLM.

Critical weaknesses

This is no significant critical weakness in this paper. But I do have a concern regarding experiment baselines, where authors seem not to be aware of related works in LLM decomposition, particularly LLM-MoE decomposition, e.g., Lujun's another work: SVD-MoE. Authors are encouraged to add this work as baseline and some discussions to demonstrate how differences between VLM-Moe and LLM-MoE affects the compression performance.

Questions for authors

none

1 comment
AuthorsOpenReview ↗

Response to Reviewer Ne8R

We thank Reviewer Ne8R for the encouraging assessment and for pointing us to decomposition-based compression, which is indeed closely related to our work.

… authors seem not to be aware of related works in LLM decomposition, particularly LLM-MoE decomposition, e.g., … SVD-MoE. Authors are encouraged to add this work as baseline and some discussions to demonstrate how differences between VLM-MoE and LLM-MoE affects the compression performance.

We have added a short discussion of decomposition-based compression at the end of the experimental section (Section 5.6, “Discussion”), including SVD-LLM [2] and MoE-SVD [1] and how they relate to SPARE. MoE-SVD reduces redundancy within the expert weights through low-rank factors shared across experts, whereas SPARE reduces the number of experts and reconstructs routing over the retained ones.

We also considered adding MoE-SVD as a baseline. As its released implementation is currently not accessible to us, it is hard to make a faithful comparison within the rebuttal period.

Regarding the differences between VLM-MoEs and LLM-MoEs, our analyses show that visual tokens route more diffusely and span a higher-dimensional expert-output subspace than text tokens in the studied VLMs (Figs. 1–2). For compression, these findings motivate preserving joint expert coverage: individual importance alone does not capture whether the retained experts can reproduce the complementary contributions of removed ones. This is the modality-dependent structure that SPARE's joint selection addresses. Additionally, we agree with the reviewer on examining decomposition-based compression from this perspective. Building on these findings, exploring how visual and textual activation structure can guide calibration and cross-expert factor sharing is a promising direction.

We thank the reviewer again for the helpful suggestion and are happy to answer any further questions.

References

[1] Wei Li, Lujun Li, Hao Gu, You-Liang Huang, Mark G. Lee, Shengjie Sun, Wei Xue, and Yike Guo. MoE-SVD: Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition. ICML 2025.

[2] Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD-LLM: Truncation-Aware Singular Value Decomposition for Large Language Model Compression. arXiv:2403.07378, 2024.

Reviewer_yKjMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes SPARE, a retraining-free compression method for VLM-MoEs that shrinks the expert pool while preserving model performance. SPARE first constructs a routing-weighted expert-output Gram matrix to model inter-expert relationships and selects a well-conditioned expert subset by greedy pivoting on the top-k eigenvectors. It then derives a closed-form router remapping from the same pivoted core to redistribute the routing mass of discarded experts. In addition, routing-weighted importance is incorporated into the Gram matrix so that geometric independence and routing importance combine in one spectral object. Experiments on Kimi-VL-A3B and Qwen3-VL-30B-A3B across six benchmarks show improvements over the strongest baseline at 50% expert retention.

Critical strengths
  1. The method unifies expert selection and routing reconstruction through the same principal subspace, enabling one-shot compression without retraining or learned parameters.
  2. SPARE achieves consistent improvements in aggregate performance across two VLM-MoEs, with larger gains under aggressive compression.
Critical weaknesses
  1. The principal expert subspace is constructed from importance-scaled expert outputs, but Eq. (5) directly remaps the original routing weights without correcting for this scaling mismatch at inference. Appendix A establishes the linear interpolation condition but doesn't show that the remapping preserves the original MoE layer output after nonlinear top-k truncation and renormalization.
  2. The observation that visual tokens route more diffusely than text tokens has already been noted in prior work (e.g., Xu et al., 2026, VisMMOE), and its causal link to SPARE's performance gains remains under-substantiated—especially since SPARE's improvements over REAP at 50% retention are concentrated on specific tasks (e.g., ChartQA, Table 1) while trailing or tying on several other visual benchmarks. Moreover, the expert-output profiling, router-weighted importance, and DEIM-based pivoting and interpolation directly adapt existing formulations, leaving the methodological novelty beyond their combination insufficiently justified.
Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer yKjM (Part 1/2)

We thank Reviewer yKjM for the positive assessment and the constructive comments. We address each point below.

Eq. (5) directly remaps the original routing weights without correcting for this scaling mismatch at inference.

We thank the reviewer for this careful observation. Eq. (5) indeed remaps the original gate gg. The mismatch affects only the gate weight that discarded experts pass to the retained ones. Each retained expert keeps its own gate value with coefficient one, since M(I,:)=IkM(I,:)=I_k.

Because the layer output is y=(W−1g)⊤O~y=(W^{-1}g)^\top\tilde O, correcting the mismatch gives the output-matched gate g^=WIM⊤W−1g\hat g=W_IM^\top W^{-1}g, which reproduces the layer output before top-k exactly when each token's weighted expert outputs lie in the principal subspace. M⊤gM^\top g is exact under the same condition on the unweighted outputs. Top-k is addressed in the next part.

In practice, SPARE applies M⊤gM^\top g (Section 4.3). Relative to g^\hat g, it scales the handover from a discarded expert ii to a retained expert jj by si/sj\sqrt{s_i/s_j}, which is below one whenever the receiving expert is the more important of the two. M⊤gM^\top g thus keeps the retained experts' gates closer to their original values, at the cost of reconstructing less of rarely routed experts.

To examine whether this choice costs accuracy, we compared both forms at 50% retention with the expert subset, calibration data, and evaluation fixed. Scores are paired on fixed 500-example subsets, so they are not directly comparable with Table 1.

Routing (+ Top-k)Kimi TextVQAKimi ChartQAQwen TextVQAQwen ChartQA
M⊤gM^\top g (Eq. (5))68.3868.6082.2072.60
g^\hat g (output-matched)68.9469.4081.9072.20

Across the four scores, the differences (g^\hat g minus M⊤gM^\top g) range from −0.40 to +0.80 points, with no consistent direction across models. In short, the mismatch affects only the weight passed from discarded experts, and keeping Eq. (5) shows no consistent cost in these tests.

We have added discussions of both forms to Section 4.3 (Lines 320–328) and Appendix A (Lines 679–701).

Appendix A … doesn't show that the remapping preserves the original MoE layer output after nonlinear top-k truncation and renormalization.

We agree with the reviewer that the guarantee of Appendix A holds for the remapping before truncation, and we are grateful for pointing this out.

Top-k is the original MoE's own selection step. SPARE keeps it with the same kk, so each token activates as many experts as before, and the selected weights are not renormalized. Truncation therefore adds only one error term, bounded by the gate mass that top-k drops.

Empirically, at 50% retention, top-k leaves the pooled layer error unchanged on Kimi-VL (0.072) and raises it from 0.27 to 0.32 on Qwen3-VL, with no consistent change in task scores.

We have amended the reconstruction statement in Section 4.3 (Lines 324–325, 334–335) to this scope, corrected the description of inference there (Lines 333–335), and added the truncation bound to Appendix A (Lines 702–717).

AuthorsOpenReview ↗

Response to Reviewer yKjM (Part 2/2)

The observation that visual tokens route more diffusely than text tokens has already been noted in prior work (e.g., … VisMMOE) …

We thank the reviewer for pointing us to VisMMOE [1]. As VisMMOE is a recent arXiv preprint, we would regard it as contemporaneous work, in line with the ICLR guidelines. We are happy to see that it also observes the diffuse routing of visual tokens, and we have cited it in Section 3.1 as additional evidence for this observation.

SPARE builds on this observation for compression. When routing is diffuse, the retained experts need to cover the output space jointly, in addition to being individually important. We have also standardized the manuscript terminology to distinguish routing events (expert-selection counts), gate mass (routing-weight sums), and Gram energy (spectral energy).

… its causal link to SPARE's performance gains remains under-substantiated … since SPARE's improvements over REAP at 50% retention are concentrated on specific tasks (e.g., ChartQA, Table 1) while trailing or tying on several other visual benchmarks.

We agree that the aggregate gain at 50% retention comes mainly from a few tasks, such as ChartQA and, on Qwen, MMStar, with ties elsewhere. Our claim is that diffuse routing makes scalar ranking unreliable in aggregate, not that it predicts uniform per-task gains.

A text-only control is consistent with this link. On the text-only Qwen3-30B-A3B, SPARE stays within 0.25 points of REAP at both 75% and 50% retention. On Qwen3-VL-30B-A3B, its margin over the strongest baseline instead grows from +2.00 to +3.61 points between the same budgets (Table 1). In short, the evidence supports the link at the level of aggregate retention under tight budgets, and we have stated the task-level variation explicitly in Section 5.2.

The expert-output profiling, router-weighted importance, and DEIM-based pivoting … directly adapt existing formulations, leaving the methodological novelty … insufficiently justified.

We thank the reviewer for the opportunity to clarify our contribution. We agree that the individual tools are standard, and SPARE uses them by design. Our contribution is to cast expert compression as interpolation of the router's coordinates in an importance-weighted expert-output subspace. Under this formulation, expert selection becomes DEIM point selection and router remapping becomes its oblique projection, so both follow from one pivoted core. Prior expert compression makes no such connection. Scoring-based pruning such as REAP ranks experts individually and leaves the router unchanged, and merging fuses parameters without reconstructing routing.

The remapping relies on the pivoted core. On the pivoted core, the remapping adds 1.7 points over plain restriction of the gate, whereas the same remapping on a core partly chosen by frequency collapses the model, even with the gate clipped and renormalized (Table 9, Appendix D.7). Across budgets, removing the remapping also lowers 46 of the 48 scores in Table 5. In short, the novelty lies in the formulation that connects these tools rather than in the tools themselves.

We thank the reviewer again for the constructive feedback and hope these responses address the remaining concerns. We are happy to answer any further questions.

References

[1] Cheng Xu, Xiaofeng Hou, Jiacheng Liu, and Chao Li. VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading. arXiv:2605.05899, 2026.

Thanks for the author's rebuttal response. I maintain my score.

Comments on the paper

1 comment

We sincerely thank Ne8R, yKjM, and Gq5R for their careful reviews and constructive feedback. During the rebuttal, we have revised the theoretical analysis, conducted additional controlled experiments and ablations, and provided detailed measurements of compression costs. These revisions clarify the connection between SPARE's formulation and implementation, further examine the role of unified expert selection and routing reconstruction, and provide additional results across compression budgets and calibration settings.

Revision Map. The table below summarizes the revisions corresponding to each reviewer's concerns.

ReviewerConcernRevisionLocation in Revised Manuscript
Ne8RW1Added decomposition-based compression discussion and modality-specific analysis.Sec. 3.1–3.2; Sec. 5.6, L492–496
yKjMW1Derived output-matched routing, established reconstruction conditions and top-k bounds, and added controlled experiments.Sec. 4.3, L320–335, Eq. 6; App. A, L679–717; App. D.9, Tables 10–11
yKjMW2Added VisMMOE citation, modality-specific analysis, text-only comparisons, and unified selection–remapping ablations.Sec. 3.1–3.2; Sec. 4.3; Sec. 5.2, Table 1; App. D.4, Table 7; App. D.7, Tables 8–9
Gq5RW1Extended compression-budget evaluation and calibration robustness analysis.Sec. 5.2–5.4, Tables 1–2; App. D.3, Table 6
Gq5RW2Added reconstruction analysis, top-k truncation bounds, and controlled sparse-inference experiments.Sec. 4.3, L320–335; App. A, L679–717; App. D.9, Table 11
Gq5RW3Added offline compression time, memory, and calibration-reuse analysis.Sec. 5.5, L480–488; App. C, L791–820, Table 3

Main Findings.

  • Theoretical analysis and sparse routing. We have refined the theoretical guarantees by explicitly specifying their validity conditions and deriving the additional reconstruction error introduced by top-k truncation. For the implemented routing g~=M⊤g\tilde{\mathbf{g}}=\mathbf{M}^{\top}\mathbf{g} and the output-matched form g^=WIM⊤W−1g\hat{\mathbf{g}}=\mathbf{W}_I\mathbf{M}^{\top}\mathbf{W}^{-1}\mathbf{g}, their respective exact reconstruction conditions are established before truncation. The additional error from top-k truncation is bounded by ∥y−FI⊤b∥2≤∥elin∥2+B(x)∥a−b∥1\|\mathbf{y}-\mathbf{F}_I^\top\mathbf{b}\|_2 \leq \|e_{\mathrm{lin}}\|_2+B(x)\|\mathbf{a}-\mathbf{b}\|_1. In controlled comparisons at 50% retention, the two routing formulations differ by −0.40 to +0.80 points across four task evaluations.

  • Unified expert selection and routing reconstruction. SPARE formulates expert compression through an importance-weighted expert-output subspace, using the same pivoted spectral basis for expert selection and closed-form routing reconstruction, M=UkUk(I,:)−1\mathbf{M}=\mathbf{U}_k\mathbf{U}_k(I,:)^{-1}. On Qwen3-VL at 50% retention, adding remapping increases the three-task aggregate retained performance from 90.34% to 91.99%. Across compression budgets, removing remapping lowers 46 of 48 benchmark scores.

  • Compression performance and calibration robustness. At mild compression levels, where most experts are retained, SPARE achieves performance comparable to the strongest baselines, while its advantages become more pronounced as the expert budget tightens. At 50% expert retention, SPARE achieves aggregate retained performance of 89.29% on Kimi-VL and 91.24% on Qwen3-VL, exceeding the strongest baselines by 1.65 and 3.61 percentage points, respectively. At 37.5% retention, its advantages over REAP remain at 2.07 and 2.38 points. Calibration experiments show that varying calibration size from 32 to 512 examples changes aggregate performance by 0.88 points on Kimi-VL and 0.43 points on Qwen3-VL. Across GQA, COCO, and TextVQA calibration sources, the corresponding variations at 75% retention are 0.33 and 1.68 points.

  • Offline compression efficiency. End-to-end offline compression measurements with 128 calibration examples on a single A100 GPU show that SPARE requires approximately 3.4 minutes for Kimi-VL and 12.2 minutes for Qwen3-VL, corresponding to 1.15× and 1.85× the total time of REAP. The spectral selection and routing reconstruction stages take less than two seconds. The collected Gram matrices and importance statistics can be reused across compression budgets, and the procedure requires no retraining or additional learned parameters.

The revised manuscript also includes an expanded discussion of related work, covering decomposition-based MoE compression and recent findings on visual-token routing, along with a clarification of the distinction between expert-level importance and joint expert-output coverage.

We appreciate the reviewers' feedback and the opportunity to improve the manuscript. Our individual responses provide the detailed derivations, experimental results, and explanations corresponding to each concern.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.