acceptodds
Under review as a conference paper at ICLR 2027

SLaMES: Sensitivity-Guided Layerwise and Modality-Aware Expert Skipping for Efficient Inference in MoE MLLMs

Abstract

Mixture-of-Experts (MoE) multimodal large language models (MLLMs) scale capacity efficiently, but fixed Top- routing activates the same number of experts per token, causing substantial inference overhead for visual-token-heavy inputs. Existing training-free expert-skipping methods use coarse, layer-shared, or skip-rate-invariant criteria, overlooking that skipping tolerance varies across layers, modalities, and ratios. We propose SLaMES, a training-free adaptive framework that exploits this heterogeneity through three components. First, *Layer–Modality Sensitivity Calibration* quantifies sensitivity across layers, modalities, and skipping ratios. Second, *Sensitivity-Guided Threshold Search* uses calibrated sensitivities and offline lookup to derive layer- and modality-specific thresholds with only a few full-model forward passes. Third, *Adaptive Routing-Based Skipping* allocates expert computation based on cumulative routing probability mass, with conditional Top-1 protection for the dominant expert. Across three MoE MLLMs and nine image and video benchmarks, SLaMES retains at least 99.25% of the original performance at a 50% expert-skipping ratio. At the most aggressive settings of 83.4%–87.5%, it retains 96.39%–98.14% of the original performance, consistently outperforming existing expert-skipping methods. Integrated into vLLM, these savings translate into prefill speedups of up to 2.2 on real hardware, demonstrating the efficiency of SLaMES.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Multimodal Large Language Models, Expert Skipping, Adaptive Routing, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2BVKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SLaMES is a mechanism to adaptively skip routed expert computation based on derived sensitivity, global target drop ratio and found dropping threshold. SLaMES achieves good performance on visual benchmarks compared to prior work.

Critical strengths
  • I find the paper well-written and at a high level easy to follow overall, with intuition and evidence backing their heuristics. I like how the authors have a systematic process to derive the skipping choice.
  • The authors have done thorough evaluation of the work on many dataset and various models, and ablation separates the contribution of each component.
Critical weaknesses
  • I think the scope of this work should be narrowed a bit: I don't see direct application of this to the whole family of MLLMs, for example, expert choice, MoMA-style/Mixture-of-Transformer-style MoE with single vision branch etc.
  • I have concern on the search heuristics in 4.2: the sensitivity value S_{l,m,r} w is derived per-layer, leaving other layers intact, but during search time, other layers may have already been skipped. It is unclear how accurate/representative this can be. Additionally, skipping the least sensitive layer towards p* may not be optimal, because it uses the current bin, not the next bin, and we don't know how much budget it releases, so it's unclear if we have any guarantee what is going to happen here.
  • I am not sure about how to pick p* - it's nice the author listed corresponding performance regression and latency improvements, but for practicality purposes, what p* should i choose if my goal is to limit regression by X% and improve latency by Y%? It sometimes also misses the budget - the actual skip is not as prescribed by p*.
  • I am interested to see how the calibration set's impact on the final results. Currently it appears to be all sourced from GQA?
Questions for authors

Please refer to the main weakness

2 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Q1. Scope of applicability. Thank you for pointing out that our scope should be more explicit. SLaMES targets multimodal MoE models with token-choice Top-K routing, where each token activates multiple experts with associated routing weights. This routing architecture is widely used in contemporary MoE models. The three models we evaluated use this design (K = 6 or 8). Our contribution in this setting is to measure how skipping tolerance varies with layer, modality, and skipping ratio, and to use those measurements to allocate expert computation without retraining. The architectures you mention require different treatment. MoMa uses expert-choice routing within modality-specific groups during training, so the number of experts assigned to a token can vary. Extending SLaMES to this setting would require accounting for those variable assignments when estimating the skipping budget. In a MoT-style model with a single visual feed-forward branch, a visual token has no lower-weight expert tail to skip. We will clarify these architectural assumptions in the revised manuscript.

Q2. Reliability of sensitivity guidance and the search heuristic. Thank you for raising these concerns. We agree that upstream skipping can change subsequent layers' inputs. The calibrated sensitivity serves as a rate-dependent priority for computation allocation, rather than an exact prediction of the loss under joint skipping. We conducted two additional experiments on Qwen3-VL-30B-A3B to test whether it remains useful in this setting.

Sensitivity with perturbed inputs. We recomputed the visual-token sensitivity at a 50% local skipping ratio, using inputs produced with upstream expert skipping. For each input condition, we compared the current layer's outputs with and without this visual-expert skipping, using identical inputs for the two branches. Search uses Sl,m=cl,mdl,mS_{l,m}=c_{l,m}d_{l,m}, where cl,mc_{l,m} is the calibrated global importance and is held fixed in this comparison. Below, all scores refer to the visual modality, whose subscript we omit for brevity. The dd and SS columns are in units of 10−310^{-3}; cc is reported without scaling. Results for a subset of layers are shown below.

LayerGlobal importance ccOriginal-input ddUpstream-skipped-input ddOriginal-input S=cdS=cdUpstream-skipped-input S=cdS=cd
20.0317500.1980.3830.0062960.012172
120.0309180.5656.2260.0174700.192503
220.0266741.14919.2960.0306420.514708
250.0233931.19120.1390.0278690.471103
400.0032772.29548.4400.0075210.158753
430.0023462.92761.2040.0068660.143600

Although the magnitudes increase, the ordering S2<S43<S40<S12<S25<S22S_{2}<S_{43}<S_{40}<S_{12}<S_{25}<S_{22} is preserved. This supports the usefulness of the calibrated priorities in the tested layers and upstream-skipping condition.

Search under joint skipping. We also replaced sensitivity-based layer priorities with random priorities while keeping the rest of the search unchanged. The table reports the lowest calibration KL within the first 16 full-model evaluations, each applying the candidate skipping configuration jointly across layers.

Search guidanceBest KL at 62.5% target ↓Best KL at 75% target ↓
Random, seed 120.0098310.012976
Random, seed 12340.0094180.012589
SLaMES0.0065800.010310

SLaMES reduces the best KL by 31.6% and 19.3%, respectively, relative to the mean of the two random runs' best values, while achieving slightly higher measured skipping ratios. This comparison directly tests the contribution of sensitivity guidance under the same evaluation budget.

Current-bin selection and budget updates. The current-bin score determines which layer to adjust. For each candidate threshold step, the offline lookup table supplies the estimated skipped-expert count at the new threshold, so its budget effect is explicitly computed. Visual restoration stops and reselects the layer when its sensitivity bin changes, and rejects a step if the estimated global skipping ratio would fall below the target. Text skipping advances by one threshold step, after which its ratio and sensitivity bin are updated. Thus, priorities change as the allocation changes. However, the current-bin score is not a next-step marginal-loss estimate, and we do not claim that it guarantees an optimal update or a globally optimal configuration. Full-model KL evaluations select the best configuration among the visited frontier candidates. The experiments above provide empirical support for this efficient heuristic.

AuthorsOpenReview ↗

Rebuttal Part 2

Q3. Choosing a target skipping ratio for deployment. Thank you for this practical question. SLaMES's fast threshold search makes it feasible to try several target ratios and select a configuration that meets the desired accuracy and latency requirements on representative validation inputs. The calibrated sensitivities and routing statistics can be reused across these searches. Section 5.3 reports an 11.5–14.3× reduction in threshold-search time relative to MoDES. In practice, one can lower the target if accuracy loss exceeds the allowed X%, or raise it if the desired Y% latency reduction is not reached, then validate the new configuration. Selection should use measured accuracy, latency, and realized skipping, accounting for deviations from the search target.

We report the target, calibration-search, and test-time ratios for SLaMES, together with the baselines' test-time ratios for comparison. All values are percentages. our realized skipping ratios on the test set are closer to the target ratios than those of MoDES.

ModelTarget skipSLaMES calibration-search skipNAEE realized skipMC-MoE realized skipMoDES realized skipSLaMES realized skip
Qwen3-VL-30B-A3B50.050.148.947.853.450.0
Qwen3-VL-30B-A3B62.562.4861.560.467.063.2
Qwen3-VL-30B-A3B75.074.9974.273.178.477.1
Qwen3-VL-30B-A3B87.587.585.684.192.589.8
Qwen3.5-122B-A10B50.050.0252.151.160.252.8
Qwen3.5-122B-A10B62.562.4964.162.972.164.3
Qwen3.5-122B-A10B75.075.0275.974.681.276.0
Qwen3.5-122B-A10B87.587.4785.884.791.788.0

We additionally report measured performance at comparable realized skipping ratios below to provide a fairer comparison. Experimental results show that our method consistently preserves higher accuracy while achieving higher realized skipping ratios than the competing methods. Each cell reports realized skipping ratio (%) / average relative accuracy (%).

Qwen3-VL-30B-A3B

MethodGroup 1Group 2Group 3Group 4
NAEE48.9 / 97.3061.5 / 89.6874.2 / 74.1785.6 / 27.56
MC-MoE47.8 / 97.6560.4 / 93.5673.1 / 79.4684.1 / 37.03
MoDES53.4 / 98.8967.0 / 98.3178.4 / 97.7690.0 / 95.95
SLaMES55.4 / 99.4968.3 / 99.1679.3 / 97.9190.0 / 96.39

Qwen3.5-122B-A10B

MethodGroup 1Group 2Group 3Group 4
NAEE52.1 / 96.7364.1 / 93.3575.9 / 85.1685.8 / 52.03
MC-MoE51.1 / 97.4262.9 / 95.2674.6 / 89.4984.7 / 66.40
MoDES49.4 / 98.0960.2 / 97.1972.1 / 96.6081.2 / 96.00
SLaMES52.8 / 99.7064.3 / 99.3076.0 / 99.0288.0 / 98.14

Q4. How does the calibration dataset affect performance? All results in the original submission used 1,024 GQA calibration examples. Following your suggestion, we also calibrated SLaMES on COCO and ChartQA and evaluated Qwen3-VL-30B-A3B on the same four image and two video benchmarks. Each benchmark column gives its native score. Avg. Rel. is the mean of the six scores after normalization by the unmodified model, expressed as a percentage.

Calibration sourceTarget skipMVBenchDocVQAInfoVQAOCRBenchMMBench-ENVideo-MMEAvg. Rel. (%)
Unmodified model0%70.6793.3081.0390787.1167.67100.00
GQA50%69.8893.1680.2990087.2967.4499.49
COCO50%70.2292.9780.4189987.0367.9399.61
ChartQA50%69.5893.1980.7590486.7767.7899.57
GQA87.5%68.9790.4873.6687685.5765.3796.15
COCO87.5%68.4089.8972.9687684.9765.1995.61
ChartQA87.5%68.6591.1574.5887986.0865.3096.52

At the 50% target, all three sources retained at least 99.49% of baseline performance on average, with a spread of 0.12 percentage points. At the 87.5% target, the spread widened to 0.91 points, and all three retained at least 95.61%. These results show that SLaMES preserved accuracy with each of the three calibration sources on this model, with greater sensitivity to source choice under aggressive skipping. We will add this table and discussion to the revised manuscript.

Reviewer_FfrnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SLaMES, a training-free expert-skipping method for multimodal MoE models, consisting of three steps. Sensitivity calibration defines the sensitivity of each layer–modality–skip-ratio combination as the product of a global importance (the output KL when all routed experts of a layer and modality are skipped) and a local perturbation (the cosine distance under isolated recomputation of one layer). Threshold search builds a threshold-to-skip-ratio lookup table from recorded routing distributions and runs full forward passes only at sensitivity-guided frontier points, yielding per-layer, per-modality thresholds. Online skipping decides how many experts each token skips based on the cumulative routing mass of the low-probability tail, and retains the top-1 expert when the routing confidence exceeds the layer average. Experiments on Qwen3-VL-30B-A3B, Kimi-VL-16B-A3B and Qwen3.5-122B-A10B over nine image and video benchmarks compare against NAEE, MC-MoE and MoDES at target skipping ratios: SLaMES retains 99.25%–100.01% of the original relative performance at 50% and 96.39%–98.14% at the highest ratios; threshold search needs 16–20 full evaluations instead of 235–242 for MoDES; and integrated into vLLM, it achieves measured prefill speedups of up to 2.2×.

Critical strengths
  1. Training-free, with speedups measured in a real inference system. The method is integrated into the fused MoE routing of vLLM, with a single-layer latency model validated against measurements under EP=4 and end-to-end prefill speedups from 720p to 4K inputs (up to 2.2×). The experiments cover three models from 16B to 122B and both image and video benchmarks. On Qwen3-VL, SLaMES remains above MoDES when accuracy is plotted against the realized skipping ratio (Figure 3). This demonstrates practical deployment value and is the main argument in favor of acceptance.

  2. Low search cost and transparent analysis in the appendix. The lookup table reduces the number of full evaluations per target ratio from 235–242 to 16–20. The appendix candidly reports the realized per-layer skipping distributions, the limitations of early stopping and the trade-offs of the top-1 protection threshold, which helps readers judge when the method works.

Critical weaknesses
  1. The main table compares methods at target rather than realized skipping ratios, so the actual compute is not matched.

    • (a) Thresholds are searched on GQA, and the realized skipping ratio at test time deviates from the target. On Kimi-VL, the 83.4% target yields only 71.4% realized skipping (Table 6), yet Table 1 compares it with baselines at the same target. Realized ratios for the baselines are given only for Qwen3-VL (Figure 3), and neither is reported for Qwen3.5. Taking Kimi-VL as an example, SLaMES scores 97.68 at 71.4% realized skipping, whereas MoDES scores 98.96 at the 66.7% target; on Qwen3-VL the realized ratios of MoDES are never below the target, and if the same holds on Kimi-VL, SLaMES may not lead at comparable realized skipping.

    • (b) In Appendix Table 3, skipping half of the experts uniformly for visual tokens in every layer, without any calibration or search, achieves 99.32 at 47.9% realized skipping, only 0.15 points below SLaMES at the 50% target (99.47 on the same six benchmarks). This "visual-only, uniform across layers" baseline does not appear in the main table.

    This bears directly on the paper's central claim that SLaMES retains more accuracy than MoDES at the same compute. Whether this claim holds across the three models can only be judged after aligning realized skipping ratios and comparing against the simplest visual-skipping baseline.

  2. The increments over MoDES are fine-grained refinements, and the contributions of individual components are small and lack variance estimates.

    • (a) MoDES already uses a KL-based layer importance, separate thresholds for visual and text tokens, and a frontier search. SLaMES adds per-layer thresholds, sensitivities conditioned on the skipping ratio, the cumulative routing mass rule and conditional top-1 protection. In the ablations, the ratio-conditioned bins contribute 0.02–0.35 points (Table 4), the cumulative mass rule 0.06–0.81 and per-layer thresholds 0.30–1.66 (Table 2); only top-1 protection has a clear effect at 87.5% (4.34), but removing it also raises the realized skipping ratio from 89.8% to 92.6% (Table 5), so the compute is again not matched.

    • (b) The two Qwen models are decoded with sampling at temperature 0.7, and all results come from a single run with no reported variance.

    This determines my assessment of the size of the contribution: the joint layer–modality–ratio sensitivity emphasized by the paper is its main distinction from MoDES, but most component gains lie within the fluctuation one would expect from single sampled runs, and the current evidence is not sufficient to show that these design choices bring real improvements.

Questions for authors
  1. Could you report the realized skipping ratio at test time for every method, model and target ratio in Table 1, and provide accuracy-versus-realized-skipping comparisons similar to Figure 3 for Kimi-VL and Qwen3.5? Please also report realized ratios for the ablations in Table 2. If SLaMES still consistently outperforms MoDES on all three models after alignment, I would raise my score.

  2. Could you add a baseline that skips experts only for visual tokens, uniformly across layers, at each target ratio? This would show at which ratios calibration and search bring most of their gain.

  3. For the two Qwen models decoded with sampling, how much do the main and ablation results vary across random seeds?

  4. What is the latency benefit of expert skipping in the decoding stage? The current end-to-end measurements cover only prefill.

  5. The following minor issues do not affect my overall assessment and are offered for revision.

    • (a) The search-efficiency comparison counts only the full evaluations in threshold search. Sensitivity calibration requires one full skip evaluation per layer and modality, so please also report the total number of forward passes and the time including calibration.

    • (b) In Table 1, the average relative score of SLaMES on Qwen3.5-122B at 62.5% computes to 99.30 from the per-benchmark numbers, but 99.25 is reported.

    • (c) Calibration uses only GQA images, while the test set includes videos and document images; the deviation of realized skipping ratios may be related to this, and a discussion of how well the thresholds transfer across input distributions would help.

10 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Q1. Comparisons at realized skipping ratios. Thank you for emphasizing the need to compare methods using realized expert computation. Table 1 was organized by the prescribed search targets, and we agree that the realized ratios should also have been reported. Applying the calibrated thresholds to different test inputs can produce deviations from these targets. The table below reports the test-time skipping ratios for every method and target configuration in Table 1. All values are percentages.

ModelTarget skipSLaMES calibration-search skipNAEE realized skipMC-MoE realized skipMoDES realized skipSLaMES realized skip
Qwen3-VL-30B-A3B50.050.148.947.853.450.0
Qwen3-VL-30B-A3B62.562.4861.560.467.063.2
Qwen3-VL-30B-A3B75.074.9974.273.178.477.1
Qwen3-VL-30B-A3B87.587.585.684.192.589.8
Qwen3.5-122B-A10B50.050.0252.151.160.252.8
Qwen3.5-122B-A10B62.562.4964.162.972.164.3
Qwen3.5-122B-A10B75.075.0275.974.681.276.0
Qwen3.5-122B-A10B87.587.4785.884.791.788.0
Kimi-VL-16B-A3B50.049.8947.645.542.040.1
Kimi-VL-16B-A3B66.766.6861.759.356.756.1
Kimi-VL-16B-A3B83.483.3974.872.771.970.7

Figure 3 already provides an accuracy-versus-realized-skipping comparison for Qwen3-VL. We now supplement it with measured comparisons for Qwen3.5 and Kimi-VL, shown alongside Qwen3-VL below. SLaMES's efficient threshold search (Section 5.3) makes it practical to obtain additional operating points for comparison at similar realized skipping ratios. Avg. Rel. is the average benchmark score relative to the unmodified model (100%). On the Qwen-series models, our realized skipping ratios on the test set are closer to the target ratios than those of MoDES. The extent of distribution shift across datasets varies across model families. To verify that the accuracy gains observed for certain models and configurations are not merely due to differences in realized skipping ratios, we conducted the following experiments. Each cell reports realized skipping ratio (%) / average relative accuracy (%).

Qwen3-VL-30B-A3B

MethodGroup 1Group 2Group 3Group 4
NAEE48.9 / 97.3061.5 / 89.6874.2 / 74.1785.6 / 27.56
MC-MoE47.8 / 97.6560.4 / 93.5673.1 / 79.4684.1 / 37.03
MoDES53.4 / 98.8967.0 / 98.3178.4 / 97.7690.0 / 95.95
SLaMES55.4 / 99.4968.3 / 99.1679.3 / 97.9190.0 / 96.39

Qwen3.5-122B-A10B

MethodGroup 1Group 2Group 3Group 4
NAEE52.1 / 96.7364.1 / 93.3575.9 / 85.1685.8 / 52.03
MC-MoE51.1 / 97.4262.9 / 95.2674.6 / 89.4984.7 / 66.40
MoDES49.4 / 98.0960.2 / 97.1972.1 / 96.6081.2 / 96.00
SLaMES52.8 / 99.7064.3 / 99.3076.0 / 99.0288.0 / 98.14

Kimi-VL-16B-A3B

MethodGroup 1Group 2Group 3
NAEE47.6 / 97.9561.7 / 92.2774.8 / 66.10
MC-MoE45.5 / 97.9559.3 / 92.7472.7 / 69.47
MoDES42.0 / 99.7156.7 / 98.9671.9 / 94.60
SLaMES47.6 / 100.0060.2 / 99.6374.6 / 96.68

Experimental results show that our method consistently preserves higher accuracy while achieving higher realized skipping ratios than the competing methods. The additional results support SLaMES's advantage over MoDES on all three models at the reported operating points. In particular, on Qwen3.5, SLaMES retains 98.14% of baseline performance at 88.0% realized skipping, while MoDES retains 96.00% at 81.2%. SLaMES thus improves relative accuracy by 2.14 percentage points while skipping 6.8 percentage points more expert assignments. On Kimi-VL, SLaMES retains 96.68% at 74.6% realized skipping, compared with 94.60% at 71.9% for MoDES, improving relative accuracy by 2.08 points while skipping 2.7 points more assignments. At the intermediate Kimi-VL operating point, SLaMES also retains 99.63% at 60.2% skipping, compared with 98.96% at 56.7% for MoDES. These comparisons show that SLaMES can preserve more accuracy even when it executes fewer expert assignments.

We will include the complete measured results with individual benchmark scores for Kimi-VL and Qwen3.5 in the revised manuscript.

AuthorsOpenReview ↗

Rebuttal Part 2

Q2. Uniform visual-only skipping. Thank you for suggesting this comparison. We extended the visual-only baseline in Appendix Table 3 across all four operating points on Qwen3-VL-30B-A3B. At every MoE layer, the baseline skips the same number of lowest-weight routed experts for every visual token and retains all routed experts for text tokens. We additionally evaluate uniform random skipping, which randomly selects the same number of routed experts to skip per visual token at every layer, again retaining all text routes. Both baselines require no calibration or threshold search. The nominal percentage denotes the fraction skipped per visual token for the uniform baselines and the target global skipping ratio for SLaMES. Realized skip counts routed-expert assignments across both modalities during prefill and decoding. Avg. Rel. is the mean percentage of the unmodified model's score retained across the six benchmarks.

Nominal (%)MethodRealized skip (%)MVBenchDocVQAInfoVQAOCRBenchMMBench-ENVideo-MMEAvg. Rel. (%)
0Unmodified0.070.6793.3081.0390787.1167.67100.00
50Lowest-weight (skip 4/8)47.969.8093.0079.4290587.2967.3399.32
50Random (skip 4/8)47.968.9791.9676.0887786.2566.1197.24
50SLaMES51.169.8893.1680.2990087.2967.4499.49
62.5Lowest-weight (skip 5/8)59.969.9792.5778.8790186.6866.0098.66
62.5Random (skip 5/8)59.969.0190.6873.6786786.2565.9896.31
62.5SLaMES63.270.0392.9079.2489187.1166.7098.88
75Lowest-weight (skip 6/8)71.869.2091.4178.2888286.8666.1397.86
75Random (skip 6/8)71.968.7589.1070.4085485.4065.4194.75
75SLaMES76.769.5891.9377.4788886.1766.7898.02
87.5Lowest-weight (skip 7/8)83.869.1790.9273.0987185.7464.8995.98
87.5Random (skip 7/8)83.867.0885.3765.5682084.4563.0491.31
87.5SLaMES89.368.9790.4873.6687685.5765.3796.15

The uniform lowest-weight baseline is competitive, particularly at the 50% setting. Its visual skipping rate changes in steps of one expert out of eight, and retaining all text routes makes its global rate lower than its visual rate. Across all four comparisons, SLaMES skipped 3.2–5.5 percentage points more routed-expert assignments while achieving a slightly higher average relative score. The largest observed score gain was 0.22 points at the 62.5% setting. The largest difference in realized skipping occurred at the 87.5% setting, where SLaMES skipped 89.3% of assignments versus 83.8% for uniform lowest-weight skipping and retained 96.15% versus 95.98% of baseline performance. Compared with uniform random skipping, SLaMES improves average relative performance by approximately 2.2–4.8 percentage points while skipping 3.2–5.5 points more assignments. These results support the benefit of adaptive allocation over both calibration-free uniform strategies. We will include the full comparison in the revised manuscript.

AuthorsOpenReview ↗

Rebuttal Part 3

Q3. Variation across decoding seeds. Thank you for asking about sampling variability. The experimental results demonstrate that our method is robust to the choice of random seed and produces consistent results. We repeated SLaMES evaluation with seeds 0, 42, and 1234, keeping temperature at 0.7 and using the same calibrated thresholds and other generation settings. Seed 0 corresponds to the original run. Below we report Qwen3-VL at the 50% and 87.5% targets and Qwen3.5 at the 50% target. Avg. Rel. is computed across all nine benchmarks using the same original unmodified-model scores as the reference for every seed.

Qwen3-VL-30B-A3B

Target skip (%)SeedChartQADocVQAInfoVQAOCRBenchMMBench-ENMMBench-CNMVBenchVideo-MMERealWorldQAAvg. Rel. (%)Realized skip (%)
50084.8893.1680.2990087.2984.7969.8067.4472.5599.2550.0
504285.0092.7479.7689586.9484.2870.3067.0072.9499.0450.1
50123484.8892.5579.8689787.2984.4570.2567.3373.7399.2750.5
87.5080.5690.4873.6687685.5782.7368.9765.3773.7396.3989.8
87.54280.2090.1073.2287585.4883.3369.0365.5473.6196.3190.4
87.5123480.3690.3773.4587685.3182.9869.6665.1273.7196.3890.2

Qwen3.5-122B-A10B

Target skip (%)SeedChartQADocVQAInfoVQAOCRBenchMMBench-ENMMBench-CNMVBenchVideo-MMERealWorldQAAvg. Rel. (%)Realized skip (%)
50087.4091.8187.8289888.9286.9471.9570.9381.4499.7052.8
504287.5691.9287.9489788.9287.6372.7071.3380.6599.9052.8
50123487.9691.6688.2889588.2387.6372.4270.8981.1899.8152.8

Across the three seeds, mean relative performance and sample standard deviation are 99.19 ± 0.13% for Qwen3-VL at the 50% target, 96.36 ± 0.04% at the 87.5% target, and 99.80 ± 0.10% for Qwen3.5 at the 50% target. The largest between-seed range in average relative performance is 0.23 percentage points across these settings. In particular, Qwen3-VL retains 96.31%–96.39% of baseline performance at the aggressive 87.5% target. These repeated evaluations show that SLaMES's aggregate performance is stable across the tested decoding seeds, including under aggressive skipping.

Q4. Decoding latency. Thank you for raising this question. In our setting, only prefill involves processing large numbers of visual tokens. Decoding generates text and introduces no new image or video tokens. This is characteristic of the image and video understanding tasks we study, so decoding latency is outside the scope of our current evaluation. We find the question of how much expert skipping can accelerate decoding both interesting and important, and we leave it for future work.

Reviewer_1hgGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SLaMES, a method that determines layer-wise and modality-wise thresholds by calibrating sensitivity across different layers, modalities, and skipping ratios. It then dynamically skips experts based on token routing probabilities. The approach requires no retraining and achieves a favorable trade-off between accuracy and efficiency across three MoE multimodal models.

Critical strengths
  1. The method requires no retraining and can be applied directly to existing models, offering practical utility.
  2. It maintains strong performance across multiple models and tasks.
Critical weaknesses
  1. Sensitivity is measured while keeping other layers unchanged; however, skipping experts in multiple layers simultaneously alters the inputs to subsequent layers. While a full forward pass can verify a chosen configuration, it does not necessarily confirm that this static sensitivity metric reliably guides the search process.
  2. The search process estimates skipping rates based on routing probabilities, but fixed thresholds do not guarantee that a specific target ratio will be met during inference. Although Figure 3 demonstrates performance advantages at realized skipping rates, the discrepancy between the target and the actual budget remains unclear, which limits the method's applicability in scenarios with strict budget constraints.
Questions for authors
  1. When experts have already been skipped in earlier layers, does the sensitivity ranking measured on the original model remain reliable? What evidence supports using this ranking to guide subsequent search steps?
  2. What is the magnitude of the deviation between the actual expert-skipping rate and the target value? If a specific computational budget must be met during deployment, how does the proposed method ensure or adjust for this?
4 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Q1. Does sensitivity-based layer selection remain useful after upstream expert skipping?

Thank you for this question. We provide two complementary experiments on Qwen3-VL-30B-A3B.

Sensitivity under upstream skipping. Experimental results indicate that accounting for upstream layer skipping has little effect on the relative ranking of layer sensitivities. We recomputed the visual-token sensitivity at a 50% local skipping ratio, using inputs produced with upstream expert skipping. For each input condition, we compared the current layer's outputs with and without this visual-expert skipping, using identical inputs for the two branches. Search uses Sl,m=cl,mdl,mS_{l,m}=c_{l,m}d_{l,m}, where cl,mc_{l,m} is the calibrated global importance and is held fixed in this comparison. Below, all scores refer to the visual modality, whose subscript we omit for brevity. The dd and SS columns are in units of 10−310^{-3}; cc is reported without scaling. Results for a subset of layers are shown below.

LayerGlobal importance ccOriginal-input ddUpstream-skipped-input ddOriginal-input S=cdS=cdUpstream-skipped-input S=cdS=cd
20.0317500.1980.3830.0062960.012172
120.0309180.5656.2260.0174700.192503
220.0266741.14919.2960.0306420.514708
250.0233931.19120.1390.0278690.471103
400.0032772.29548.4400.0075210.158753
430.0023462.92761.2040.0068660.143600

Both rankings are preserved after upstream skipping: d2<d12<d22<d25<d40<d43d_{2}<d_{12}<d_{22}<d_{25}<d_{40}<d_{43} for local sensitivity and S2<S43<S40<S12<S25<S22S_{2}<S_{43}<S_{40}<S_{12}<S_{25}<S_{22} for the weighted score used in search. This provides direct evidence that the calibrated search score retains useful layer priorities under the tested upstream skipping configuration.

Search with randomized guidance. We additionally replaced the sensitivity ranking with random layer priorities using seeds 12 and 1234, while keeping the rest of the search procedure unchanged. At each target skipping ratio, we compare the lowest KL divergence achieved under the same evaluation budget. Each evaluation tests one candidate configuration on the same calibration set, with skipping applied jointly across layers. Our sensitivity-guided search consistently achieves substantially lower KL divergence, demonstrating that the layer sensitivity ranking effectively guides the search toward better threshold configurations.

Search guidanceBest KL at 62.5% target ↓Best KL at 75% target ↓
Random, seed 120.0098310.012976
Random, seed 12340.0094180.012589
SLaMES0.0065800.010310

In our experiments, lower KL generally corresponds to higher downstream accuracy under otherwise comparable settings. SLaMES outperforms both random seeds, reducing the best KL by 31.6% and 19.3% relative to the mean of the random runs' best values. Together, these experiments support the stability of the weighted search-score ordering in the tested layers and the practical value of the full sensitivity score for guiding search under joint skipping.

AuthorsOpenReview ↗

Rebuttal Part 2

Q2. Realized skipping ratios and performance.

Thank you for asking us to quantify the gap between target and realized skipping. We understand the importance of reporting realized skipping ratios when assessing both budget adherence and accuracy. The thresholds in Table 1 were searched at the prescribed targets using calibration routing statistics and then applied to the test benchmarks. Differences in token composition and routing distributions can shift the realized ratio at test time. Below, we report the target, calibration-search, and test-time ratios for SLaMES, together with the baselines' test-time ratios for comparison. All values are percentages.

ModelTarget skipSLaMES calibration-search skipNAEE realized skipMC-MoE realized skipMoDES realized skipSLaMES realized skip
Qwen3-VL-30B-A3B50.050.148.947.853.450.0
Qwen3-VL-30B-A3B62.562.4861.560.467.063.2
Qwen3-VL-30B-A3B75.074.9974.273.178.477.1
Qwen3-VL-30B-A3B87.587.585.684.192.589.8
Qwen3.5-122B-A10B50.050.0252.151.160.252.8
Qwen3.5-122B-A10B62.562.4964.162.972.164.3
Qwen3.5-122B-A10B75.075.0275.974.681.276.0
Qwen3.5-122B-A10B87.587.4785.884.791.788.0

(1)The calibration-search ratios differ from the targets by at most 0.11 percentage points. At test time, SLaMES remains closer to the prescribed target than MoDES at every reported operating point on both Qwen models.

(2)For deployment with a precise aggregate skipping requirement, the efficiency of our search allows the target skipping ratio to be adjusted upward or downward based on measurements from representative deployment inputs, followed by a new threshold search and validation of the realized skipping ratio. Section 5.3 reports 16–20 full evaluations for SLaMES versus 235–242 for MoDES, with an 11.5–14.3× reduction in search time under the matched 60% target setting. For example, on Qwen3-VL-30B-A3B, our method requires only 12.2 minutes for threshold search, compared with 175 minutes for MoDES. Such retuning allows the operating point to be calibrated to the deployment workload. A strict per-request budget would additionally require runtime budget enforcement.

We additionally report measured performance at comparable realized skipping ratios below to provide a fairer comparison. Experimental results show that our method consistently preserves higher accuracy while achieving higher realized skipping ratios than the competing methods. Each cell reports realized skipping ratio (%) / average relative accuracy (%).

Qwen3-VL-30B-A3B

MethodGroup 1Group 2Group 3Group 4
NAEE48.9 / 97.3061.5 / 89.6874.2 / 74.1785.6 / 27.56
MC-MoE47.8 / 97.6560.4 / 93.5673.1 / 79.4684.1 / 37.03
MoDES53.4 / 98.8967.0 / 98.3178.4 / 97.7690.0 / 95.95
SLaMES55.4 / 99.4968.3 / 99.1679.3 / 97.9190.0 / 96.39

Qwen3.5-122B-A10B

MethodGroup 1Group 2Group 3Group 4
NAEE52.1 / 96.7364.1 / 93.3575.9 / 85.1685.8 / 52.03
MC-MoE51.1 / 97.4262.9 / 95.2674.6 / 89.4984.7 / 66.40
MoDES49.4 / 98.0960.2 / 97.1972.1 / 96.6081.2 / 96.00
SLaMES52.8 / 99.7064.3 / 99.3076.0 / 99.0288.0 / 98.14

Kimi-VL-16B-A3B

MethodGroup 1Group 2Group 3
NAEE47.6 / 97.9561.7 / 92.2774.8 / 66.10
MC-MoE45.5 / 97.9559.3 / 92.7472.7 / 69.47
MoDES42.0 / 99.7156.7 / 98.9671.9 / 94.60
SLaMES47.6 / 100.0060.2 / 99.6374.6 / 96.68

Figure 3 in the submitted paper already compares accuracy against realized skipping ratios for Qwen3-VL. We now extend this comparison to Qwen3.5 and Kimi-VL. Across the reported groups, SLaMES achieves the highest average relative score, while its realized skipping ratio is at least as high as that of the baselines in most comparisons. For example, on Qwen3.5, SLaMES retains 98.14% of baseline performance at 88.0% realized skipping, compared with 96.00% at 81.2% for MoDES. We will include the complete measured results, with realized skipping ratios and individual benchmark scores, in the revised manuscript.

Thank you for the additional experiments. The results provide useful evidence that sensitivity-guided search remains effective under upstream skipping and clarify the aggregate target-to-realized gap on the two Qwen models. Although strict per-request budgets still require runtime enforcement and the new analyses are limited in model coverage, the response partially addresses my concerns, so I raise my rating to 3.

AuthorsOpenReview ↗

Thank you for your comments

Thank you for your positive feedback and thoughtful discussion. We appreciate your time and constructive suggestions, which have helped us improve the clarity and completeness of our work. Your feedback helped us strengthen the experimental analysis and we will incorporate the additional results and clarifications into the final version.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Message to AC and Reviewers

Dear Reviewers and AC,

Thank you for your time, constructive feedback, and careful reassessments. We would like to summarize the additional experiments and clarifications provided during the discussion, along with new system optimization results.

  • More complete and fairer comparisons. We report realized skipping ratios alongside individual benchmark scores and provide additional comparisons at comparable realized skipping ratios. These results support SLaMES’s accuracy advantages beyond differences in realized skipping rates.
  • Calibration-source robustness. We add experiments using COCO and ChartQA alongside GQA, showing consistent performance across the tested calibration sources. Evaluations across multiple decoding seeds also show stable performance.
  • Uniform visual-only baselines. We add lowest-weight and random expert-skipping baselines that apply a uniform rule across layers while retaining all text experts. These comparisons support the value of sensitivity calibration and threshold search.
  • Direct evaluation of upstream skipping. We examine closely ranked layers and observe that individual layer rankings can reverse. We further compare search guided by original-input sensitivities with guidance calibrated on upstream-skipped inputs. Under the same evaluation budget, original-input guidance achieves lower best KL at both tested target ratios. Comparisons with randomized guidance provide complementary evidence of its practical search effectiveness.
  • Full costs and deployment considerations. We report calibration and search costs for both single and multiple target ratios, clarify aggregate skipping targets versus strict per-request budgets, and specify the architectural scope of our method.

Additional system optimizations. We also implemented kernel-level optimizations for SLaMES prefill, including fused expert filtering in Triton and more efficient handling of valid expert-token slots in DeepEP. With skipping configurations held fixed, these optimizations provide an additional 1.10–1.23× speedup over the SLaMES implementation reported in the paper.

Nominal (%)Method1080p latency (ms)Speedup4K latency (ms)Speedup
50SLaMES (paper implementation)6561.00×26161.00×
50SLaMES + system optimizations5721.15×23711.10×
87.5SLaMES (paper implementation)4611.00×16801.00×
87.5SLaMES + system optimizations3751.23×13801.22×

All speedups are relative to the paper implementation at the same nominal skipping ratio. These comparisons isolate the incremental gains from system optimizations without changing the skipping configurations.

We appreciate the reviewers’ thoughtful discussions and updated assessments. We will incorporate the additional findings, implementation details, and clarifications into the final version of the manuscript.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.