- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SLaMES is a mechanism to adaptively skip routed expert computation based on derived sensitivity, global target drop ratio and found dropping threshold. SLaMES achieves good performance on visual benchmarks compared to prior work.
Critical strengths
- I find the paper well-written and at a high level easy to follow overall, with intuition and evidence backing their heuristics. I like how the authors have a systematic process to derive the skipping choice.
- The authors have done thorough evaluation of the work on many dataset and various models, and ablation separates the contribution of each component.
Critical weaknesses
- I think the scope of this work should be narrowed a bit: I don't see direct application of this to the whole family of MLLMs, for example, expert choice, MoMA-style/Mixture-of-Transformer-style MoE with single vision branch etc.
- I have concern on the search heuristics in 4.2: the sensitivity value S_{l,m,r} w is derived per-layer, leaving other layers intact, but during search time, other layers may have already been skipped. It is unclear how accurate/representative this can be. Additionally, skipping the least sensitive layer towards p* may not be optimal, because it uses the current bin, not the next bin, and we don't know how much budget it releases, so it's unclear if we have any guarantee what is going to happen here.
- I am not sure about how to pick p* - it's nice the author listed corresponding performance regression and latency improvements, but for practicality purposes, what p* should i choose if my goal is to limit regression by X% and improve latency by Y%? It sometimes also misses the budget - the actual skip is not as prescribed by p*.
- I am interested to see how the calibration set's impact on the final results. Currently it appears to be all sourced from GQA?
Questions for authors
Please refer to the main weakness
2 comments
Rebuttal Part 2
Q3. Choosing a target skipping ratio for deployment. Thank you for this practical question. SLaMES's fast threshold search makes it feasible to try several target ratios and select a configuration that meets the desired accuracy and latency requirements on representative validation inputs. The calibrated sensitivities and routing statistics can be reused across these searches. Section 5.3 reports an 11.5–14.3× reduction in threshold-search time relative to MoDES. In practice, one can lower the target if accuracy loss exceeds the allowed X%, or raise it if the desired Y% latency reduction is not reached, then validate the new configuration. Selection should use measured accuracy, latency, and realized skipping, accounting for deviations from the search target.
We report the target, calibration-search, and test-time ratios for SLaMES, together with the baselines' test-time ratios for comparison. All values are percentages. our realized skipping ratios on the test set are closer to the target ratios than those of MoDES.
| Model | Target skip | SLaMES calibration-search skip | NAEE realized skip | MC-MoE realized skip | MoDES realized skip | SLaMES realized skip |
|---|---|---|---|---|---|---|
| Qwen3-VL-30B-A3B | 50.0 | 50.1 | 48.9 | 47.8 | 53.4 | 50.0 |
| Qwen3-VL-30B-A3B | 62.5 | 62.48 | 61.5 | 60.4 | 67.0 | 63.2 |
| Qwen3-VL-30B-A3B | 75.0 | 74.99 | 74.2 | 73.1 | 78.4 | 77.1 |
| Qwen3-VL-30B-A3B | 87.5 | 87.5 | 85.6 | 84.1 | 92.5 | 89.8 |
| Qwen3.5-122B-A10B | 50.0 | 50.02 | 52.1 | 51.1 | 60.2 | 52.8 |
| Qwen3.5-122B-A10B | 62.5 | 62.49 | 64.1 | 62.9 | 72.1 | 64.3 |
| Qwen3.5-122B-A10B | 75.0 | 75.02 | 75.9 | 74.6 | 81.2 | 76.0 |
| Qwen3.5-122B-A10B | 87.5 | 87.47 | 85.8 | 84.7 | 91.7 | 88.0 |
We additionally report measured performance at comparable realized skipping ratios below to provide a fairer comparison. Experimental results show that our method consistently preserves higher accuracy while achieving higher realized skipping ratios than the competing methods. Each cell reports realized skipping ratio (%) / average relative accuracy (%).
Qwen3-VL-30B-A3B
| Method | Group 1 | Group 2 | Group 3 | Group 4 |
|---|---|---|---|---|
| NAEE | 48.9 / 97.30 | 61.5 / 89.68 | 74.2 / 74.17 | 85.6 / 27.56 |
| MC-MoE | 47.8 / 97.65 | 60.4 / 93.56 | 73.1 / 79.46 | 84.1 / 37.03 |
| MoDES | 53.4 / 98.89 | 67.0 / 98.31 | 78.4 / 97.76 | 90.0 / 95.95 |
| SLaMES | 55.4 / 99.49 | 68.3 / 99.16 | 79.3 / 97.91 | 90.0 / 96.39 |
Qwen3.5-122B-A10B
| Method | Group 1 | Group 2 | Group 3 | Group 4 |
|---|---|---|---|---|
| NAEE | 52.1 / 96.73 | 64.1 / 93.35 | 75.9 / 85.16 | 85.8 / 52.03 |
| MC-MoE | 51.1 / 97.42 | 62.9 / 95.26 | 74.6 / 89.49 | 84.7 / 66.40 |
| MoDES | 49.4 / 98.09 | 60.2 / 97.19 | 72.1 / 96.60 | 81.2 / 96.00 |
| SLaMES | 52.8 / 99.70 | 64.3 / 99.30 | 76.0 / 99.02 | 88.0 / 98.14 |
Q4. How does the calibration dataset affect performance? All results in the original submission used 1,024 GQA calibration examples. Following your suggestion, we also calibrated SLaMES on COCO and ChartQA and evaluated Qwen3-VL-30B-A3B on the same four image and two video benchmarks. Each benchmark column gives its native score. Avg. Rel. is the mean of the six scores after normalization by the unmodified model, expressed as a percentage.
| Calibration source | Target skip | MVBench | DocVQA | InfoVQA | OCRBench | MMBench-EN | Video-MME | Avg. Rel. (%) |
|---|---|---|---|---|---|---|---|---|
| Unmodified model | 0% | 70.67 | 93.30 | 81.03 | 907 | 87.11 | 67.67 | 100.00 |
| GQA | 50% | 69.88 | 93.16 | 80.29 | 900 | 87.29 | 67.44 | 99.49 |
| COCO | 50% | 70.22 | 92.97 | 80.41 | 899 | 87.03 | 67.93 | 99.61 |
| ChartQA | 50% | 69.58 | 93.19 | 80.75 | 904 | 86.77 | 67.78 | 99.57 |
| GQA | 87.5% | 68.97 | 90.48 | 73.66 | 876 | 85.57 | 65.37 | 96.15 |
| COCO | 87.5% | 68.40 | 89.89 | 72.96 | 876 | 84.97 | 65.19 | 95.61 |
| ChartQA | 87.5% | 68.65 | 91.15 | 74.58 | 879 | 86.08 | 65.30 | 96.52 |
At the 50% target, all three sources retained at least 99.49% of baseline performance on average, with a spread of 0.12 percentage points. At the 87.5% target, the spread widened to 0.91 points, and all three retained at least 95.61%. These results show that SLaMES preserved accuracy with each of the three calibration sources on this model, with greater sensitivity to source choice under aggressive skipping. We will add this table and discussion to the revised manuscript.
Rebuttal Part 1
Q1. Scope of applicability. Thank you for pointing out that our scope should be more explicit. SLaMES targets multimodal MoE models with token-choice Top-K routing, where each token activates multiple experts with associated routing weights. This routing architecture is widely used in contemporary MoE models. The three models we evaluated use this design (K = 6 or 8). Our contribution in this setting is to measure how skipping tolerance varies with layer, modality, and skipping ratio, and to use those measurements to allocate expert computation without retraining. The architectures you mention require different treatment. MoMa uses expert-choice routing within modality-specific groups during training, so the number of experts assigned to a token can vary. Extending SLaMES to this setting would require accounting for those variable assignments when estimating the skipping budget. In a MoT-style model with a single visual feed-forward branch, a visual token has no lower-weight expert tail to skip. We will clarify these architectural assumptions in the revised manuscript.
Q2. Reliability of sensitivity guidance and the search heuristic. Thank you for raising these concerns. We agree that upstream skipping can change subsequent layers' inputs. The calibrated sensitivity serves as a rate-dependent priority for computation allocation, rather than an exact prediction of the loss under joint skipping. We conducted two additional experiments on Qwen3-VL-30B-A3B to test whether it remains useful in this setting.
Sensitivity with perturbed inputs. We recomputed the visual-token sensitivity at a 50% local skipping ratio, using inputs produced with upstream expert skipping. For each input condition, we compared the current layer's outputs with and without this visual-expert skipping, using identical inputs for the two branches. Search uses Sl,m=cl,mdl,m, where cl,m is the calibrated global importance and is held fixed in this comparison. Below, all scores refer to the visual modality, whose subscript we omit for brevity. The d and S columns are in units of 10−3; c is reported without scaling. Results for a subset of layers are shown below.
Although the magnitudes increase, the ordering S2<S43<S40<S12<S25<S22 is preserved. This supports the usefulness of the calibrated priorities in the tested layers and upstream-skipping condition.
Search under joint skipping. We also replaced sensitivity-based layer priorities with random priorities while keeping the rest of the search unchanged. The table reports the lowest calibration KL within the first 16 full-model evaluations, each applying the candidate skipping configuration jointly across layers.
SLaMES reduces the best KL by 31.6% and 19.3%, respectively, relative to the mean of the two random runs' best values, while achieving slightly higher measured skipping ratios. This comparison directly tests the contribution of sensitivity guidance under the same evaluation budget.
Current-bin selection and budget updates. The current-bin score determines which layer to adjust. For each candidate threshold step, the offline lookup table supplies the estimated skipped-expert count at the new threshold, so its budget effect is explicitly computed. Visual restoration stops and reselects the layer when its sensitivity bin changes, and rejects a step if the estimated global skipping ratio would fall below the target. Text skipping advances by one threshold step, after which its ratio and sensitivity bin are updated. Thus, priorities change as the allocation changes. However, the current-bin score is not a next-step marginal-loss estimate, and we do not claim that it guarantees an optimal update or a globally optimal configuration. Full-model KL evaluations select the best configuration among the visited frontier candidates. The experiments above provide empirical support for this efficient heuristic.