- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies computational redundancy in pretrained MoE-based VLMs from two levels: layer-wise routing redundancy and token-wise routing redundancy. Based on two empirical observations, the authors propose Layer-level Sensitivity Profiling (LSP) to measure the effect of reducing routing capacity at different layers, Sensitivity-Guided Layer-wise Allocation (SLA) to allocate expert budgets across layers, and Token-wise Adaptive Routing (TAR) to further redistribute each layer’s budget across visual tokens according to routing weights. The method is training-free and improves the performance efficiency trade-off under aggressive visual routing sparsity.
Critical strengths
1.The overall story is clean and well motivated.
The paper starts from clear layer-level and token-level observations, and the proposed LSP/SLA and TAR components follow naturally from these observations.
2. The ablation study is reasonably thorough.
The contributions of SLA and TAR are separated, and the paper further studies TAR selection strategies, calibration robustness, and visual-vs-text routing reduction, making the source of the gains relatively clear.
3. The method is practical.
It does not require retraining the pretrained MoE model, and the authors demonstrate actual prefill latency improvements rather than only reporting theoretical FLOPs reduction.
Critical weaknesses
1. The technical novelty is limited.
The proposed techniques are simple, and the main motivations that non-uniform layer importance and redundancy among visual-token computation are already widely recognized. In particular, exploiting redundancy in visual tokens for pruning or computation reduction has been extensively studied, making the overall contribution feel incremental.
2. The analysis of “layer redundancy” is not sufficiently rigorous.
LSP measures the sensitivity of the final predictive distribution to routing-capacity reduction, but sensitivity and redundancy are not equivalent. A layer can be insensitive for reasons other than redundant expert computation, such as weak expert specialization, low layer importance, or downstream compensation. The current paper largely interprets low sensitivity as high redundancy without further validating this connection.
3. The model scale is limited for a training-free method.
The largest evaluated model is Qwen3-VL-30B-A3B. Since the proposed method only requires offline profiling and no model training, stronger validation on substantially larger MoE-VLMs would be expected to demonstrate scalability and generality.
4. The evaluation is limited to image-based benchmarks.
Since the proposed method specifically reduces computation on visual tokens and mainly accelerates the prefill stage, video understanding is a particularly relevant setting where the number of visual tokens can be much larger and prefill overhead becomes more significant. Evaluating on video benchmarks would better demonstrate the practical value and generality of the method.
Questions for authors
-
Could the authors clarify the relationship between sensitivity and redundancy? Why should a small JS divergence after reducing Top-K be interpreted as expert redundancy rather than low layer importance, insufficient expert specialization, or downstream compensation?
-
Why does the speedup decrease as the number of visual tokens increases in Sec. 5.3? Since M3MoE mainly accelerates visual-token computation during prefill, it would also be valuable to report end-to-end latency under realistic generation lengths. This is particularly important for video-based VLM benchmarks, where the number of visual tokens is substantially larger and prefill overhead can be more significant. How does M3MoE perform in such settings in terms of both task performance and end-to-end speedup?
-
How does the method scale to substantially larger MoE-VLMs? Given that the method is training-free and only requires lightweight calibration, experiments beyond the current 30B-A3B scale would significantly strengthen the paper.
-
Is the proposed routing reduction applied exclusively to visual tokens throughout LSP/SLA/TAR? If so, how does the method compare with existing visual-token pruning or visual-token reduction methods under matched computation budgets?
6 comments
Comment 2/4 for Reviewer Yx6C
Critical Weakness 2 & Question 1: Sensitivity versus redundancy
Thank you for highlighting this important distinction. We agree that a small JS divergence alone does not establish intrinsic functional duplication among experts.
Our intended notion of redundancy is operational rather than mechanistic: it measures the extent to which routing capacity can be reduced while approximately preserving the model's predictive distribution under the profiling conditions.
Specifically, LSP measures the change in the first generated token's predictive distribution after reducing routing capacity in one target layer, while keeping all other layers at their default capacities.
Low sensitivity may arise from several mechanisms, including a small contribution from the routed-expert branch, limited expert specialization, or downstream compensation. LSP does not distinguish these mechanisms, nor does it claim to identify intrinsically interchangeable experts. Instead, it directly measures the end-to-end predictive effect of reducing expert computation.
The practical utility of this signal is supported by our allocation experiments. For example, at , SLA improves normalized average performance from 91.60% to 95.76% over Descend on Kimi-VL under the same budget. This demonstrates that the measured sensitivity is useful for budget allocation, although it does not establish a unique mechanistic explanation for low sensitivity.
We will revise Secs. 3.1 and 4.2 to interpret small JS divergence as greater tolerance to routing-capacity reduction, explicitly distinguishing operational routing redundancy from intrinsic redundancy among expert functions.
Critical Weakness 3 & Question 3: Validation at a substantially larger model scale
Thank you for requesting larger-scale validation. We have extended our experiments to Qwen3.5-122B-A10B, a 122B-parameter MoE-VLM with approximately four times the total parameter count of the largest model in our original submission. Qwen3.5-122B-A10B is a 48-layer MoE-VLM with 256 routed experts per layer plus one shared expert, activating 8 routed experts and 1 shared expert per token.
We evaluate M³MoE on six benchmarks: TextVQA, ChartQA, MMStar, MMBench, AI2D, and DocVQA. All reduction methods preserve default text-token routing and target the same average visual-token expert budgets.
| Method | |||
|---|---|---|---|
| Uniform | 97.73 | 88.64 | - |
| Ascend | 97.23 | 91.44 | 86.73 |
| Descend | 92.11 | 94.50 | 92.17 |
| FastMMoE | 92.22 | 88.79 | 88.25 |
| Alloc-MoE | 99.45 | 98.63 | 96.77 |
| MoDES | 99.75 | 98.80 | 97.52 |
| M³MoE | 99.82 | 99.04 | 97.51 |
M³MoE achieves the highest measured average performance at and , retaining 99.82% and 99.04% of full-model performance, respectively, while removing 75% and 87.5% of visual-token routed-expert activations. It also outperforms Alloc-MoE at all three exactly matched budgets by 0.37, 0.41, and 0.74 percentage points.
Actual activation budgets under extreme sparsity. MoDES uses frozen thresholds calibrated offline, so its evaluation-time budget can deviate from the target. Its measured average expert counts are 2.0064, 1.0456, and 0.5503, respectively. Other evaluated configurations, including M³MoE, meet their target budgets exactly.
At , M³MoE retains 97.51% performance, compared with 97.52% for MoDES, while using 9.14% fewer visual-token routed-expert activations: .
Thus, M³MoE achieves nearly identical aggregate performance with fewer activations and exact budget adherence. These results extend our validation to the 100B-plus regime and demonstrate that the allocation benefit persists at a substantially larger model scale. We will include the complete per-benchmark results and actual activation budgets in the revision.
Comment 3/4 for Reviewer Yx6C
Critical Weakness 4 & Question 2: Video understanding and prefill efficiency
Thank you for this valuable suggestion. We agree that video understanding is an important setting for evaluating M³MoE, given its large number of visual tokens and substantial prefill computation. We have therefore added video-task evaluations and shared-backend video-prefill measurements.
1. Video understanding performance. We evaluate Qwen3-VL-30B-A3B-Instruct on 300 VideoMMMU questions in each of the Perception, Comprehension, and Adaptation categories, using 32 frames and a maximum generation length of 1,024 tokens.
| Method | Perception | Comprehension | Adaptation | average(%) |
|---|---|---|---|---|
| top-8 | 69.00 | 57.33 | 40.00 | 100.00 |
| Alloc-MoE | 68.00 | 49.67 | 36.67 | 92.29 |
| MoDES | 65.00 | 53.67 | 35.67 | 92.33 |
| M³MoE | 68.67 | 56.67 | 38.33 | 98.07 |
| Alloc-MoE | 63.67 | 47.33 | 34.00 | 86.61 |
| MoDES | 65.67 | 51.67 | 34.67 | 90.66 |
| M³MoE | 65.33 | 54.67 | 35.00 | 92.51 |
| Alloc-MoE | 63.67 | 47.33 | 36.33 | 88.55 |
| MoDES | 63.33 | 48.67 | 33.33 | 86.67 |
| M³MoE | 65.67 | 52.00 | 33.00 | 89.46 |
M³MoE achieves the highest normalized average performance across all three target budgets. At , it retains 98.07% of full-model performance while removing 75% of visual-token routed-expert activations. It also consistently outperforms Alloc-MoE and MoDES in aggregate across the evaluated budgets.
2. Shared-backend prefill acceleration on long video inputs. We additionally measure prefill latency on three VideoMMMU Perception cases, covering 4–32 frames and batch sizes of 1–8. Default and M³MoE use the same optimized gating and fused Triton expert-execution backend, isolating the latency benefit of routing reduction.
| Batch size | 4 frames | 8 frames | 16 frames | 24 frames | 32 frames |
|---|---|---|---|---|---|
| 1 | 1.11× | 1.23× | 1.36× | 1.39× | 1.44× |
| 2 | 1.26× | 1.38× | 1.48× | 1.49× | 1.48× |
| 4 | 1.40× | 1.51× | 1.54× | 1.51× | 1.54× |
| 8 | 1.52× | 1.58× | 1.59× | 1.57× | 1.56× |
Each request contains 720–5,760 visual tokens. M³MoE achieves speedups across all 20 evaluated settings, ranging from 1.11× to 1.59×. At batch size 1, speedup increases from 1.11× to 1.44× as the input grows from 4 to 32 frames. For workloads with at least 16 frames and batch size 2 or higher, speedup remains between 1.48× and 1.59×. At 32 frames and batch size 8, prefill latency decreases from 1,479.2 ms to 949.7 ms, corresponding to a 1.56× speedup.
These results also clarify the trend in our original Sec. 5.3: the original comparison used asymmetric execution paths, mixing routing savings with backend optimization. Under a shared backend, the latency benefit increases with video input length over the evaluated range.
3. Effect of unchanged decoding on generation latency. Since generated tokens retain the default Top-K, decoding latency remains unchanged by design. We further conduct a three-stage latency analysis on 30 Kimi-VL evaluation questions, measuring prefill and per-token decoding within the same generation runs.
| Backbone stage | Default (ms) | M³MoE (ms) | Speedup |
|---|---|---|---|
| Prefill | 216.50 | 155.30 | 1.40× |
| Decode (per generated token) | 35.81 | 35.89 | 1.00× |
| Prefill + 1 decode step | 252.31 | 191.19 | 1.32× |
For a 32-token generation (31 decode steps), backbone latency decreases from 1,326.60 ms to 1,267.90 ms, corresponding to a 1.046× speedup. This demonstrates that the overall benefit depends on generation length: M³MoE is particularly beneficial for prefill-intensive, short-answer workloads, whereas its relative benefit diminishes for longer generations because decoding is unaffected.
Overall, these experiments demonstrate that M³MoE generalizes to video understanding, maintains substantial prefill acceleration on long video inputs under a shared backend, and provides measurable backbone-level generation latency savings. We will incorporate these results and clearly distinguish prefill acceleration from generation-level speedup in the revision.
Comment 1/4 for Reviewer Yx6C
Critical Weakness 1: Limited technical novelty
Thank you for this important comment. We agree that non-uniform layer importance and visual-token redundancy have been extensively studied, and we do not claim these observations as novel. We would like to clarify the specific problem addressed by M³MoE.
Distinction from visual-token reduction methods. Methods such as FastV[1], SparseVLM[2], VisionZip[3], and Skip-Vision[4] reduce visual computation through token pruning, compression, or FFN skipping. In contrast, M³MoE preserves the visual-token sequence and specifically reduces routed-expert activations in MoE layers. The former primarily reduces sequence-level computation, whereas M³MoE targets conditional expert computation. These approaches therefore address different computational components and can be complementary.
Distinction from multimodal MoE acceleration methods. We acknowledge that FastMMoE[5] and MoDES[6] have already explored expert-activation reduction in multimodal MoE models. Our contribution is not the first application of expert skipping to MLLMs, but a fixed-budget formulation that explicitly allocates visual-token expert activations across both MoE layers and individual tokens. LSP measures changes in the first generated token's predictive distribution under isolated layer-wise routing perturbations. SLA converts these sensitivity profiles into a budget-preserving layer allocation, while TAR redistributes each layer's budget through global token–expert ranking. Unlike methods relying on depth-dependent schedules or importance-based skipping thresholds, M³MoE explicitly optimizes the allocation of a fixed global visual-token expert budget.
Distinction from Alloc-MoE and component-level evidence. We recognize Alloc-MoE[7] as the closest methodological predecessor. To isolate the contribution of our design beyond adapting Alloc-MoE to visual tokens, we conduct a 2×2 component-swap experiment on Kimi-VL, keeping the visual-only reduction scope and total activation budget fixed.
Replacing Alloc-L with LSP+SLA improves normalized average performance by 1.44, 5.14, and 9.01 percentage points at Kˉ=2,1,0.5, respectively, while retaining Alloc-T. Replacing Alloc-T with TAR provides additional gains under both layer-allocation strategies. The complete M³MoE framework outperforms the original Alloc-L+Alloc-T configuration by 1.80, 6.21, and 10.58 percentage points, respectively.
These results demonstrate that the gains arise from the sensitivity-profiling and allocation design, rather than merely transferring existing multi-level allocation to visual tokens. We will revise the related work, motivation, and contribution statements to acknowledge prior advances and more precisely position M³MoE.
[1] Chen, Liang, et al. "An Image Is Worth 1/2 Tokens after Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models."
[2] Zhang, Yuan, et al. "SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference."
[3] Yang, Senqiao, et al. "VisionZip: Longer Is Better but Not Necessary in Vision Language Models."
[4] Zeng, Weili, et al. "Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skipping."
[5] Xia, Guoyang, et al. "FastMMoE: Accelerating Multimodal Large Language Models through Dynamic Expert Activation and Routing-Aware Token Pruning."
[6] Huang, Yushi, et al. "MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping."
[7] Liu, Baihui, et al. "Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference."