acceptodds
Under review as a conference paper at ICLR 2027

MMoE: Multi-Level Routing Capacity Allocation for Extremely Sparse Multimodal MoE Inference

Abstract

Mixture-of-Experts (MoE) has become a prevalent architecture for scaling multimodal large language models (MLLMs), yet activating multiple experts for hundreds of visual tokens can still incur substantial inference overhead. In this work, we investigate routing redundancy in MoE-based MLLMs and identify two complementary forms of heterogeneity: (i) different MoE layers exhibit markedly different sensitivities to routing-capacity reduction, and (ii) visual tokens within the same layer require substantially different amounts of expert computation. Motivated by these observations, we propose MMoE, a multimodal-specific multi-level routing-capacity allocation framework for efficient MoE inference under a fixed visual-token expert budget. MMoE first profiles layer sensitivity through output-distribution perturbation and allocates routing capacity across MoE layers accordingly. It then redistributes each layer's budget among visual tokens by prioritizing token-expert pairs with larger routing weights, thereby preserving more routing mass under the same budget. Experiments on two representative MoE-based MLLMs across eight vision-language benchmarks show that MMoE achieves the highest normalized average performance compared with existing routing strategies across multiple capacity budgets. Under aggressive sparsity, it improves normalized average performance by up to 5.57% over the strongest baseline while skipping more than 91% of visual-token expert activations, and achieves up to 2.62× prefill speedup on Kimi-VL-A3B-Instruct.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal large language models, mixture-of-experts, routing capacity allocation, inference acceleration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Yx6COpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies computational redundancy in pretrained MoE-based VLMs from two levels: layer-wise routing redundancy and token-wise routing redundancy. Based on two empirical observations, the authors propose Layer-level Sensitivity Profiling (LSP) to measure the effect of reducing routing capacity at different layers, Sensitivity-Guided Layer-wise Allocation (SLA) to allocate expert budgets across layers, and Token-wise Adaptive Routing (TAR) to further redistribute each layer’s budget across visual tokens according to routing weights. The method is training-free and improves the performance efficiency trade-off under aggressive visual routing sparsity.

Critical strengths

1.The overall story is clean and well motivated.

The paper starts from clear layer-level and token-level observations, and the proposed LSP/SLA and TAR components follow naturally from these observations.

2. The ablation study is reasonably thorough.

The contributions of SLA and TAR are separated, and the paper further studies TAR selection strategies, calibration robustness, and visual-vs-text routing reduction, making the source of the gains relatively clear.

3. The method is practical.

It does not require retraining the pretrained MoE model, and the authors demonstrate actual prefill latency improvements rather than only reporting theoretical FLOPs reduction.

Critical weaknesses

1. The technical novelty is limited.

The proposed techniques are simple, and the main motivations that non-uniform layer importance and redundancy among visual-token computation are already widely recognized. In particular, exploiting redundancy in visual tokens for pruning or computation reduction has been extensively studied, making the overall contribution feel incremental.

2. The analysis of “layer redundancy” is not sufficiently rigorous.

LSP measures the sensitivity of the final predictive distribution to routing-capacity reduction, but sensitivity and redundancy are not equivalent. A layer can be insensitive for reasons other than redundant expert computation, such as weak expert specialization, low layer importance, or downstream compensation. The current paper largely interprets low sensitivity as high redundancy without further validating this connection.

3. The model scale is limited for a training-free method.

The largest evaluated model is Qwen3-VL-30B-A3B. Since the proposed method only requires offline profiling and no model training, stronger validation on substantially larger MoE-VLMs would be expected to demonstrate scalability and generality.

4. The evaluation is limited to image-based benchmarks.

Since the proposed method specifically reduces computation on visual tokens and mainly accelerates the prefill stage, video understanding is a particularly relevant setting where the number of visual tokens can be much larger and prefill overhead becomes more significant. Evaluating on video benchmarks would better demonstrate the practical value and generality of the method.

Questions for authors
  1. Could the authors clarify the relationship between sensitivity and redundancy? Why should a small JS divergence after reducing Top-K be interpreted as expert redundancy rather than low layer importance, insufficient expert specialization, or downstream compensation?

  2. Why does the speedup decrease as the number of visual tokens increases in Sec. 5.3? Since M3MoE mainly accelerates visual-token computation during prefill, it would also be valuable to report end-to-end latency under realistic generation lengths. This is particularly important for video-based VLM benchmarks, where the number of visual tokens is substantially larger and prefill overhead can be more significant. How does M3MoE perform in such settings in terms of both task performance and end-to-end speedup?

  3. How does the method scale to substantially larger MoE-VLMs? Given that the method is training-free and only requires lightweight calibration, experiments beyond the current 30B-A3B scale would significantly strengthen the paper.

  4. Is the proposed routing reduction applied exclusively to visual tokens throughout LSP/SLA/TAR? If so, how does the method compare with existing visual-token pruning or visual-token reduction methods under matched computation budgets?

6 comments
AuthorsOpenReview ↗

Comment 1/4 for Reviewer Yx6C

Critical Weakness 1: Limited technical novelty

Thank you for this important comment. We agree that non-uniform layer importance and visual-token redundancy have been extensively studied, and we do not claim these observations as novel. We would like to clarify the specific problem addressed by M³MoE.

Distinction from visual-token reduction methods. Methods such as FastV[1], SparseVLM[2], VisionZip[3], and Skip-Vision[4] reduce visual computation through token pruning, compression, or FFN skipping. In contrast, M³MoE preserves the visual-token sequence and specifically reduces routed-expert activations in MoE layers. The former primarily reduces sequence-level computation, whereas M³MoE targets conditional expert computation. These approaches therefore address different computational components and can be complementary.

Distinction from multimodal MoE acceleration methods. We acknowledge that FastMMoE[5] and MoDES[6] have already explored expert-activation reduction in multimodal MoE models. Our contribution is not the first application of expert skipping to MLLMs, but a fixed-budget formulation that explicitly allocates visual-token expert activations across both MoE layers and individual tokens. LSP measures changes in the first generated token's predictive distribution under isolated layer-wise routing perturbations. SLA converts these sensitivity profiles into a budget-preserving layer allocation, while TAR redistributes each layer's budget through global token–expert ranking. Unlike methods relying on depth-dependent schedules or importance-based skipping thresholds, M³MoE explicitly optimizes the allocation of a fixed global visual-token expert budget.

Distinction from Alloc-MoE and component-level evidence. We recognize Alloc-MoE[7] as the closest methodological predecessor. To isolate the contribution of our design beyond adapting Alloc-MoE to visual tokens, we conduct a 2×2 component-swap experiment on Kimi-VL, keeping the visual-only reduction scope and total activation budget fixed.

Layer allocationToken allocationKˉ=2\bar K=2Kˉ=1\bar K=1Kˉ=0.5\bar K=0.5
Alloc-LAlloc-T97.8992.1886.59
SLAAlloc-T99.33 (+1.44)97.32 (+5.14)95.60 (+9.01)
Alloc-LTAR98.15 (+0.26)92.26 (+0.08)86.71 (+0.12)
SLATAR99.69 (+1.80)98.39 (+6.21)97.17 (+10.58)

Replacing Alloc-L with LSP+SLA improves normalized average performance by 1.44, 5.14, and 9.01 percentage points at Kˉ=2,1,0.5\bar K=2,1,0.5, respectively, while retaining Alloc-T. Replacing Alloc-T with TAR provides additional gains under both layer-allocation strategies. The complete M³MoE framework outperforms the original Alloc-L+Alloc-T configuration by 1.80, 6.21, and 10.58 percentage points, respectively.

These results demonstrate that the gains arise from the sensitivity-profiling and allocation design, rather than merely transferring existing multi-level allocation to visual tokens. We will revise the related work, motivation, and contribution statements to acknowledge prior advances and more precisely position M³MoE.


[1] Chen, Liang, et al. "An Image Is Worth 1/2 Tokens after Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models."

[2] Zhang, Yuan, et al. "SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference."

[3] Yang, Senqiao, et al. "VisionZip: Longer Is Better but Not Necessary in Vision Language Models."

[4] Zeng, Weili, et al. "Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skipping."

[5] Xia, Guoyang, et al. "FastMMoE: Accelerating Multimodal Large Language Models through Dynamic Expert Activation and Routing-Aware Token Pruning."

[6] Huang, Yushi, et al. "MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping."

[7] Liu, Baihui, et al. "Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference."

AuthorsOpenReview ↗

Comment 2/4 for Reviewer Yx6C

Critical Weakness 2 & Question 1: Sensitivity versus redundancy

Thank you for highlighting this important distinction. We agree that a small JS divergence alone does not establish intrinsic functional duplication among experts.

Our intended notion of redundancy is operational rather than mechanistic: it measures the extent to which routing capacity can be reduced while approximately preserving the model's predictive distribution under the profiling conditions.

Specifically, LSP measures the change in the first generated token's predictive distribution after reducing routing capacity in one target layer, while keeping all other layers at their default capacities.

Low sensitivity may arise from several mechanisms, including a small contribution from the routed-expert branch, limited expert specialization, or downstream compensation. LSP does not distinguish these mechanisms, nor does it claim to identify intrinsically interchangeable experts. Instead, it directly measures the end-to-end predictive effect of reducing expert computation.

The practical utility of this signal is supported by our allocation experiments. For example, at Kˉ=0.5\bar K=0.5, SLA improves normalized average performance from 91.60% to 95.76% over Descend on Kimi-VL under the same budget. This demonstrates that the measured sensitivity is useful for budget allocation, although it does not establish a unique mechanistic explanation for low sensitivity.

We will revise Secs. 3.1 and 4.2 to interpret small JS divergence as greater tolerance to routing-capacity reduction, explicitly distinguishing operational routing redundancy from intrinsic redundancy among expert functions.


Critical Weakness 3 & Question 3: Validation at a substantially larger model scale

Thank you for requesting larger-scale validation. We have extended our experiments to Qwen3.5-122B-A10B, a 122B-parameter MoE-VLM with approximately four times the total parameter count of the largest model in our original submission. Qwen3.5-122B-A10B is a 48-layer MoE-VLM with 256 routed experts per layer plus one shared expert, activating 8 routed experts and 1 shared expert per token.

We evaluate M³MoE on six benchmarks: TextVQA, ChartQA, MMStar, MMBench, AI2D, and DocVQA. All reduction methods preserve default text-token routing and target the same average visual-token expert budgets.

MethodKvˉ=2\bar {K_v}=2Kvˉ=1\bar {K_v}=1Kvˉ=0.5\bar {K_v}=0.5
Uniform97.7388.64-
Ascend97.2391.4486.73
Descend92.1194.5092.17
FastMMoE92.2288.7988.25
Alloc-MoE99.4598.6396.77
MoDES99.7598.8097.52
M³MoE99.8299.0497.51

M³MoE achieves the highest measured average performance at Kvˉ=2\bar {K_v}=2 and Kvˉ=1\bar {K_v}=1, retaining 99.82% and 99.04% of full-model performance, respectively, while removing 75% and 87.5% of visual-token routed-expert activations. It also outperforms Alloc-MoE at all three exactly matched budgets by 0.37, 0.41, and 0.74 percentage points.

Actual activation budgets under extreme sparsity. MoDES uses frozen thresholds calibrated offline, so its evaluation-time budget can deviate from the target. Its measured average expert counts are 2.0064, 1.0456, and 0.5503, respectively. Other evaluated configurations, including M³MoE, meet their target budgets exactly.

At Kˉv=0.5\bar K_v=0.5, M³MoE retains 97.51% performance, compared with 97.52% for MoDES, while using 9.14% fewer visual-token routed-expert activations: 1−0.50000.5503=9.141-\frac{0.5000}{0.5503}=9.14\\%.

Thus, M³MoE achieves nearly identical aggregate performance with fewer activations and exact budget adherence. These results extend our validation to the 100B-plus regime and demonstrate that the allocation benefit persists at a substantially larger model scale. We will include the complete per-benchmark results and actual activation budgets in the revision.

AuthorsOpenReview ↗

Comment 3/4 for Reviewer Yx6C

Critical Weakness 4 & Question 2: Video understanding and prefill efficiency

Thank you for this valuable suggestion. We agree that video understanding is an important setting for evaluating M³MoE, given its large number of visual tokens and substantial prefill computation. We have therefore added video-task evaluations and shared-backend video-prefill measurements.

1. Video understanding performance. We evaluate Qwen3-VL-30B-A3B-Instruct on 300 VideoMMMU questions in each of the Perception, Comprehension, and Adaptation categories, using 32 frames and a maximum generation length of 1,024 tokens.

MethodPerceptionComprehensionAdaptationaverage(%)
top-869.0057.3340.00100.00
Kvˉ=2\bar {K_v}=2
Alloc-MoE68.0049.6736.6792.29
MoDES65.0053.6735.6792.33
M³MoE68.6756.6738.3398.07
Kvˉ=1\bar {K_v}=1
Alloc-MoE63.6747.3334.0086.61
MoDES65.6751.6734.6790.66
M³MoE65.3354.6735.0092.51
Kvˉ=0.5\bar {K_v}=0.5
Alloc-MoE63.6747.3336.3388.55
MoDES63.3348.6733.3386.67
M³MoE65.6752.0033.0089.46

M³MoE achieves the highest normalized average performance across all three target budgets. At Kˉv=2\bar K_v=2, it retains 98.07% of full-model performance while removing 75% of visual-token routed-expert activations. It also consistently outperforms Alloc-MoE and MoDES in aggregate across the evaluated budgets.

2. Shared-backend prefill acceleration on long video inputs. We additionally measure prefill latency on three VideoMMMU Perception cases, covering 4–32 frames and batch sizes of 1–8. Default and M³MoE use the same optimized gating and fused Triton expert-execution backend, isolating the latency benefit of routing reduction.

Batch size4 frames8 frames16 frames24 frames32 frames
11.11×1.23×1.36×1.39×1.44×
21.26×1.38×1.48×1.49×1.48×
41.40×1.51×1.54×1.51×1.54×
81.52×1.58×1.59×1.57×1.56×

Each request contains 720–5,760 visual tokens. M³MoE achieves speedups across all 20 evaluated settings, ranging from 1.11× to 1.59×. At batch size 1, speedup increases from 1.11× to 1.44× as the input grows from 4 to 32 frames. For workloads with at least 16 frames and batch size 2 or higher, speedup remains between 1.48× and 1.59×. At 32 frames and batch size 8, prefill latency decreases from 1,479.2 ms to 949.7 ms, corresponding to a 1.56× speedup.

These results also clarify the trend in our original Sec. 5.3: the original comparison used asymmetric execution paths, mixing routing savings with backend optimization. Under a shared backend, the latency benefit increases with video input length over the evaluated range.

3. Effect of unchanged decoding on generation latency. Since generated tokens retain the default Top-K, decoding latency remains unchanged by design. We further conduct a three-stage latency analysis on 30 Kimi-VL evaluation questions, measuring prefill and per-token decoding within the same generation runs.

Backbone stageDefault (ms)M³MoE (ms)Speedup
Prefill216.50155.301.40×
Decode (per generated token)35.8135.891.00×
Prefill + 1 decode step252.31191.191.32×

For a 32-token generation (31 decode steps), backbone latency decreases from 1,326.60 ms to 1,267.90 ms, corresponding to a 1.046× speedup. This demonstrates that the overall benefit depends on generation length: M³MoE is particularly beneficial for prefill-intensive, short-answer workloads, whereas its relative benefit diminishes for longer generations because decoding is unaffected.

Overall, these experiments demonstrate that M³MoE generalizes to video understanding, maintains substantial prefill acceleration on long video inputs under a shared backend, and provides measurable backbone-level generation latency savings. We will incorporate these results and clearly distinguish prefill acceleration from generation-level speedup in the revision.

Reviewer_wdMYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Mixture-of-Experts models. M3MoE profiles the effect of reducing expert capacity in each layer, allocates a fixed budget across layers, and then selects token-expert pairs by routing weights. It reports improved performance under sparse routing and faster prefill on the evaluated models.

Critical strengths
  1. The paper is well-written and easy to follow. It identifies differences in routing sensitivity across layers and in routing weights across visual tokens. These observations give a clear reason to avoid uniform expert allocation.

  2. The method works without retraining the underlying model and has a simple fixed-budget formulation.

  3. The evaluation includes two MoE-based vision-language models and comparisons with existing routing methods. The paper measures actual prefill latency rather than relying only on a count of skipped experts.

Critical weaknesses
  1. Limited literature review. The claim that capacity allocation for visual tokens "remains insufficiently understood" is overstated.

    • Layer- and token-adaptive expert counts are already well studied. LExI [1] allocates active experts per layer according to layer importance. Top-P routing [2] already reports that lower layers need more experts, which anticipates both Motivation (i) and (ii).
    • The redundancy of visual tokens in FFNs is also known. Skip-Vision [3] skips FFNs for redundant visual tokens, so Table 5 is not a new finding.
    • Visual token pruning (FastV [4], SparseVLM [5], VisionZip [6]) is neither discussed nor compared. Yet pruning a visual token is equivalent to assigning it zero experts, and it additionally saves attention and KV-cache cost.
  2. Limited increment over Alloc-MoE.

    • Alloc-MoE [7] already combines sensitivity-guided layer allocation with token-level redistribution by global top-k over routing scores. TAR (Eq. 7) is Alloc-T with K_base = 0, and LSP/SLA closely follow Alloc-L, down to the component names.
    • The remaining differences are small: JS divergence instead of perplexity, proportional allocation instead of DP, and visual-only reduction. Visual-only reduction is also already used by FastMMoE [8] and MoDES [9]. The paper should state its contribution relative to Alloc-MoE explicitly and show which of these changes drives the gains.
  3. Doubtful speedup results.

    • The 2.62× prefill speedup exceeds what the reduced computation can explain. In Kimi-VL's LLM, routed experts account for roughly 60% of per-token linear compute. Removing 91.67% of them caps the LLM-side speedup at about 2.3×, even before counting text tokens, attention, and the ViT.
    • The speedup also shrinks with more visual tokens (2.62×→1.92×) and larger batches (1.80×→1.31×). This pattern points to reduced fixed overhead from the custom kernels in Appendix E rather than to saved computation. Please double-check this and report the speedup with Default running on the same fused kernels.

References

[1] Chitty-Venkata, Krishna Teja, et al. "LExI: Layer-Adaptive Active Experts for Efficient MoE Inference."

[2] Huang, Quzhe, et al. "Harder Task Needs More Experts: Dynamic Routing in MoE Models."

[3] Zeng, Weili, et al. "Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skipping."

[4] Chen, Liang, et al. "An Image Is Worth 1/2 Tokens after Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models."

[5] Zhang, Yuan, et al. "SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference."

[6] Yang, Senqiao, et al. "VisionZip: Longer Is Better but Not Necessary in Vision Language Models."

[7] Liu, Baihui, et al. "Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference."

[8] Xia, Guoyang, et al. "FastMMoE: Accelerating Multimodal Large Language Models through Dynamic Expert Activation and Routing-Aware Token Pruning."

[9] Huang, Yushi, et al. "MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping."

Questions for authors
  1. The reproduced MoDES seems weaker than reported. MoDES reports 97.33% retained performance at 88% expert skipping on Qwen3-VL-30B-A3B. Here it reaches only 95.32% at 87.5%, even though only visual tokens are skipped. Can the authors explain the gap?
  2. Decode cost is unchanged by design, since generated tokens keep the full Top-K. What share of end-to-end latency does prefill account for in typical MLLM workloads, and what is the resulting end-to-end speedup?
  3. Motivation (ii) argues that tokens with dispersed routing weights need more experts. However, TAR maximizes total retained mass, which under tight budgets favors peaked tokens and can leave dispersed tokens with no expert at all. Why is total mass the right objective?
7 comments
AuthorsOpenReview ↗

Comment 1/5 for Reviewer wdMY

Critical Weakness 1: Related work and novelty positioning

Thank you for these insightful comments. We agree that our original literature review and motivation did not sufficiently acknowledge prior work on adaptive expert allocation and visual-token redundancy. We will revise the related work and narrow our claims to better clarify the contribution of M³MoE.

Relation to adaptive expert allocation. We acknowledge that LExI[1] has explored layer-wise active-expert allocation, while Top-P[2] routing dynamically determines the number of experts per token based on cumulative routing probabilities. These works establish that layer-wise and token-wise routing heterogeneity are not new observations. Moreover, Alloc-MoE[3] has studied multi-level budget allocation in text-only MoE models. Our contribution is the multimodal-specific formulation for allocating a fixed visual-token expert budget. Specifically, M³MoE combines perturbation-based layer sensitivity profiling (LSP), sensitivity-guided layer-wise allocation (SLA), and global token–expert ranking (TAR) to exploit routing heterogeneity at both levels.

Relation to visual-token FFN skipping and pruning. We also acknowledge that Skip-Vision[4] has investigated redundant visual-token FFN computation. However, skipping routed experts and removing visual tokens are not equivalent. When a token activates zero routed experts, it remains in the sequence and continues participating in subsequent self-attention, while its routed-expert contribution is omitted (with shared-expert computation retained when applicable). In contrast, token pruning removes the token from subsequent computation, reducing both FFN/MoE and attention/KV-cache costs. These approaches therefore target different computational components and can be complementary.

Comparison with visual-token reduction methods. We additionally evaluate FastV[5] and VisionZip[6] on Kimi-VL:

MethodTextVQAChartQAMMStarMMBench EN DevMMEAI2DDocVQA Val Liteaverage(%)
top-688.7089.4449.5583.33220183.6197.78100
Kvˉ=2\bar {K_v}=2
FastV88.2885.6047.3181.96216481.8794.2797.39
VisionZip86.1085.6047.0980.67212080.2591.6995.81
M3MoE88.1889.4049.1582.90219583.2697.3299.55
Kvˉ=1\bar {K_v}=1
FastV87.5181.4844.0279.98206779.6689.0293.54
VisionZip81.8179.9644.6579.55203276.0778.2090.07
M3MoE86.9488.4048.6782.22216382.4597.2698.59
Kvˉ=0.5\bar {K_v}=0.5
FastV86.6376.8843.0579.21203277.4382.0290.62
VisionZip77.4973.2443.1878.61196873.7470.4185.76
M3MoE84.2386.6547.9582.39215682.8097.0697.68

Across seven benchmarks, M³MoE achieves normalized average performance of 99.55%, 98.59%, and 97.68% at Kvˉ=2,1,0.5\bar {K_v}=2,1,0.5, respectively. These results exceed FastV by 2.16, 5.05, and 7.06 percentage points, and VisionZip by 3.74, 8.52, and 11.92 percentage points, respectively. The advantage becomes more pronounced under aggressive reduction. We will incorporate these results and discuss FastV and VisionZip in the revised related work.

Importantly, we do not claim that expert reduction universally replaces visual-token pruning. Token pruning additionally reduces attention and KV-cache costs, whereas M³MoE specifically targets MoE expert computation while retaining visual tokens. We will accordingly revise the statements to position M³MoE as a visual-token-specific, multi-level routing-capacity allocation framework for MoE-based MLLMs.


Reference

[1] Chitty-Venkata, Krishna Teja, et al. "LExI: Layer-Adaptive Active Experts for Efficient MoE Inference."

[2] Huang, Quzhe, et al. "Harder Task Needs More Experts: Dynamic Routing in MoE Models."

[3] Liu, Baihui, et al. "Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference."

[4] Zeng, Weili, et al. "Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skipping."

[5] Chen, Liang, et al. "An Image Is Worth 1/2 Tokens after Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models."

[6] Yang, Senqiao, et al. "VisionZip: Longer Is Better but Not Necessary in Vision Language Models."

AuthorsOpenReview ↗

Comment 2/5 for Reviewer wdMY

Critical Weakness 2: Limited methodological increment over Alloc-MoE

Thank you for identifying Alloc-MoE as the closest related work. We agree that M³MoE shares the general principle of multi-level budget allocation with Alloc-MoE. Our distinction lies in the sensitivity measurement and allocation criterion, which we further validate through controlled component-swap experiments.

1. Distribution-preserving sensitivity modeling. Alloc-L profiles perplexity under a layer-wise allocation procedure that constrains deeper layers to Top-1. In contrast, LSP independently reduces visual-token routing capacity in each target layer while keeping all other layers at their default capacities. It measures the change in the first generated token's predictive distribution using JS divergence. Thus, LSP measures distributional fidelity to the original model rather than likelihood on reference text, and does not require reference answers to compute its sensitivity signal. These are modality-specific adaptations in sensitivity modeling, not simply a change in the modality being processed.

2. Different layer-allocation criterion. Alloc-L uses dynamic programming (DP) to minimize the sum of layer-wise sensitivity costs. SLA instead aggregates each layer's sensitivity across reduction levels and performs capped, budget-preserving proportional allocation. This distinction matters because sensitivities measured under isolated perturbations need not be additive when multiple layers are simultaneously reduced.

Appendix G directly evaluates this difference using the same JS sensitivity matrix, capacity constraints, and global budgets, with token redistribution disabled. SLA outperforms DP by 0.40, 1.11, and 2.24 percentage points on Qwen3-VL at Kˉ=2,1,0.5\bar K=2,1,0.5, respectively. This demonstrates that the allocation rule itself contributes measurable gains independently of the sensitivity metric or token-level policy. An exact solution to the additive proxy does not necessarily yield the best end-to-end allocation.

3. Controlled component-swap experiments beyond visual-only adaptation. To isolate the contributions of our allocation components beyond adapting Alloc-MoE to visual tokens, we conduct a 2×2 component-swap experiment on Kimi-VL. All configurations reduce only visual-token expert activations, preserve default text-token routing, and use identical total visual-token expert budgets.

Layer allocationToken allocationKˉ=2\bar K=2Kˉ=1\bar K=1Kˉ=0.5\bar K=0.5
Alloc-LAlloc-T97.8992.1886.59
LSP+SLAAlloc-T99.33 (+1.44)97.32 (+5.14)95.60 (+9.01)
Alloc-LTAR98.15 (+0.26)92.26 (+0.08)86.71 (+0.12)
LSP+SLATAR99.69 (+1.80)98.39 (+6.21)97.17 (+10.58)

The results reveal two complementary contributions:

  • Layer-level allocation: Keeping Alloc-T unchanged, replacing Alloc-L with LSP+SLA improves normalized average performance by 1.44, 5.14, and 9.01 percentage points at Kˉ=2,1,0.5\bar K=2,1,0.5, respectively. Since all configurations already use visual-only reduction, these gains isolate the benefit of our sensitivity-profiling and layer-allocation design.
  • Token-level allocation: Keeping Alloc-L unchanged, replacing Alloc-T with TAR improves performance at all three budgets. When combined with LSP+SLA, TAR further improves performance by 0.36, 1.07, and 1.57 percentage points over LSP+SLA+Alloc-T.

Together, these experiments identify the distinguishing contribution of M³MoE: LSP+SLA provides the primary improvement by allocating capacity more effectively across layers, while TAR further refines its distribution among visual tokens. The complete method achieves the highest average performance at every budget, improving upon the visual-only Alloc-MoE baseline by up to 10.58 percentage points. We will include these controlled results and clarify the methodological distinctions in the revised manuscript.

AuthorsOpenReview ↗

Comment 3/5 for Reviewer wdMY

Critical Weakness 3: Shared-backend re-evaluation of prefill speedup

Thank you for this careful analysis. We agree that our original comparison did not provide Default with the same implementation optimizations as M³MoE, conflating routing reduction with backend improvements. We have therefore repeated the evaluation with Default using the same optimized gating and fused expert kernels.

Shared-backend evaluation. Both Default and M³MoE use the same lightweight gating implementation and Triton grouped-GEMM backend on a single H100 PCIe GPU with BF16 and FlashAttention 2. Inputs are identical across execution paths, and measurements are randomly interleaved to reduce ordering effects.

Visual tokensBatchDefault, original (ms)M³MoE, original (ms)Original-path speedupDefault, fused (ms)M³MoE, fused (ms)Fused-path speedup
2561140.6107.81.30×37.238.40.97×
5121144.4112.81.28×42.039.51.06×
10241150.0118.01.27×53.243.81.21×
20481170.2135.71.25×80.860.31.34×
5122161.8132.01.23×67.358.41.15×
5124180.7151.91.19×90.170.91.27×
5128226.3197.71.14×155.0109.91.41×

The measurements use three warm-up runs and 20 synchronized measurements per execution path, with the timing interval covering prefill from prepared multimodal embeddings through final normalization. The visual-token routing budget is Kˉv=0.5\bar K_v=0.5.

Under the original execution path, the speedup decreases from 1.30× to 1.25× as visual-token count increases, and from 1.28× to 1.14× as batch size increases. Under the shared optimized backend, the trend reverses: speedup increases from 0.97× to 1.34× with visual-token count, and from 1.06× to 1.41× with batch size. At 512 visual tokens, the absolute latency saving increases from 2.5 ms to 45.1 ms as batch size increases from 1 to 8.

The 0.97× result indicates that, for small workloads, additional routing overhead can offset the savings from reduced expert computation. Larger workloads better amortize this overhead, allowing the computational savings to translate into greater latency improvements.

Video-prefill validation. We further evaluate three VideoMMMU Perception cases, covering 4–32 frames and batch sizes of 1–8, with both configurations using the same optimized backend.

Batch size4 frames / 720 tokens8 frames / 1,440 tokens16 frames / 2,880 tokens24 frames / 4,320 tokens32 frames / 5,760 tokens
11.11×1.23×1.36×1.39×1.44×
21.26×1.38×1.48×1.49×1.48×
41.40×1.51×1.54×1.51×1.54×
81.52×1.58×1.59×1.57×1.56×

M³MoE achieves speedups across all 20 evaluated settings, ranging from 1.11× to 1.59×. At batch size 1, speedup increases from 1.11× to 1.44× as video length grows from 4 to 32 frames. For workloads with at least 16 frames and batch size 2 or higher, speedup remains between 1.48× and 1.59×. At 32 frames and batch size 8, median prefill latency decreases from 1,479.2 ms to 949.7 ms, corresponding to a 1.56× speedup.

These results confirm that routing reduction provides measurable latency benefits under a shared execution backend, particularly for larger workloads. We will replace the original mixed-backend acceleration claims with these controlled measurements and clearly distinguish routing-related savings from implementation optimizations.

Reviewer_n1ZoOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents M3MoE, a method for reducing visual-token expert activations in multimodal mixture-of-experts models. It first estimates layer-wise sensitivity by measuring changes in output distributions when routing capacity is selectively reduced. Based on these estimates, a layer-level allocator assigns the overall capacity budget, while token-level routing retains the highest-weighted token–expert connections within each layer. Routing for text tokens is kept unchanged. Experiments on Kimi-VL and Qwen3-VL demonstrate competitive performance across eight benchmarks under reduced expert activation, along with measured prefill speedups on Kimi-VL.

Critical strengths
  • The method preserves strong task performance even at extreme sparsity levels across two different model architectures.
  • The allocation strategy is straightforward, with its effectiveness supported by component-wise and calibration ablations. The observed prefill speedups further demonstrate that reducing expert activations can yield practical runtime benefits.
Critical weaknesses
  • The reported acceleration reflects both routing modifications and optimized implementation techniques. Appendix E introduces sparse dispatch, custom gating, and fused Triton kernels, making it difficult to isolate the speedup attributable to the routing strategy without comparisons under a shared execution backend.
  • Modifications to the baselines raise concerns about comparison fairness. FastMMoE disables token pruning, Alloc-MoE adjusts its minimum token capacity, and MoDES aligns budgets only during calibration (Appendix C). The actual evaluation-time budgets and original accuracy–latency trade-offs of these methods remain unclear.
Questions for authors

1. How much of the reported speedup persists when both standard and sparse routing are evaluated using the same optimized backend? Including a simple capacity-reduction baseline under this setting would help clarify the gains.

2. What are the actual expert activation budgets of the modified baselines across benchmarks? Reporting results under their original configurations would provide a clearer assessment of practical performance.

3. Can TAR improve the layer-wise allocation of a competitive baseline, and can SLA enhance its token-level allocation? Such experiments would help distinguish their contributions from existing multi-level allocation approaches.

4. Are the selected expert weights renormalized following TAR? Additionally, how does the model handle tokens for which no routed experts are activated?

6 comments
AuthorsOpenReview ↗

Comment 1/3 for Reviewer n1Zo

Critical Weakness 1 & Question 1: Shared-backend evaluation and the role of implementation optimization

Thank you for raising this important concern. We agree that our original comparison used an optimized sparse execution path for M³MoE but a less optimized execution path for Default, making it difficult to isolate the speedup attributable to routing reduction. We have therefore re-evaluated both configurations using a shared optimized backend and separately report the original-path results.

Shared-backend evaluation. Both Default and M³MoE use the same lightweight gating implementation and Triton grouped-GEMM backend. We use identical cached inputs on a single H100 PCIe GPU with BF16 and FlashAttention 2. The four execution paths are randomly interleaved, with three warm-up runs and 20 synchronized measurements per path. The visual-token routing budget is Kvˉ=0.5\bar {K_v}=0.5.

Visual tokensBatch SizeDefault, original (ms)M³MoE, original (ms)Original-path speedupDefault, optimized (ms)M³MoE, optimized (ms)Shared-backend speedup
2561140.6107.81.30×37.238.40.97×
5121144.4112.81.28×42.039.51.06×
10241150.0118.01.27×53.243.81.21×
20481170.2135.71.25×80.860.31.34×
5122161.8132.01.23×67.358.41.15×
5124180.7151.91.19×90.170.91.27×
5128226.3197.71.14×155.0109.91.41×

Under the original execution path, the speedup decreases from 1.30× to 1.25× as the visual-token count increases from 256 to 2048 (batch size 1), and from 1.28× to 1.14× as the batch size increases from 1 to 8 (512 visual tokens). Under the shared optimized backend, the speedup instead increases from 0.97× to 1.34× with visual-token count, and from 1.06× to 1.41× with batch size. At 512 visual tokens, the absolute latency saving increases from 2.5 ms (batch size 1) to 45.1 ms (batch size 8).

The 0.97× result at 256 visual tokens indicates that, for small workloads, routing overhead can offset the savings from reduced expert computation. As workloads grow, however, the reduction in expert computation translates into increasingly substantial latency savings. This trend is consistent with the original per-expert Python-loop execution being more constrained by fragmented execution and dispatch overhead, whereas grouped GEMM better exposes the benefits of reduced expert computation.

Video-prefill evaluation. We further evaluate video-derived inputs from three VideoMMMU Perception cases, covering 4–32 frames and batch sizes of 1–8. Default and M³MoE again use the same optimized gating and fused expert-execution backend.

Batch size4 frames / 720 tokens8 frames / 1,440 tokens16 frames / 2,880 tokens24 frames / 4,320 tokens32 frames / 5,760 tokens
11.11×1.23×1.36×1.39×1.44×
21.26×1.38×1.48×1.49×1.48×
41.40×1.51×1.54×1.51×1.54×
81.52×1.58×1.59×1.57×1.56×

M³MoE achieves speedups across all 20 evaluated settings, ranging from 1.11× to 1.59×. At batch size 1, increasing the video length from 4 to 32 frames improves the speedup from 1.11× to 1.44×. At 32 frames and batch size 8, median prefill latency decreases from 1,479.2 ms to 949.7 ms, corresponding to a 1.56× speedup.

These shared-backend results demonstrate that the latency benefit of routing reduction persists independently of the execution backend, particularly for larger video-prefill workloads. We will use the shared-backend measurements for our routing-related efficiency claims and report the original-path results separately.

AuthorsOpenReview ↗

Comment 2/3 for Reviewer n1Zo

Critical Weakness 2 & Question 2: Baseline adaptations and actual activation budgets

Thank you for raising these concerns. We agree that baseline adaptations and their realized evaluation-time budgets should be clarified. Our main comparison targets a common visual-token expert budget while retaining default text-token routing capacity. We clarify the adaptations and provide additional evaluations under original baseline components and routing scopes.

Comparison protocol and baseline adaptations.

  1. FastMMoE: We retain expert-activation reduction but disable routing-aware token pruning in the main comparison, isolating expert allocation without changing the visual-token sequence.
  2. Alloc-MoE: We preserve sensitivity profiling and dynamic-programming allocation while restricting capacity reduction to visual tokens. At Kˉv=0.5\bar K_v=0.5, we set Kbase=0K_{\mathrm{base}}=0, as a minimum of one expert per token is incompatible with a sub-unit average budget.
  3. MoDES: We retain its original global/local importance scoring, offline calibration, and frozen-threshold inference, restricting expert skipping to visual tokens.

Actual evaluation-time budgets.

Uniform, Ascend, Descend, and our adapted FastMMoE and Alloc-MoE preserve the same visual-token activation count as M³MoE through explicit budget allocation. MoDES uses an offline-calibrated frozen threshold, so its realized budget may vary across inputs.

We measure its average visual-token activations on each Kimi-VL benchmark:

Target (Kˉv\bar K_v)TextVQAChartQAMMStarMMBenchMMEAI2DDocVQAOCRBench
22.02401.93271.97992.00032.01831.96661.96441.9678
11.01320.96340.98331.00041.03330.96000.95430.9601
0.50.48290.51900.49800.49880.47790.48600.49780.5011

The maximum relative deviation is 4.57%, on DocVQA at Kˉv=1\bar K_v=1. We will report the measured budgets and frozen thresholds, characterizing MoDES as approximately rather than exactly budget-matched.

Additional evaluations under original components and routing scopes.

We evaluate the full FastMMoE pipeline, modality-agnostic Alloc-MoE, and jointly searched MoDES on Qwen3-VL using the same eight benchmarks and full-model normalization baseline. Their configurations are calibrated toward a common total routing budget, using M³MoE's realized budget at Kv=2K_v=2 as the reference.

Budget definition. BvB_v and BtB_t denote the average routed-expert activations per original visual or text token per MoE layer during prefill. All configurations use original, unpruned token counts as the denominator; pruned visual tokens contribute zero activations at subsequent layers but remain in the denominator. BtotalB_{\text{total}} is the token-weighted average across both modalities. These metrics quantify realized routing activations.

MethodConfigurationNormalized performance (%)Visual budget (BvB_v)Text budget (BtB_t)Total budget (BtotalB_{\text{total}})
FastMMoEExpert reduction + token pruning88.592.00182.537
Expert-reduction-only92.73282.536
Alloc-MoEAll-token reduction91.662.5402.5552.541
Visual-only reduction96.60282.536
MoDESJoint text/visual search98.262.2187.7702.714
Visual-only search97.291.97982.517
M³MoEVisual-only allocation98.86282.536

The results reveal three distinctions:

  • FastMMoE: At nearly identical total budgets (2.537 vs. 2.536), expert-reduction-only achieves 92.73%, compared with 88.59% for the full pipeline with token pruning, highlighting the performance difference between these configurations.
  • Alloc-MoE: All-token reduction achieves 91.66% at Btotal=2.541B_{\text{total}}=2.541, while visual-only reduction achieves 96.60% at Btotal=2.536B_{\text{total}}=2.536. Their similar total budgets but different modality-wise allocations highlight the importance of preserving text-token routing capacity.
  • MoDES: Joint text/visual search achieves 98.26% at Btotal=2.714B_{\text{total}}=2.714, versus 97.29% at Btotal=2.517B_{\text{total}}=2.517 for visual-only search. Joint calibration changes modality-wise allocation and yields a higher realized budget than the reference, reflecting the approximate nature of frozen-threshold calibration.

Within the controlled visual-only comparison, M³MoE and adapted FastMMoE and Alloc-MoE have identical realized budgets, while adapted MoDES is approximately budget-matched. These additional evaluations illustrate performance–budget trade-offs under different baseline components and modality-wise allocation strategies. We will incorporate the updated results and configuration details into the revision.

AuthorsOpenReview ↗

Comment 3/3 for Reviewer n1Zo

Question 3: Cross-combination of SLA and TAR with Alloc-MoE

Thank you for this valuable suggestion. We conduct a 2×2 component-swap experiment with Alloc-MoE on Kimi-VL at Kˉ=2,1,0.5\bar {K}=2,1,0.5, where Alloc-L and Alloc-T denote its original layer-level and token-level allocation components, respectively.

All configurations use the same visual-token expert budget and preserve default text-token routing. Alloc-T follows the same minimum-capacity setting in both combinations, including Kbase=0K_{\mathrm{base}}=0 at Kˉ=0.5\bar K=0.5.

Layer allocationToken allocationKˉ=2\bar K=2Kˉ=1\bar K=1Kˉ=0.5\bar K=0.5
Alloc-LAlloc-T97.8992.1886.59
SLAAlloc-T99.33 (+1.44)97.32 (+5.14)95.60 (+9.01)
Alloc-LTAR98.15 (+0.26)92.26 (+0.08)86.71 (+0.12)
SLATAR99.69 (+1.80)98.39 (+6.21)97.17 (+10.58)

The results demonstrate that both components can improve an existing multi-level allocation framework:

  • TAR improves Alloc-MoE's token allocation. Replacing Alloc-T with TAR while retaining Alloc-L improves normalized average performance by 0.26, 0.08, and 0.12 percentage points at Kˉ=2,1,0.5\bar K=2,1,0.5, respectively.
  • SLA improves Alloc-MoE's layer allocation. Replacing Alloc-L with SLA while retaining Alloc-T yields larger gains of 1.44, 5.14, and 9.01 percentage points, respectively.
  • Combining SLA and TAR achieves the best performance. Compared with SLA + Alloc-T, adding TAR provides further gains of 0.36, 1.07, and 1.57 percentage points across the three budgets.

These controlled substitutions distinguish the contributions of the two components beyond their original combination in M³MoE. In particular, SLA provides the larger improvement, while TAR contributes a consistent complementary gain under both layer allocators. We will include the complete crossover results and per-benchmark breakdown in the revision.


Question 4: Routing-weight renormalization and zero-expert tokens

Thank you for requesting this clarification. TAR does not apply additional renormalization after expert selection. As defined in Sec. 4.4, routing weights are normalized over the original Kmax⁡K_{\max} selected experts. TAR masks discarded token–expert assignments while preserving the original normalized coefficients of retained experts, together with any model-specific routing scale. Renormalizing the retained weights would amplify their aggregate contribution, changing the original mixing coefficients.

When no routed expert is activated, the routed-expert contribution is zero. Let hth_t denote the residual input to the MoE sublayer and sts_t the shared-expert contribution computed using the model's original operations. The residual update is:

\begin{cases} h_t+s_t, & \text{Kimi-VL (with shared experts)},\\\\ h_t, & \text{Qwen3-VL (without shared experts)}. \end{cases}$$ Thus, for Qwen3-VL, a token with zero routed experts receives an identity update at the residual MoE sublayer. The token remains in the sequence and proceeds through subsequent layers normally. For Kimi-VL, the shared-expert contribution is retained even when no routed experts are activated. We will clarify the routing-weight normalization convention and zero-activation behavior in the revised manuscript.

Comments on the paper

2 comments

Thank you for your careful reassessment and for recognizing the additional experimental evidence. We appreciate your remaining concerns and would like to briefly clarify the contribution and its practical scope.

Technical contribution. We acknowledge that adaptive expert allocation and multi-level budget allocation have established precedents. Our contribution lies in how visual-token routing sensitivity is measured and translated into a fixed-budget allocation: LSP measures output-distribution changes under isolated layer perturbations, while SLA aggregates these sensitivity profiles for capped proportional allocation rather than optimizing an additive sensitivity objective. The controlled comparison using the same sensitivity matrix isolates the contribution of the allocation rule, and the component-swap experiments with Alloc-MoE further demonstrate benefits beyond visual-only adaptation.

Practical scope. We agree that prefill acceleration provides smaller overall benefits when long autoregressive decoding dominates inference time. Our intended scope is therefore multimodal prefill, particularly workloads with substantial visual inputs and short outputs, rather than universal acceleration across generation lengths. On Qwen3.5-122B-A10B, the shared-backend experiments show prefill speedups of 1.12–1.19×, 1.21–1.27×, and 1.24–1.35× at average visual-token expert budgets of 2, 1, and 0.5, respectively, with the full-model baseline receiving the same backend optimizations.

We will incorporate these distinctions into the revision, clearly separating the methodological contribution from its precedents and the demonstrated prefill benefits from claims about end-to-end generation.

AuthorsOpenReview ↗

Summary of Revisions

Summary of Revisions

We thank the reviewers for their constructive feedback. We have revised the manuscript to clarify the methodological contribution, strengthen the controlled comparisons, and establish the scope of the empirical conclusions. The main revisions and findings are summarized below.

  1. Methodological distinctions and component contributions. (Reviewer n1Zo, Reviewer wdMY, Reviewer Yx6C)

    • Sensitivity definition and profiling context. We clarified that LSP changes both the sensitivity measure and the perturbation setting relative to Alloc-L. Alloc-L profiles perplexity with downstream layers constrained to Top-1; LSP applies isolated visual-routing perturbations while preserving default routing elsewhere, and measures JS divergence from the full model’s first-token predictive distribution. This evaluates distributional fidelity under targeted visual-capacity reduction in the context of the otherwise unchanged model.

    • Layer-allocation criterion. SLA aggregates sensitivity across reduction levels into layer priorities and performs capped, budget-preserving proportional allocation. Alloc-L instead uses DP to minimize an additive sensitivity-cost objective. This distinction concerns how sensitivity profiles determine the allocation, beyond the choice of optimization procedure. With identical JS profiles, capacity constraints, and budgets, and TAR disabled, SLA improves normalized performance on Qwen3-VL by 0.40–2.24 percentage points, isolating the contribution of the allocation rule.

    • Controlled component attribution. Under the same visual-only routing scope and expert budgets, replacing Alloc-L with LSP+SLA while retaining Alloc-T improves normalized performance on Kimi-VL by 1.44–9.01 percentage points. These results identify the combined profiling-and-allocation branch as the larger contributor, with further gains from token redistribution. They also show that the improvement extends beyond simply adapting Alloc-MoE to visual tokens.

  2. Routing rationale and interpretation. (Reviewer wdMY)

    • We clarified that layer sensitivity measures tolerance to the profiled capacity reduction, and that retained routing mass is an aggregate surrogate with a local truncation-error bound.
    • The coverage-aware control improves coverage within positive-budget layers but yields 0.24–1.32 points lower normalized performance. This demonstrates that improving token coverage does not necessarily improve task performance.
    • We specified that retained routing coefficients are preserved without post-selection renormalization. Zero-routed-expert tokens remain in attention, with residual and any shared-expert computation retained.
  3. Baseline fairness and realized budgets. (Reviewer n1Zo, Reviewer wdMY, Reviewer Yx6C)

    • We documented the baseline adaptations and expanded the native-configuration table to report visual, text, and overall prefill routing budgets alongside performance, using original token counts and accounting for pruned tokens.
    • The revised comparison distinguishes exactly matched budgets for adapted FastMMoE, adapted Alloc-MoE, and M³MoE; approximate matching for MoDES; and different operating points for native configurations.
  4. Efficiency attribution. (Reviewer n1Zo, Reviewer wdMY, Reviewer Yx6C)

    • Primary speedup comparisons now use shared execution backends, separating routing-reduction benefits from backend optimizations. Shared-backend speedups range from 0.97–1.41× on Kimi-VL image-prefill workloads and 1.12–1.35× on Qwen3.5-122B-A10B.
  5. Broader evaluation and its limits. (Reviewer n1Zo, Reviewer wdMY, Reviewer Yx6C)

    • We added VideoMMMU evaluations across Perception, Comprehension, and Adaptation. M³MoE achieves the highest normalized average among the compared configurations at all three tested budgets.
    • On Qwen3.5-122B-A10B, gains over Alloc-MoE are positive: 0.37–0.74 points. These results support applicability across the evaluated models.
    • We added FastV and VisionZip comparisons. M³MoE retains higher normalized performance in the reported settings, while the revision explicitly distinguishes routing-budget comparisons from matching attention, sequence, or total-model computation.

For ease of review, changes to the original text are highlighted in yellow, while newly added content, including supplementary appendix material, is shown in red.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.