acceptodds
Desk rejected at ICLR 2027

DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inference

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal mixture-of-experts (MoE) models combine sparse expert activation with visual-language capabilities, yet their inference remains costly because long visual-token sequences repeatedly incur attention, routing, dispatch, and expert-MLP computation. Existing methods typically compress either the token or expert dimension, leaving redundancy along the other. Our analysis reveals two complementary regularities: the depth required for visual propagation varies across inputs, while text-token routing exhibits concentrated and recurrent expert-importance patterns. Based on these observations, we propose DecoMoE, a two-dimensional structured compression framework that decouples visual propagation from expert computation. The Sample-Adaptive Visual Boundary (SAVB) predicts an input-dependent visual-exit layer at which the visual-token block is removed. The Routing-Calibrated Expert Prefix (RCEP) reorders experts offline using text-token routed mass and, from this predicted exit layer onward, retains at each MoE layer the shortest contiguous prefix covering a target routed-mass fraction. We evaluate DecoMoE on Qwen3-VL-MoE and InternVL3.5-30B-A3B across six benchmarks. On Qwen3-VL-MoE, DecoMoE retains 97.91% of dense-baseline performance while reducing computation from 27.06 to 16.73 TFLOPs and latency from 0.44 to 0.26 seconds, yielding a 1.69 speedup.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MoE, MLLM, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VkMrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DecoMoE, a two-dimensional structured compression framework for efficient multimodal MoE inference. The key observation is that existing methods compress either the visual-token dimension or the expert dimension, leaving redundancy in the other. DecoMoE decouples these two decisions. The Sample-Adaptive Visual Boundary (SAVB) predicts an input-dependent visual-exit layer from pooled text states and removes the complete visual-token block before self-attention at that layer. The Routing-Calibrated Expert Prefix (RCEP) offline reorders experts by mean text-token routed mass and, from the predicted exit layer onward, retains at each MoE layer the shortest physically contiguous expert prefix covering a target routed-mass fraction. Experiments on Qwen3-VL-MoE and InternVL3.5-30B-A3B across six benchmarks show that DecoMoE retains 97.91% and 97.87% of dense performance while reducing computation from 27.06 to 16.73 and 9.86 to 5.97 TFLOPS, with measured speedups of 1.69× and 1.41×. Ablations validate SAVB, RCEP, and their combination.

Critical strengths
  1. DecoMoE decouples visual propagation from expert computation, combining an input-dependent visual exit (SAVB) with a text-routing-calibrated expert prefix (RCEP). This is consequential because prior methods compress only one dimension, leaving substantial redundancy in the other. The reported 1.69× speedup on Qwen3-VL-MoE with 97.91% average performance retention, and 1.41× on InternVL3.5-30B-A3B, shows that the two mechanisms complement each other and translate into real latency reductions, not just theoretical FLOP savings.

  2. SAVB removes a contiguous visual-token block, and RCEP restricts computation to a contiguous expert prefix. This is consequential because it reduces irregular selection, fragmented dispatch, and memory movement, which are common reasons why theoretical FLOP savings fail to become real latency gains. The TTFT and TPOT reductions of 30.8% each support the practical value of this design.

Critical weaknesses
  1. Although the average retention is 97.91% on Qwen3-VL-MoE, HallusionBench drops from 74.97 to 71.92 and MMMU drops by about 2 points; on InternVL3.5-30B-A3B, HallusionBench drops from 69.70 to 65.51. This is consequential because the average metric hides tasks where the method may be unacceptable, and the paper does not analyze why some tasks are more fragile or how to mitigate this.

  2. The paper admits that RCEP and SAVB calibration samples are not enforced to be disjoint, and that POPE/MME results including calibration IDs are transductive. This is consequential because the method's main selling point is that it can be calibrated offline and applied to new inputs. If calibration and evaluation overlap, or if the calibration distribution leaks into reported numbers, the evidence for out-of-distribution generalization is weakened.

Questions for authors

1.Can you provide sensitivity analyses for SAVB's NLL percentile, gate threshold, and candidate-layer interval? How stable is performance under these choices?

  1. Were FastV, SparseVLM, DyVTE, and FastMMoE tuned with the same calibration data and budget as DecoMoE on each backbone? Can you report their best-tuned results under identical conditions?

  2. How does expert reordering interact with expert parallelism and load balancing? Does deploying DecoMoE require re-exporting the model or modifying existing inference kernels?

  3. Why does HallusionBench degrade more than other benchmarks, and can this be mitigated by a more conservative SAVB or RCEP setting?

  4. The online RCEP step computes full-pool router logits, then recomputes softmax and Top-k inside the selected prefix. What is the exact overhead of this two-pass routing, and is it included in the reported latency?

  5. Does DecoMoE generalize to other multimodal MoE models such as DeepSeek-VL2 or Kimi-VL, and to video or high-resolution inputs where visual-token counts are much larger?

  6. Can the authors report calibration and evaluation on strictly disjoint POPE/MME splits, given that the appendix notes potential overlap between SAVB and RCEP calibration samples?

11 comments
AuthorsOpenReview ↗

Response to Reviewer VkMr (W1 & W2): Task Sensitivity and Calibration–Evaluation Separation

W1: Task Sensitivity and Performance Degradation

HallusionBench and MMMU place stronger demands on precise visual evidence and deep cross-modal reasoning, which may make them more sensitive to visual exit and expert compression. We thank the reviewer for highlighting this important issue. We agree that the 3.05- and 4.19-percentage-point drops on HallusionBench for Qwen and InternVL, respectively, and the approximately two-point drop on MMMU deserve separate analysis rather than being obscured by average retention.

The potential risks arise from premature removal of visual evidence and insufficient expert coverage. HallusionBench requires answers grounded in actual visual evidence rather than language priors, while MMMU involves charts, specialized symbols, and multi-step reasoning. If SAVB triggers before critical details are sufficiently encoded into text states, subsequent layers lose direct access to that visual evidence. Meanwhile, RCEP orders experts using average routed mass across calibration examples and dynamically selects a prefix, but high cumulative routed mass does not guarantee coverage of every expert important to a specialized question. Appendix A.7.3 (Case 3, Figure 11) provides a relevant observation: for the same fire-hydrant image, presence recognition, attribute explanation, and detailed analysis including identification-number recognition require reported propagation depths of 18, 22, and 42, respectively. This illustrates that visual-propagation requirements depend on the task, not merely the image or token count.

Our focus is to improve inference efficiency while preserving performance within task-acceptable limits; what constitutes an acceptable loss depends on the application. The current method keeps the multimodal backbone frozen. Future work could explore adapting the LM head to compressed hidden states while freezing the rest of the network, investigating whether this mitigates compression-induced output-distribution shifts and improves the performance–efficiency trade-off on sensitive tasks.

W2: Separation Between Calibration and Evaluation

None of the six main evaluation benchmarks is used for calibration, and the POPE/MME test subsets are also separate from calibration samples. We thank the reviewer for identifying the insufficiently clear description. We first partition POPE/MME into calibration and test subsets, then use the calibration portion to construct SAVB forced-exit supervision and collect RCEP text-routing statistics. SAVB and RCEP may share calibration samples, but evaluation excludes the union of samples used by either component. Thus, overlap between the two calibration sets is distinct from calibration–test overlap. MMBench, ScienceQA, AI2D, AOKVQA, MMMU, and HallusionBench are not used in these calibration procedures.

Including POPE in Table 2 can indeed obscure the distinction between calibration sources and evaluation subsets, reflecting a shortcoming in our presentation and protocol description. Although that result uses a separate test subset, we will remove the POPE results from this ablation and recompute the corresponding aggregate metrics. We will also revise Appendix A.4 to clarify the sample-ID boundaries between calibration and evaluation and the permitted overlap between SAVB and RCEP calibration sets.

AuthorsOpenReview ↗

Response to Reviewer VkMr (Q1): Sensitivity to SAVB Hyperparameters

Q1: Sensitivity to SAVB Hyperparameters

Performance is relatively stable across NLL percentiles P60–P99; the gate uses a fixed 0.5 binary decision threshold; and a candidate-layer interval of 4 reduces execution overhead while closely matching denser checks in accuracy. We thank the reviewer for this suggestion, which helps clarify the rationale and scope of our default settings.

The NLL percentile reduces noise in exit supervision. We derive the threshold from the distribution of reference-answer first-token NLL at each calibration sample's first correct exit layer. A sample-layer pair is labeled exit-safe only when both task correctness and the NLL condition hold, reducing the influence of correct but insufficiently confident predictions. Default P95 balances filtering the high-NLL tail against retaining sufficient supervision. In Table R3-A, mean accuracy across P60–P99 ranges from 89.66% to 89.85%. P95 achieves 89.83%, differing from P90 and P99 by only 0.02 and 0.01 percentage points, respectively, supporting its placement within a relatively stable region.

Table R3-A. Qwen performance under different NLL percentiles (%).

NLL percentileMMBenchScienceQAMean
Baseline88.9890.9089.94
P2081.6188.8185.21
P4087.9590.9989.47
P6088.4291.0889.75
P7588.3091.0289.66
P9088.4891.2189.85
P95 (default)88.5491.1189.83
P9988.5291.1189.82

The candidate-layer interval controls decision frequency and available exit positions. Intervals 1, 2, and 4 achieve accuracies of 72.03%, 72.03%, and 71.92%, respectively. Compared with interval 1, interval 4 loses only 0.11 percentage points, reduces mean checks from 27.506 to 7.637, and lowers TTFT and generation latency on the timing samples by approximately 10.0% and 7.5%. Increasing the interval to 16 provides only another 4.2% TTFT reduction relative to interval 4, while accuracy drops by 2.20 percentage points. Thus, the default interval of 4 reduces the number of checks while closely matching denser checking in accuracy, offering a favorable performance–efficiency trade-off.

Table R3-B. HallusionBench interval study: original-path accuracy on 951 questions and optimized cached latency on five shared questions.

IntervalAccuracy (%)Mean exit (0-based)Mean checksTTFT (ms)TPOT (ms/token)Generation Latency (ms)
172.0329.45027.506127.85740.219168.437
272.0329.46014.267119.45640.747160.560
4 (default)71.9229.4757.637115.09240.341155.793
870.1431.9614.718112.12040.394152.871
1669.7234.9873.044110.30840.172150.836

The gate threshold is fixed at 0.5. Unlike the NLL percentile used to construct training labels, this threshold applies to the inference-time softmax score of the exit-safe class: exit occurs when the score reaches 0.5. Exit depth therefore depends on the learned classification boundary and input state rather than a manually specified uniform depth. We did not independently sweep this threshold, so the results above should not be interpreted as evidence of insensitivity to the gate threshold.

AuthorsOpenReview ↗

Response to Reviewer VkMr (Q2–Q4): Baseline Settings, Parallel Deployment, and Task Sensitivity

Q2: Baseline Configurations and Computational Budgets

Our main comparison targets performance and efficiency at comparable theoretical FLOPs, rather than a uniform hyperparameter-search budget. FastV and SparseVLM retain approximately 40–50% of visual tokens, using relatively generous retention budgets to avoid overly aggressive pruning. DyVTE and SAVB construct their respective exit supervision from POPE/MME; FastV, SparseVLM, and FastMMoE do not require training an exit classifier.

Main-experiment configurations and computational costs.

MethodQwen3-VL-MoE configurationFLOPs (TFLOPs)InternVL3.5 configurationFLOPs (TFLOPs)
BaselineNo compression27.06No compression9.86
FastV50% retention; pruning-layer parameter 416.6548% retention; pruning-layer parameter 65.86
SparseVLM44% final retention16.2140% final retention5.03
DyVTEExit threshold 0.142; all experts retained18.29Not reported in the main table-
FastMMoEr=25%, l_v=6, k_v=518.88r=25%, l_v=6, k_v=58.02
DecoMoESAVB + RCEP16.73SAVB + RCEP5.97

For FastMMoE, rr denotes visual-token retention, lvl_v the starting layer for visual-expert compression, and kvk_v the reduced visual routing width. FastV and SparseVLM operate at computational costs relatively close to DecoMoE, while FastMMoE uses more FLOPs than DecoMoE on both backbones.

Q3: Expert Reordering and Parallel Deployment

Expert reordering is compatible with expert parallelism and requires neither retraining nor mandatory model re-export, but the runtime must support prefix selection, within-prefix routing, and restricted expert execution. Consistently permuting router outputs and their corresponding expert parameters mathematically preserves the uncompressed model's function. This transformation can be performed during loading or saved in a reordered checkpoint. Existing matrix-multiplication kernels can be reused; fused MoE backends require corresponding adaptations to routing and dispatch interfaces so that experts outside the prefix are excluded from computation.

For expert parallelism, reordering itself should be distinguished from prefix restriction. If expert device ownership is preserved and IDs and dispatch mappings are updated consistently, reordering alone does not change device workloads; prefix restriction and rerouting can change their distribution. A contiguous prefix in the global expert order does not require all retained experts to reside on one GPU. Distributed deployment can assign prefix experts across devices using estimated post-compression token loads, with compact local layouts on each device. This provides a deployment strategy for retaining parallel execution and addressing load concentration, while load balancing still depends on expert placement and scheduling rather than being guaranteed by reordering alone.

Q4: Sensitivity on HallusionBench

HallusionBench's degradation may reflect its reliance on fine-grained visual evidence and correction of language priors. If SAVB exits before relevant evidence is sufficiently incorporated into text states, later layers cannot directly revisit that visual information. Likewise, retaining high cumulative routing mass does not guarantee coverage of every question's critical experts. Average retention therefore cannot replace task-level analysis. Our objective is to improve efficiency while preserving acceptable performance, rather than guarantee lossless compression on every task. A larger RCEP expert budget provides broader expert coverage, although performance improvements require task-specific validation. As future work, we will also explore lightweight LM-head adaptation with the backbone frozen, investigating whether it can mitigate compression-induced shifts in the output distribution.

Reviewer_HmkVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets the high inference cost of multimodal MoE models caused by redundant computation along both the visual-token and expert dimensions. It proposes DecoMoE, which combines a Sample-Adaptive Visual Boundary (SAVB) to determine when visual tokens can be removed with a Routing-Calibrated Expert Prefix (RCEP) to reduce expert computation after the visual exit. SAVB is trained with forced-exit supervision while keeping the backbone frozen, whereas RCEP is calibrated offline using text-token routing statistics and dynamically selects a compact expert prefix during inference. Experiments across multiple multimodal MoE models show that DecoMoE substantially improves inference efficiency while largely preserving baseline performance.

Critical strengths
  1. The paper is clearly written and easy to follow, and the figures are helpful and well-illustrated for understanding the proposed method.
  2. The proposed framework is relatively practical, as it keeps the multimodal backbone frozen and introduces only lightweight calibration and structured modifications for efficient inference.
  3. The core motivation is clear and intuitive, which jointly considers redundancy along the visual-token and expert dimensions.
Critical weaknesses
  1. Insufficient experimental validation and analysis. My main concerns lie in the experimental setup and the analysis of the empirical results. The current evaluation is not sufficiently challenging, and the analysis lacks the depth needed to convincingly support the claimed effectiveness and efficiency of DecoMoE. Specifically:

    (a) Benchmark and baseline selection. Many benchmarks in Table 1 appear relatively easy for the evaluated backbone, with baseline performance above 85% on MMBench, ScienceQA, AI2D, and AOKVQA. Under more challenging visual reasoning scenarios, visual tokens would remain necessary until much deeper layers, or even throughout the entire network, potentially delaying SAVB’s exit decision and limiting computational savings from both visual-token removal and RCEP. Evaluation on harder benchmarks would help establish whether DecoMoE can maintain its performance–efficiency trade-off under such conditions.

    Moreover, under these relatively saturated evaluation settings, the performance advantage of DecoMoE over existing token-compression methods (FastV and SparseVLM) remains limited. Comparisons with more recent approaches, such as [1–3], would help better establish the effectiveness of the proposed method.

    (b) Granularity of the latency analysis. DecoMoE’s computational savings depend on the layer at which SAVB triggers and RCEP’s retained expert count. However, the TTFT and TPOT comparison reports only aggregate values for each method, without analyzing how these adaptive decisions affect the actual speedup. It would be useful to report latency or speedup conditioned on different visual-exit depths and expert-prefix sizes.

  2. Could the authors clarify SAVB’s methodological differences from DyVTE in more detail? Clarifying this distinction is important for assessing the novelty of SAVB and the overall method.

  3. In the main results table, FastV and SparseVLM accelerate Qwen3-VL-MoE but are much slower than the dense baseline on InternVL3.5. Could the authors explain why these methods exhibit opposite latency behavior across the two backbones?

[1] VisionZip: Longer is Better but Not Necessary in Vision Language Models. CVPR 2025.
[2] MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping. CVPR 2026.
[3] MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs. ICLR 2026.

Questions for authors

See weakness

6 comments
AuthorsOpenReview ↗

Response to Reviewer HmkV (W1 Part 1): Challenging Tasks and Additional Baselines

W1 Part 1: Challenging Tasks and Additional Baselines

The additional results show that DecoMoE preserves performance on a challenging video reasoning task and a smaller MoE architecture, with lower latency than the added baselines under the evaluated configurations. We thank the reviewer for these suggestions. We agree that harder tasks are needed, but high baseline performance does not automatically make compression easy: for accuracy metrics, fewer existing errors leave less room for corrections to offset compression-induced errors. Moreover, selection, merging, data movement, and expert dispatch can offset theoretical FLOPs savings.

Table R2-A: Qwen3-VL-30B-A3B on QuantiPhy, a video-based quantitative physical reasoning benchmark [1]. S2/D2 denote 2D inference with static/dynamic priors, and S3/D3 their 3D counterparts. Static priors provide size references; dynamic priors provide velocity or acceleration references. These tasks require physical quantities in real-world units from spatial, temporal, and depth evidence, not just category recognition. Baseline Macro MRA is only 38.36%. Video inputs typically contain more visual tokens than single images, offering greater scope for compression: FastV, VisionZip, and MMTok reduce TTFT to 0.44-0.47×, but TPOT remains essentially unchanged. By coupling visual exit with structured expert-prefix execution, DecoMoE retains 98.91% performance while reducing TTFT/TPOT/generation latency to 0.41×/0.66×/0.53×.

Table R2-A. Qwen3-VL-30B-A3B on QuantiPhy.

MethodS2D2S3D3Macro MRARetentionTTFT (ms)TPOT (ms/token)Generation Latency (ms)
Baseline36.5635.6834.1947.0238.36100.00473.362 (1.00×)62.986 (1.00×)801.266 (1.00×)
FastV31.8730.0032.3346.1735.0991.48207.258 (0.44×)62.951 (1.00×)546.727 (0.68×)
VisionZip37.1936.7628.6047.4537.5097.75221.647 (0.47×)63.025 (1.00×)623.214 (0.78×)
MMTok38.1237.3033.0242.9837.8698.68211.144 (0.45×)62.913 (1.00×)624.874 (0.78×)
DecoMoE34.6934.0535.3547.6637.9498.91192.607 (0.41×)41.880 (0.66×)421.602 (0.53×)

Table R2-B: DeepSeek-VL2-Tiny across six benchmarks. Tiny uses a 3B-class language backbone and approximately 1.0B activated parameters overall. All methods retain nearly all baseline performance, but only DecoMoE achieves net acceleration in these sampled measurements, retaining 99.59% performance with latency ratios of 0.60×/0.48×/0.58×. Preserving task performance and accelerating execution are distinct: Tiny's smaller single-image workload may make selection and dispatch overhead harder to amortize. With KV caching, visual-token removal shortens the attention context but does not directly reduce each new text token's expert MLP work. These observations concern our evaluated implementations, not an inherent inability of the comparison methods to accelerate. Due to reproduction costs, we evaluated MoDES on Tiny rather than Qwen.

Table R2-B. DeepSeek-VL2-Tiny on six benchmarks.

MethodMMBench ENScienceQAAI2DAOKVQAMMMUHallusionBenchAvg. RetentionAvg. TTFT (ms)Avg. TPOT (ms/token)Avg. Generation Latency (ms)
Baseline79.7887.8966.8779.8335.8959.62100.0079.841 (1.00×)15.608 (1.00×)98.065 (1.00×)
VisionZip79.0387.7466.1979.2136.0060.3699.7890.367 (1.13×)15.619 (1.00×)108.610 (1.11×)
MMTok79.1487.2768.8878.8636.8960.15100.66116.995 (1.47×)15.698 (1.01×)135.311 (1.38×)
MoDES79.5187.7967.0379.7435.6759.6299.8480.759 (1.01×)19.199 (1.23×)103.413 (1.05×)
DecoMoE77.7287.8466.6578.7835.7860.8899.5948.102 (0.60×)7.417 (0.48×)56.760 (0.58×)

[1] Puyin L, Xiang T, Mao E, et al. QuantiPhy: A quantitative benchmark evaluating physical reasoning abilities of vision-language models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026: 33174-33184.

AuthorsOpenReview ↗

Response to Reviewer HmkV (W1 Part 2): Latency Sensitivity to Visual-Exit Depth and Expert-Prefix Size

Our controlled measurements show that earlier visual exit primarily reduces prefill latency, while smaller expert prefixes reduce cached-decoding latency in the evaluated Dense-BMM implementation. We thank the reviewer for requesting this finer-grained analysis. We use real Qwen3-VL-30B-A3B weights, identical synthetic visual embeddings representing N=800N=800 decoder tokens, a fixed text prompt, and exactly M=10M=10 generated tokens. All measurements use BF16, batch size 1, native KV caching, and repeated measurements with one seed.

Visual-exit depth. We vary the forced exit layer while retaining dynamic RCEP with p=0.7p=0.7, including online prefix-selection costs. Earlier exit avoids propagating the long visual sequence through more layers: moving from L=40L=40 to L=8L=8 reduces Decoder TTFT from 173.329 to 104.826 ms and generation latency from 347.515 to 280.626 ms. TPOT remains within 18.974–19.515 ms/token, consistent with the similar observed decode prefix sizes. Here, LL is zero-based, and eviction occurs after block LL.

Table R2-C. Exit-depth sensitivity with dynamic RCEP (p=0.7p=0.7).

Exit Layer LLMean Decode KKDecoder TTFT (ms)TPOT (ms/token)Generation Latency (ms)
844.04104.82619.515280.626
1640.83121.12218.974292.406
2441.81139.59019.028313.152
2842.25149.01819.187321.239
3244.52156.83019.289330.380
4044.60173.32919.355347.515

Expert-prefix size. Fixing L=28L=28, we vary the candidate prefix from K=8K=8 to K=128K=128, while every token still selects eight experts. TPOT increases monotonically from 11.835 to 24.823 ms/token, and generation latency increases from 246.643 to 365.316 ms. Decoder TTFT changes little because the first 29 prefill blocks retain all visual tokens and the full expert pool; changing KK affects only the subsequent short text sequence during prefill, but all layers during decode.

Table R2-D. Expert-prefix sensitivity at fixed L=28L=28.

Expert Prefix KKDecoder TTFT (ms)TPOT (ms/token)Generation Latency (ms)
8139.98711.835246.643
16141.92513.202261.207
32143.25815.064279.059
64140.89318.265305.430
128141.60724.823365.316
AuthorsOpenReview ↗

Response to Reviewer HmkV (W2 & W3): SAVB vs. DyVTE and Backbone-Dependent Latency

W2: Methodological Differences Between SAVB and DyVTE

SAVB differs from DyVTE in its exit supervision and in how visual exit enables joint token-and-expert structured compression, rather than in the lightweight classifier architecture itself. We thank the reviewer for requesting this clarification. Both methods use text-state-driven lightweight gates and supervision derived from predictions following forced visual exit. DecoMoE connects this sample-adaptive boundary to subsequent expert compression, coordinating the two compression dimensions.

For supervision, DyVTE combines agreement with the uncompressed model's answer and a prediction-uncertainty constraint; SAVB combines task correctness with data-driven confidence calibration. SAVB evaluates each forced-exit answer using the dataset's evaluation rule and derives a percentile threshold from reference-answer first-token NLL values at calibration samples' first correct exit layers. Each sample-layer pair is labeled exit-safe only if the answer is correct and its NLL satisfies the threshold. Later layers are evaluated individually rather than assumed safe after the first correct exit. Thus, the target is calibrated task-correct exit, not merely agreement with an earlier prediction.

At the framework level, DyVTE decides when visual tokens exit; DecoMoE additionally uses that boundary to activate expert compression. After SAVB removes the complete visual block, RCEP uses offline text-routing statistics to consistently reorder expert parameters and router dimensions. It then selects a contiguous prefix covering the target activated routing mass and reroutes within it while preserving the original Top-kk width. The boundary therefore determines both sequence shortening and the computation targeted by expert compression. In Table 2 of the original submission, adding RCEP to SAVB increases average performance retention from 97.41% to 97.83% and reduces reported CUDA time from 0.30 s to 0.26 s, supporting their complementary roles.

W3: Different Latency Behavior Across Backbones

The contrasting latency behavior is closely related to how the evaluated backbones organize expert computation: Qwen-type Dense-BMM and InternVL-type expert-wise sparse execution respond differently to token reduction at the hardware level. We thank the reviewer for highlighting this observation. We explain this distinction through the execution paths used in our experiments, rather than inferring acceleration from theoretical FLOPs alone.

In the evaluated Qwen-type Dense-BMM path, input tokens are expanded across the expert batch, batched expert matrix multiplications are performed, and sparse routing weights are applied afterward. Logical Top-kk activation therefore does not mean that this path computes only the selected token-expert pairs. Removing visual tokens shortens the matrix inputs across the entire expert batch, directly eliminating this batched computation. FastV and SparseVLM can consequently obtain substantial savings by shortening the sequence in this execution path.

InternVL3.5's expert-wise sparse path already computes only selected token-expert pairs, distributing the workload across separate expert matrix operations. Token pruning reduces the tokens assigned to each expert, but does not necessarily reduce the number of active experts or expert invocations. The compressed workload may therefore retain approximately the same number of expert calls while making each matrix smaller, reducing GPU parallelism and matrix-operation utilization; dispatch and aggregation costs need not decrease proportionally either. For some matrix shapes, this loss of execution efficiency can offset the arithmetic savings or even increase latency. The observed contrast is therefore consistent with the different expert execution organizations: shortening regular batched computation and further shrinking fragmented expert-wise computation do not yield equivalent acceleration.

Reviewer_968KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DecoMoE, a framework for accelerating multimodal mixture-of-experts models through sample-adaptive visual-token removal and structured expert restriction. The Sample-Adaptive Visual Boundary (SAVB) uses textual hidden states to predict when the complete visual-token block can be removed. After visual exit, the Routing-Calibrated Expert Prefix (RCEP) restricts routing to a contiguous prefix of experts, ordered offline using text-token routing statistics. Experiments on Qwen3-VL-30B-A3B and InternVL3.5-30B-A3B cover six evaluation benchmarks. The paper reports approximately 98% average performance retention and yields 1.69x speedup on Qwen3-VL-30B-A3B.

Critical strengths

1.The framework addresses an important efficiency problem with a concrete execution-aware design. Removing the complete visual-token block and restricting experts to an offline-reordered contiguous prefix provide a clear implementation strategy. Offline reordering avoids repeatedly rearranging expert parameters during inference. This makes the approach potentially useful within execution settings that benefit from contiguous expert evaluation.

2.The experiments show better performance across models and datasets. The main evaluation covers two multimodal models and six benchmarks distinct from the stated POPE/MME calibration sources. The component ablation also suggests an incremental benefit from RCEP: in Table 2, adding RCEP to SAVB reduces reported latency from 0.30 to 0.26 seconds while increasing average retention from 97.41% to 97.83%. Although the calibration overlap in this ablation needs clarification, the results support further investigation of the proposed combination.

Critical weaknesses
  1. The efficiency claim is not established under sparse expert execution. Appendix A.3 explicitly states that the evaluated Qwen path computes a dense batched BMM over every physically retained expert and applies sparse routing weights afterward. Accordingly, the baseline expert computation is charged for all 128 experts per token, while RCEP is charged for its retained prefix. However, Section 3.3 specifies that each token is routed to exactly eight experts both before and after compression. Under an implementation that computes only the selected token–expert pairs, the dominant expert-MLP arithmetic scales with the number of tokens times the per-token routing width, rather than the size of the available expert pool. Restricting the pool therefore does not, by itself, reduce this arithmetic. RCEP additionally computes full router logits, aggregates routing mass, and repeats selection within the prefix. Concentrating routing could still improve parameter access or execution efficiency, but the paper does not isolate these benefits under sparse execution. The current results establish acceleration for the evaluated dense-BMM path; they do not yet establish comparable benefits for efficient sparse MoE inference. This distinction directly affects the significance and practical scope of the paper’s main contribution.

  2. The incremental contribution over DyVTE and the necessity of the proposed calibration choices are insufficiently isolated. SAVB shares more than its input features with DyVTE. DyVTE already uses a binary MLP over pooled preceding text states and the final text state, with supervision combining forced-exit answer agreement and prediction uncertainty. SAVB’s differences primarily concern the correctness criterion, percentile calibration of first-token NLL, and candidate-layer schedule. Although DyVTE is included as a baseline, the paper does not adequately document whether it receives comparable calibration data and tuning, or compare DecoMoE against DyVTE combined with the same RCEP module. Likewise, comparing RCEP ordering against native and random ordering demonstrates the value of importance-based ordering, but does not establish that text-only calibration is superior to mixed-modality calibration.

Questions for authors
  1. Explain execution backend and source of speedup: Under the same sparse MoE backend, hardware, precision, batch size, and input/output lengths, what are the latencies of the baseline, SAVB-only, and full DecoMoE? Which operations account for RCEP’s incremental savings, and how do these savings compare with its routing and prefix-selection overhead?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 968K (W1): RCEP Speedup and Execution Mechanisms

W1: RCEP's acceleration and execution mechanisms

RCEP provides measurable expert-execution savings. We thank the reviewer for highlighting the distinction between per-token routing width and candidate-pool size. Through more detailed experiments and theoretical analysis, we clarify the difference between theoretical arithmetic and actual execution costs.

Tables R1-A and R1-B show complementary module- and system-level benefits. With per-token activated expert count unchanged, dynamic RCEP (p=0.70p=0.70) reduces expert GPU time by 29.06% and 29.03% in the two backends. Prefix-selection GPU time is 1.38% and 0.57% of uncompressed expert GPU time. Combined with SAVB's visual exit and our execution implementation, DecoMoE achieves 1.16×1.16\times TTFT, 1.49×1.49\times TPOT, and 1.71×1.71\times natural-EOS generation speedups.

BackendUncompressed experts (ms)Compressed experts (ms)RCEP policy (ms)Expert ratioPolicy ratio
Qwen-type Dense-BMM2.289301.623980.0315470.94%1.38%
InternVL-type sparse5.512143.912140.0312470.97%0.57%

Table R1-A. GPU kernel-duration sums: BF16, input tokens N=512N=512, eight activated experts/token, p=0.70p=0.70. Policy denotes prefix-selection kernels; ratios use uncompressed expert GPU time.

MethodTTFT (ms)TPOT (ms/token)Generation latency (ms)
Baseline147.11374.715492.594
DecoMoE127.25950.072287.859

Table R1-B. Qwen3-VL-30B-A3B; 30 randomly sampled cases each from MMBench EN, ScienceQA, AI2D, AOKVQA, MMMU, and HallusionBench (180 total); p=0.70p=0.70, KV cache, warmed timing. Baseline uses the local Torch runtime. Natural EOS; output lengths are not fixed.

From an execution perspective, we agree with the reviewer: RCEP does not reduce the number of experts activated per token or the dominant MLP arithmetic associated with those activated experts. However, this arithmetic alone does not account for the complete execution cost of real inference. In the evaluated implementations, the GPU executes dense matrix operations organized by the backend; logical Top-kk sparse activation does not automatically turn expert computation into an unstructured sparse matrix multiplication. We must therefore examine which matrix computations actually execute and the costs of token dispatch, kernel launches, and output aggregation. Let NN denote token count, EE full-pool size, KK restricted-pool size, kk experts activated per token, and d,md,m the hidden and expert intermediate dimensions:

  1. Qwen-type Dense-BMM. The referenced Transformers 4.57.1 inference branch performs expert batched matrix multiplications before applying sparse routing weights. Gate, up, and down projections require approximately 6dm6dm FLOPs per token-expert pair (two FLOPs per multiply-add). Full-pool execution therefore costs approximately 6NEdm6NEdm, rather than only 6Nkdm6Nkdm. Restricting actual execution to KK experts reduces this to 6NKdm6NKdm, removing extra backend computation without reducing per-token activated expert count.

  2. InternVL-type sparse execution. Only selected tokens are dispatched to their assigned experts, giving approximately 6Nkdm6Nkdm expert FLOPs regardless of candidate-pool size. However, dispatch, weight access, kernel launches, and aggregation also contribute to latency. These costs depend on active expert counts, tokens per expert, and matrix-operation granularity. Reorganizing execution can therefore reduce latency without reducing token-expert pair count.

RCEP combines offline expert ordering with online contiguous-prefix selection. It consistently reorders expert parameters and routing dimensions, selects a prefix meeting the target cumulative routing mass, and reroutes within it while preserving Top-kk. It does not compute all experts and merely zero outputs afterward. This reduces the executed expert dimension in Dense-BMM. In expert-wise sparse execution, it can concentrate the same number of token-expert pairs into fewer active experts, replacing many small matrix operations with fewer larger operations and improving dispatch and execution efficiency. This explains why comparable theoretical FLOPs in Table 1 of the submission need not imply comparable latency: how the backend executes the remaining computation also determines efficiency.

AuthorsOpenReview ↗

Response to Reviewer 968K (W2): Distinctions from DyVTE and the rationale for text-routing calibration

W2: Distinctions from DyVTE and the rationale for text-routing calibration

DecoMoE's main distinction from DyVTE is the coordination of SAVB and RCEP to achieve structured compression along both token and expert dimensions during inference; text-routing calibration matches the computation after visual exit and is supported by our appendix routing analyses. We thank the reviewer for raising these questions, which help us clarify the roles of the components and their connection.

The differences from DyVTE concern both exit supervision and the compression mechanism. SAVB retains a lightweight binary classifier using pooled preceding text states and the final text state. However, DyVTE constructs supervision primarily from answer agreement with the original prediction and an uncertainty constraint relative to that prediction. SAVB instead evaluates forced-exit answer correctness using task-specific evaluation rules, determines a percentile threshold from first-token NLL values at calibration samples' earliest correct exit layers, and independently labels each candidate layer. Gating is evaluated at sparse candidate layers. More importantly, SAVB determines not only when visual tokens exit but also when RCEP begins. Following complete visual-block removal, RCEP dynamically selects a contiguous expert prefix at each subsequent MoE layer and reroutes within it, using expert parameters and routing dimensions reordered offline according to the calibrated order. Unlike DyVTE, which leaves the original expert pool unchanged after visual exit, this mechanism simultaneously shortens the sequence and restricts expert execution, connecting visual exit and expert compression through a unified structured execution process.

Text-routing calibration targets the computation that RCEP actually serves, rather than disregarding visual information. When RCEP activates, visual tokens have already exited, and subsequent experts process the remaining text tokens. Expert ordering should therefore prioritize the experts required by these tokens, without continuing to incorporate the routing demands of visual tokens that are no longer processed. Calibration inputs remain image-text examples, and the measured text states already incorporate visual information; "text calibration" refers only to extracting routing statistics from text positions. Appendix Figure 6 shows diffuse visual routing but a more concentrated expert structure for text, suggesting that mixed statistics may introduce expert preferences associated with a different computational workload. Figure 7 shows largely unchanged text-routing patterns before and after visual eviction at layer 40 for three examples. The lower panel of Figure 8 further shows that, for the analyzed example across layers 40-47, text-routing cosine similarity remains above 0.997, while mean absolute error remains below 0.0003. These results indicate that the remaining text tokens retain stable principal expert dependencies after visual exit, supporting text-routing statistics as the basis for subsequent expert-prefix construction.

AuthorsOpenReview ↗

Response to Reviewer 968K (Q1): Execution Mechanism, Incremental Savings, and Online Overhead

Q1: Execution Mechanism, Incremental Savings, and Online Overhead

DecoMoE reduces latency through complementary visual exit and structured expert compression; in our module experiments, RCEP's expert-stage savings exceed its added routing and prefix-selection costs. In the Qwen Dense-BMM ablation in Table 2, the reported times for Baseline, SAVB-only, and full DecoMoE are 0.37 s, 0.30 s, and 0.26 s, respectively. Full DecoMoE reduces this time by approximately 29.7% relative to Baseline and by a further 13.3% relative to SAVB-only.

RCEP changes the scope and organization of expert execution. Dense-BMM reduces the executed expert dimension from the full pool to a contiguous prefix. Expert-wise sparse execution instead concentrates routing into fewer active experts, which can improve dispatch and matrix-call organization without reducing the leading expert MLP arithmetic at fixed Top-k. The full-pool router projection is performed only once and its logits are reused; additional work primarily comprises routed-mass aggregation, prefix selection, and routing within the prefix.

BackendUncompressed experts (ms)Compressed experts (ms)Expert savings (ms)Routing increment (ms)Prefix selection (ms)Total added overhead (ms)
Qwen-type Dense-BMM2.289301.623980.665320.00078 (0.03%)0.03154 (1.38%)0.03232 (1.41%)
InternVL-type sparse5.512143.912141.600000.00058 (0.01%)0.03124 (0.57%)0.03182 (0.58%)

Table R1-A. Module-level GPU kernel-duration sums: N=512N=512, BF16, eight activated experts/token, and p=0.70p=0.70. Percentages use uncompressed expert time as the denominator. Expert stages include dispatch and aggregation.

The added routing and selection GPU costs amount to 4.86% and 1.99% of the expert-stage savings, respectively. Separately, relative to matched-K controls without online selection, the measured online increments including required control and synchronization effects are 0.38032 ms and 0.28589 ms.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.