- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DecoMoE, a two-dimensional structured compression framework for efficient multimodal MoE inference. The key observation is that existing methods compress either the visual-token dimension or the expert dimension, leaving redundancy in the other. DecoMoE decouples these two decisions. The Sample-Adaptive Visual Boundary (SAVB) predicts an input-dependent visual-exit layer from pooled text states and removes the complete visual-token block before self-attention at that layer. The Routing-Calibrated Expert Prefix (RCEP) offline reorders experts by mean text-token routed mass and, from the predicted exit layer onward, retains at each MoE layer the shortest physically contiguous expert prefix covering a target routed-mass fraction. Experiments on Qwen3-VL-MoE and InternVL3.5-30B-A3B across six benchmarks show that DecoMoE retains 97.91% and 97.87% of dense performance while reducing computation from 27.06 to 16.73 and 9.86 to 5.97 TFLOPS, with measured speedups of 1.69× and 1.41×. Ablations validate SAVB, RCEP, and their combination.
Critical strengths
-
DecoMoE decouples visual propagation from expert computation, combining an input-dependent visual exit (SAVB) with a text-routing-calibrated expert prefix (RCEP). This is consequential because prior methods compress only one dimension, leaving substantial redundancy in the other. The reported 1.69× speedup on Qwen3-VL-MoE with 97.91% average performance retention, and 1.41× on InternVL3.5-30B-A3B, shows that the two mechanisms complement each other and translate into real latency reductions, not just theoretical FLOP savings.
-
SAVB removes a contiguous visual-token block, and RCEP restricts computation to a contiguous expert prefix. This is consequential because it reduces irregular selection, fragmented dispatch, and memory movement, which are common reasons why theoretical FLOP savings fail to become real latency gains. The TTFT and TPOT reductions of 30.8% each support the practical value of this design.
Critical weaknesses
-
Although the average retention is 97.91% on Qwen3-VL-MoE, HallusionBench drops from 74.97 to 71.92 and MMMU drops by about 2 points; on InternVL3.5-30B-A3B, HallusionBench drops from 69.70 to 65.51. This is consequential because the average metric hides tasks where the method may be unacceptable, and the paper does not analyze why some tasks are more fragile or how to mitigate this.
-
The paper admits that RCEP and SAVB calibration samples are not enforced to be disjoint, and that POPE/MME results including calibration IDs are transductive. This is consequential because the method's main selling point is that it can be calibrated offline and applied to new inputs. If calibration and evaluation overlap, or if the calibration distribution leaks into reported numbers, the evidence for out-of-distribution generalization is weakened.
Questions for authors
1.Can you provide sensitivity analyses for SAVB's NLL percentile, gate threshold, and candidate-layer interval? How stable is performance under these choices?
-
Were FastV, SparseVLM, DyVTE, and FastMMoE tuned with the same calibration data and budget as DecoMoE on each backbone? Can you report their best-tuned results under identical conditions?
-
How does expert reordering interact with expert parallelism and load balancing? Does deploying DecoMoE require re-exporting the model or modifying existing inference kernels?
-
Why does HallusionBench degrade more than other benchmarks, and can this be mitigated by a more conservative SAVB or RCEP setting?
-
The online RCEP step computes full-pool router logits, then recomputes softmax and Top-k inside the selected prefix. What is the exact overhead of this two-pass routing, and is it included in the reported latency?
-
Does DecoMoE generalize to other multimodal MoE models such as DeepSeek-VL2 or Kimi-VL, and to video or high-resolution inputs where visual-token counts are much larger?
-
Can the authors report calibration and evaluation on strictly disjoint POPE/MME splits, given that the appendix notes potential overlap between SAVB and RCEP calibration samples?
11 comments
Response to Reviewer VkMr (Q1): Sensitivity to SAVB Hyperparameters
Q1: Sensitivity to SAVB Hyperparameters
Performance is relatively stable across NLL percentiles P60–P99; the gate uses a fixed 0.5 binary decision threshold; and a candidate-layer interval of 4 reduces execution overhead while closely matching denser checks in accuracy. We thank the reviewer for this suggestion, which helps clarify the rationale and scope of our default settings.
The NLL percentile reduces noise in exit supervision. We derive the threshold from the distribution of reference-answer first-token NLL at each calibration sample's first correct exit layer. A sample-layer pair is labeled exit-safe only when both task correctness and the NLL condition hold, reducing the influence of correct but insufficiently confident predictions. Default P95 balances filtering the high-NLL tail against retaining sufficient supervision. In Table R3-A, mean accuracy across P60–P99 ranges from 89.66% to 89.85%. P95 achieves 89.83%, differing from P90 and P99 by only 0.02 and 0.01 percentage points, respectively, supporting its placement within a relatively stable region.
Table R3-A. Qwen performance under different NLL percentiles (%).
| NLL percentile | MMBench | ScienceQA | Mean |
|---|---|---|---|
| Baseline | 88.98 | 90.90 | 89.94 |
| P20 | 81.61 | 88.81 | 85.21 |
| P40 | 87.95 | 90.99 | 89.47 |
| P60 | 88.42 | 91.08 | 89.75 |
| P75 | 88.30 | 91.02 | 89.66 |
| P90 | 88.48 | 91.21 | 89.85 |
| P95 (default) | 88.54 | 91.11 | 89.83 |
| P99 | 88.52 | 91.11 | 89.82 |
The candidate-layer interval controls decision frequency and available exit positions. Intervals 1, 2, and 4 achieve accuracies of 72.03%, 72.03%, and 71.92%, respectively. Compared with interval 1, interval 4 loses only 0.11 percentage points, reduces mean checks from 27.506 to 7.637, and lowers TTFT and generation latency on the timing samples by approximately 10.0% and 7.5%. Increasing the interval to 16 provides only another 4.2% TTFT reduction relative to interval 4, while accuracy drops by 2.20 percentage points. Thus, the default interval of 4 reduces the number of checks while closely matching denser checking in accuracy, offering a favorable performance–efficiency trade-off.
Table R3-B. HallusionBench interval study: original-path accuracy on 951 questions and optimized cached latency on five shared questions.
| Interval | Accuracy (%) | Mean exit (0-based) | Mean checks | TTFT (ms) | TPOT (ms/token) | Generation Latency (ms) |
|---|---|---|---|---|---|---|
| 1 | 72.03 | 29.450 | 27.506 | 127.857 | 40.219 | 168.437 |
| 2 | 72.03 | 29.460 | 14.267 | 119.456 | 40.747 | 160.560 |
| 4 (default) | 71.92 | 29.475 | 7.637 | 115.092 | 40.341 | 155.793 |
| 8 | 70.14 | 31.961 | 4.718 | 112.120 | 40.394 | 152.871 |
| 16 | 69.72 | 34.987 | 3.044 | 110.308 | 40.172 | 150.836 |
The gate threshold is fixed at 0.5. Unlike the NLL percentile used to construct training labels, this threshold applies to the inference-time softmax score of the exit-safe class: exit occurs when the score reaches 0.5. Exit depth therefore depends on the learned classification boundary and input state rather than a manually specified uniform depth. We did not independently sweep this threshold, so the results above should not be interpreted as evidence of insensitivity to the gate threshold.
Response to Reviewer VkMr (Q2–Q4): Baseline Settings, Parallel Deployment, and Task Sensitivity
Q2: Baseline Configurations and Computational Budgets
Our main comparison targets performance and efficiency at comparable theoretical FLOPs, rather than a uniform hyperparameter-search budget. FastV and SparseVLM retain approximately 40–50% of visual tokens, using relatively generous retention budgets to avoid overly aggressive pruning. DyVTE and SAVB construct their respective exit supervision from POPE/MME; FastV, SparseVLM, and FastMMoE do not require training an exit classifier.
Main-experiment configurations and computational costs.
| Method | Qwen3-VL-MoE configuration | FLOPs (TFLOPs) | InternVL3.5 configuration | FLOPs (TFLOPs) |
|---|---|---|---|---|
| Baseline | No compression | 27.06 | No compression | 9.86 |
| FastV | 50% retention; pruning-layer parameter 4 | 16.65 | 48% retention; pruning-layer parameter 6 | 5.86 |
| SparseVLM | 44% final retention | 16.21 | 40% final retention | 5.03 |
| DyVTE | Exit threshold 0.142; all experts retained | 18.29 | Not reported in the main table | - |
| FastMMoE | r=25%, l_v=6, k_v=5 | 18.88 | r=25%, l_v=6, k_v=5 | 8.02 |
| DecoMoE | SAVB + RCEP | 16.73 | SAVB + RCEP | 5.97 |
For FastMMoE, denotes visual-token retention, the starting layer for visual-expert compression, and the reduced visual routing width. FastV and SparseVLM operate at computational costs relatively close to DecoMoE, while FastMMoE uses more FLOPs than DecoMoE on both backbones.
Q3: Expert Reordering and Parallel Deployment
Expert reordering is compatible with expert parallelism and requires neither retraining nor mandatory model re-export, but the runtime must support prefix selection, within-prefix routing, and restricted expert execution. Consistently permuting router outputs and their corresponding expert parameters mathematically preserves the uncompressed model's function. This transformation can be performed during loading or saved in a reordered checkpoint. Existing matrix-multiplication kernels can be reused; fused MoE backends require corresponding adaptations to routing and dispatch interfaces so that experts outside the prefix are excluded from computation.
For expert parallelism, reordering itself should be distinguished from prefix restriction. If expert device ownership is preserved and IDs and dispatch mappings are updated consistently, reordering alone does not change device workloads; prefix restriction and rerouting can change their distribution. A contiguous prefix in the global expert order does not require all retained experts to reside on one GPU. Distributed deployment can assign prefix experts across devices using estimated post-compression token loads, with compact local layouts on each device. This provides a deployment strategy for retaining parallel execution and addressing load concentration, while load balancing still depends on expert placement and scheduling rather than being guaranteed by reordering alone.
Q4: Sensitivity on HallusionBench
HallusionBench's degradation may reflect its reliance on fine-grained visual evidence and correction of language priors. If SAVB exits before relevant evidence is sufficiently incorporated into text states, later layers cannot directly revisit that visual information. Likewise, retaining high cumulative routing mass does not guarantee coverage of every question's critical experts. Average retention therefore cannot replace task-level analysis. Our objective is to improve efficiency while preserving acceptable performance, rather than guarantee lossless compression on every task. A larger RCEP expert budget provides broader expert coverage, although performance improvements require task-specific validation. As future work, we will also explore lightweight LM-head adaptation with the backbone frozen, investigating whether it can mitigate compression-induced shifts in the output distribution.
Response to Reviewer VkMr (W1 & W2): Task Sensitivity and Calibration–Evaluation Separation
W1: Task Sensitivity and Performance Degradation
HallusionBench and MMMU place stronger demands on precise visual evidence and deep cross-modal reasoning, which may make them more sensitive to visual exit and expert compression. We thank the reviewer for highlighting this important issue. We agree that the 3.05- and 4.19-percentage-point drops on HallusionBench for Qwen and InternVL, respectively, and the approximately two-point drop on MMMU deserve separate analysis rather than being obscured by average retention.
The potential risks arise from premature removal of visual evidence and insufficient expert coverage. HallusionBench requires answers grounded in actual visual evidence rather than language priors, while MMMU involves charts, specialized symbols, and multi-step reasoning. If SAVB triggers before critical details are sufficiently encoded into text states, subsequent layers lose direct access to that visual evidence. Meanwhile, RCEP orders experts using average routed mass across calibration examples and dynamically selects a prefix, but high cumulative routed mass does not guarantee coverage of every expert important to a specialized question. Appendix A.7.3 (Case 3, Figure 11) provides a relevant observation: for the same fire-hydrant image, presence recognition, attribute explanation, and detailed analysis including identification-number recognition require reported propagation depths of 18, 22, and 42, respectively. This illustrates that visual-propagation requirements depend on the task, not merely the image or token count.
Our focus is to improve inference efficiency while preserving performance within task-acceptable limits; what constitutes an acceptable loss depends on the application. The current method keeps the multimodal backbone frozen. Future work could explore adapting the LM head to compressed hidden states while freezing the rest of the network, investigating whether this mitigates compression-induced output-distribution shifts and improves the performance–efficiency trade-off on sensitive tasks.
W2: Separation Between Calibration and Evaluation
None of the six main evaluation benchmarks is used for calibration, and the POPE/MME test subsets are also separate from calibration samples. We thank the reviewer for identifying the insufficiently clear description. We first partition POPE/MME into calibration and test subsets, then use the calibration portion to construct SAVB forced-exit supervision and collect RCEP text-routing statistics. SAVB and RCEP may share calibration samples, but evaluation excludes the union of samples used by either component. Thus, overlap between the two calibration sets is distinct from calibration–test overlap. MMBench, ScienceQA, AI2D, AOKVQA, MMMU, and HallusionBench are not used in these calibration procedures.
Including POPE in Table 2 can indeed obscure the distinction between calibration sources and evaluation subsets, reflecting a shortcoming in our presentation and protocol description. Although that result uses a separate test subset, we will remove the POPE results from this ablation and recompute the corresponding aggregate metrics. We will also revise Appendix A.4 to clarify the sample-ID boundaries between calibration and evaluation and the permitted overlap between SAVB and RCEP calibration sets.