acceptodds
Withdrawn from ICLR 2027

MAP-MoE: Modality-Aware Structured Channel Pruning for Multimodal Mixture-of-Experts Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sparse Mixture-of-Experts (MoE) models improve performance by scaling up while activating only a few experts for each token. This scaling advantage requires all experts to be stored during deployment, leading to substantial storage and GPU memory costs. Existing solutions reduce these costs by pruning experts, merging experts, or pruning expert feed-forward network (FFN) channels. However, balancing model accuracy and inference speed remains challenging. We therefore introduce MAP-MoE, a training-free framework that combines modality-aware expert FFN channel pruning with our Ragged-Hybrid GPU executor. Using a small unlabeled calibration set, MAP-MoE groups experts into visual, mixed, and textual groups and assigns different pruning rates to them under a shared global budget. Within each MoE layer, MAP-MoE ranks FFN channels across experts in the same modality group and retains only the selected rows of the FFN gate and up projections and the corresponding columns of the down projection. Experts retain different numbers of channels after pruning, making standard execution much slower. Ragged-Hybrid addresses this with custom Triton kernels for small token batches and grouped GEMM for larger inputs. Our main evaluation covers two multimodal MoE models at FFN channel pruning ratios of 20%, 30%, and 50% across 10 benchmarks. At 50% pruning, MAP-MoE improves average accuracy by 1.34 points over the SOTA baseline (CAMERA-P) on Qwen3-VL while retaining 98.5% of the original Qwen3-VL average accuracy. For batched generation on one A800, our framework achieves up to 1.65× end-to-end generation speedup over the original model and reduces peak allocated GPU memory by up to 46%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal mixture-of-experts, structured pruning, model compression, modality-aware pruning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_88hwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

MAP-MoE provides a mechanism for per-channel MoE expert pruning. MAP-MoE first pools experts's channels of the same group and find the best pruning ratio for different experts from different modality, subject to a global pruning rate, based on activation energy, then it chooses which experts to actually prune via RWE. MAP-MoE achieves good performance with existing baselines.

Critical strengths

The paper is well motivated, and comes with a clear package of motivation, solution, and it did a good job aiming for a complete validation cycle, grounding its contribution to actual end-to-end speedup during inference with an implementation co-design.

Critical weaknesses
  1. I find 4.1 and 4.2 a bit confusing to me - why do we use two different metrics for ranking the experts and for selection? Why couldn't we just use a single metric?
  2. The metrics used, while meaningful, does not have enough evidence to establish their connections to the final prediction. What is each terms contribution to the final prediction error? Maybe a KL term would help?
  3. The activation/stats are from the original unpruned model, and earlier pruning decisions would contribute to the current layers, but the scores wouldn't see this. How accurate would this be?
  4. The speedup appears to be deriving from two different software? One is HF Transformer, the other is on the custom kernel. To really show the benefit of the custom kernel, it would be great to separate the effect of FLOPs reduction (running with original Transformer lib) and improved implementation. If the naive is slower, I think it makes the new kernel shine.
Questions for authors

Please see weakness. Is there a way to blend in speedup into the search process? Is there any tradeoff we can make to incorporate speedup to equation (3)? Perhaps a small accuracy loss can bring in larger speedup at certain pruning region? Why does some of the experimental pruned results outperform unpruned dataset? Is there a clear explanation or we don't know yet?

1 comment

Thank you for highlighting the distinction between budget and channel scores, the need to connect proxies to predictions, and the importance of separating arithmetic reduction from execution engineering. Your questions motivated direct shared-score controls, fixed-budget selector diagnostics and a recalibration comparison.

Question 1. Why use two scores, and can one score serve both stages?

For Question 1,We use activation energy to construct group-level loss curves that guide budget allocation. For channel selection, RWE additionally accounts for a channel’s contribution through the down projection and favors intermittently large activations through its concentration term. The two stages thus address group-level capacity allocation and fine-grained channel retention, respectively.

Following your suggestion, we compare our allocation/selection combination with alternatives that use the same score in both stages. The table reports accuracy averaged over all tasks for Qwen and InternVL:

Allocation curve / channel selectionQwen P20Qwen P30Qwen P50InternVL P20InternVL P30InternVL P50
Original model0.7730.7730.7730.7760.7760.776
Activation energy / RWE (ours)0.7770.7740.7610.7750.7700.746
Activation energy / activation energy0.7780.7730.7550.7760.7660.741
RWE / RWE0.7770.7730.7610.7740.7680.746

The results show that our combination (Activation energy / RWE) achieves higher accuracy than both shared-score alternatives at P20 and P30. At P50, our method is subject to a capacity-protection constraint under which Activation energy / RWE (ours) and RWE / RWE effectively select the same channels; compared with Activation energy / activation energy, our method is nevertheless better at both P30 and P50.

Reviewer_inmgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

To address the challenge of balancing accuracy and inference speed in existing MoE models, the authors design a training-free framework, MAP-MoE. Using a small unlabeled calibration set, it divides experts into visual, mixed, and textual groups and assigns different pruning rates under a total compression budget. It combines customized Triton kernels and grouped GEMM to alleviate the inference-speed degradation after compression.

Critical strengths

For this type of pruning method, engineering optimizations—customized Triton kernels and grouped GEMM—are adapted to alleviate the inference-speed degradation after compression.

Its performance is close to or even better than CAMERA-P. In terms of speed, it achieves up to 1.65× end-to-end generation acceleration over the original model and reduces peak allocated GPU memory by up to 46%.

Critical weaknesses

There have already been some works [1] that design pruning schemes for experts of different modalities. At the same time, I am skeptical about distinguishing expert types using a calibration dataset. It is difficult to say whether this can generalize to different multimodal MoE models.

The authors conducted experiments on three models. On InternVL3.5, why does the accuracy of CAMERA-P drop sharply under the P50 budget, becoming much lower than REAP? This issue does not occur under P20 or P30, and it is also inconsistent with the experimental results on Qwen3-VL.

Regarding speed, I think you should also report the results of the comparison methods. At the same time, you need to adapt the same engineering optimizations used by MAP-MoE—customized Triton kernels and grouped GEMM—to ensure that only the efficiency of pruning itself is compared. The current speed and memory results themselves are not sufficiently convincing.

The spacing between the table headers and the tables is compressed too tightly. Although this does not affect my score, it is not very appropriate.

[1] MAPEX: Modality-Aware Pruning of Experts for Remote Sensing Foundation Models, TGRS, 2026.

Questions for authors

Overall, I think the novelty is relatively weak, and the gains of the method itself are also weak. As an ICLR paper, it should provide convincing performance in at least one of these two aspects.

It does not achieve the best results across different pruning rates on different models. It only shows some improvement under P50 (around 1.3–2.6). The acceleration gains themselves come from engineering optimizations, and comparisons with methods of the same type are lacking.

The amount of work is fairly solid. A main figure illustrating the method is missing. It is not necessary, but it could make the paper clearer. I am also curious whether it can be combined with visual token reduction methods.

2 comments
Reviewer_inmgOpenReview ↗

Response to the author rebuttal

Thank you for your reply. So far, I don't seem to have received the updated PDF. The changes you mentioned in your reply are not visible in the current version.

Please upload the revised manuscript PDF alongside your response and ensure that it reflects the changes you report as complete. I’m concerned about the discrepancy between your response and the version currently available for review, as it makes it difficult to verify whether my comments have been addressed.

Thank you for your comment, and please accept our apologies for taking additional time to revise the PDF. We have now updated and uploaded the revised PDF alongside this response. The changes we reported as complete are reflected in this updated version, and we would be grateful if you could check it against your comments.If the updated PDF is still not visible or does not match the changes described, please let us know and we will re-upload it immediately.

Reviewer_5rBvOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MAP-MoE, a training-free FFN channel pruning method for multimodal MoE models, along with a GPU execution engine called Ragged-Hybrid. Based on the observation that visual experts are much less sensitive to channel removal than textual experts, the method groups experts into visual, mixed, and textual categories using a calibration set, allocates group-specific pruning rates under protection caps, and selects channels within each group using a Rarity-Weighted Energy (RWE) score. To handle the irregular channel counts across pruned experts, Ragged-Hybrid combines custom Triton kernels (≤64\le 64 tokens) and grouped GEMM (>64>64 tokens) to achieve practical speedups on Qwen3-VL.

Critical strengths
  1. The motivation is clear and well-supported. The controlled experiment in Fig. 1b (removing the same number of channels from visual vs. textual groups) convincingly shows that visual experts have higher redundancy while textual experts are pruning-sensitive, which naturally motivates modality-specific pruning budgets.
  2. The paper goes beyond reporting theoretical parameter reductions and addresses actual inference speed. It shows that standard PyTorch execution slows down after non-uniform channel pruning (Table 7) and provides a working kernel solution (Ragged-Hybrid) with measured decoding speedups and memory savings.
Critical weaknesses
  1. Unclear attribution of the P50 accuracy gains (Eq. 3 is bypassed and gains are task-skewed). Because the hard caps (uv=0.8,um=0.6)(u_v=0.8, u_m=0.6) limit non-textual pruning capacity to Pvm≈0.44P_{vm} \approx 0.44 (Qwen) and 0.340.34 (InternVL), the equal-marginal optimizer (Eq. 3) is completely bypassed at the main P50 setting, where (rv,rm)(r_v, r_m) are simply fixed to (0.8,0.6)(0.8, 0.6). Meanwhile, at P20/P30 where Eq. 3 actually operates, gains over CAMERA-P are minimal (+0.12 pts on Qwen-P20; tied/behind on InternVL). In addition, the selection-granularity ablation (Table 3, Layer-global vs. MAP-MoE) is only reported at P20 rather than P50, and Qwen-P50's +1.34-point average lead over CAMERA-P comes almost entirely from a single task (OK-VQA, +11.89 pts, vs. +0.17 pts average across the other 9 tasks).
  2. Noticeable prefill latency (TTFT) degradation. Tables 6 and 10 show that TTFT is consistently slower than the unpruned model across all workloads (0.488x–0.827x; e.g., Text M3 degrades from 208.2 ms to 426.8 ms). Since many practical VLM workloads (like single-turn VQA) have long visual inputs and short answers, this prefill slowdown narrows the E2E speedup to 1.12x at 16 output tokens (M1) and could make short-answer generation slower than the original model.
Questions for authors
  1. Since (rv,rm)(r_v, r_m) at P50 are fixed by the (0.8,0.6)(0.8, 0.6) caps, how would P20 and P30 perform if Eq. 3 were replaced by simply scaling the caps proportionally (e.g., rv:rm=4:3r_v:r_m = 4:3 with rt=0r_t=0)? Also, could you report the Table 3 comparison (Per-expert vs. Layer-global vs. MAP-MoE) on Qwen-P50?
  2. Why is the grouped GEMM path (>64>64 tokens) up to 2×2\times slower than the unpruned model during prefill despite removing 50% of the channels? Is the bottleneck caused by token sorting/regrouping, or by having too many distinct channel-count buckets (up to 768/16=48768/16 = 48 after 16-alignment)? Would a coarser alignment (e.g., 64 or 128) improve TTFT?
  3. Why does Qwen3-VL achieve higher OK-VQA accuracy at P50 (0.521) than both the unpruned model (0.502) and P30 (0.496), while CAMERA-P drops to 0.244 on InternVL-P50 OK-VQA?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.