acceptodds
Under review as a conference paper at ICLR 2027

MDMixQ: Modality-Decoupled Mixed-Precision Quantization for Multimodal MoE Compression

Abstract

Post-training quantization is widely adopted for reducing the memory footprint of large-scale Mixture-of-Experts (MoE) models, and expert-level mixed precision further improves the accuracy-compression trade-off by allocating heterogeneous bit-widths across experts. In multimodal MoE settings, however, we find that existing mixed-precision pipelines overlook inter- and intra-modality routing non-uniformities that skew expert bit allocation and degrade accuracy under compression. To address these failure modes, we propose MDMixQ, a mixed-precision quantization framework comprising three targeted components: Modality-Decoupled Calibration (MDC), which isolates a clean text-routing signal from multimodal inputs; Salience-Guided Filtering and Rectification, which retains only confident text-side routing events and rectifies sign-inconsistent logits; and Dynamic Mixed-Precision Allocation, which ranks experts by the refined salience scores under a global bit budget and applies modality-aware activation quantization during MoE inference. At a 2.52-bit average expert budget, the best MDMixQ configurations retain 96.29%, 96.05%, and 95.33% of FP16 performance on Qwen3-VL-30B-A3B, Kimi-VL-16B-A3B, and Qwen3-Omni-30B-A3B, respectively, outperforming prior expert-priority methods across two model families. The framework also transfers to the MXFP format with 99.72% performance retained on Qwen3-VL-30B-A3B, confirming compatibility with emerging hardware-oriented quantization standards.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Post-Training Quantization, Multimodal Large Language Models, Mixed-Precision Allocation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_diEPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies expert mixed-precision quantization for multimodal MoE. MDMixQ separates routing statistics by modality, filters confidence, and corrects the estimated importance of experts, then assigns different weight precisions to experts on the text side. The experiments cover three backbone models.

Critical strengths
  1. The importance estimator can simply use the routing output, without computing gradients or extracting quantization errors from each expert one by one. The allocation logic is also simple and easy to understand. It maintains good retention while covering multimodal scenarios, making it quite convincing.

  2. The calibration experiments distinguish between “text-side routing statistics” and “pure text input,” making comparisons easier. These conclusions about how modality-aware quantization is actually implemented are quite valuable as a reference.

Critical weaknesses
  1. My main concern is novelty. PreMoE has employed Top-K local softmax and average maximum probability threshold. These are highly consistent with SGFR, but the paper does not discuss the relationship between the two. Modal separation statistics and two-level bit allocation based on sorting have precedents in MODE and METGG as well. As it is currently written, the novelty in the method is not as great as the contribution statement suggests.
  2. The adaptation part of MODE and MBQ is not clearly explained. Neither does it specify exactly which original components are retained, nor does it clarify how each method is transformed into an expert priority baseline.
Questions for authors

1.What are the differences between the PEU process of SGFR and PreMoE v2? 2. Compared with the modal separation statistics in MODE and the bit allocation based on sorting in METGG, which parts of MDC and DMA are new? 3. How do MODE and MBQ adapt to the shared quantizer and expert weight budget?

4 comments
AuthorsOpenReview ↗

W1 & Q1 — Relationship to PreMoE and multimodal extensions

We thank the reviewer for pointing out that this relationship was insufficiently explained. MDMixQ is follow-up work building on PreMoE v2, extending its Predicted Expert Utility (PEU) approach to multimodal MoE mixed-precision quantization. SGFR adapts PEU's local Top-K softmax, adaptive confidence threshold, and logit transformation. Both approaches aggregate importance over offline calibration data.

Our extensions are modality-decoupled collection and modality-conditioned scoring, text-prioritized allocation of expert-weight precision under a global budget, and the study of asymmetric activation precision within MoE layers. MDMixQ retains the experts and their native routing configuration while assigning heterogeneous weight precision. The NT4/T8 extension additionally provides operator implementation guidance and NPU measurements.

We will explicitly attribute the shared PEU operations to PreMoE and substantially revise the related work, methodology, and contribution statements to explain MDMixQ as a multimodal quantization extension. The ablations quantify the benefit of applying these operations within our multimodal quantization framework.

AuthorsOpenReview ↗

Q2 — Contributions of MDC and DMA relative to MODE and METGG

We will distinguish the importance signal from the allocation operation:

ComponentRelationship to prior workRole in MDMixQ
MDC relative to MODEBoth preserve modality-specific routing statisticsMaintains separate text and non-text accumulators and uses text-side salience for the default allocation; non-text reservation is evaluated separately
DMA relative to METGGRanking-based precision allocation also appears in prior workUses text-anchored salience to assign {2,3}\{2,3\}-bit expert precision under the fixed global budget

MDC supplies modality-separated calibration statistics, and DMA converts the resulting text-side salience into precision assignments under the budget. We will state these roles explicitly in the revision.

AuthorsOpenReview ↗

W2 & Q3 — MODE and MBQ adaptations

Within each quantizer setting, the controlled comparisons use the same calibration examples, quantizer configuration, target average expert bits, and evaluation within each backbone. Methods differ in importance estimation and allocation.

MethodRetained or transferred informationAdaptation to the shared pipeline
MODEModality-decomposed frequencies, key-vision-token filtering, per-expert quantization sensitivity, and ILP allocationThese importance and allocation components are retained, with ILP choices restricted to {2,3}\{2,3\} as in MDMixQ. The original GPTQ stage is replaced by the shared quantizer.
MBQCalibration sensitivitySensitivity is converted into expert-level importance and used to assign expert precision under the shared budget and quantizer.

MBQ uses calibration sensitivity to construct expert importance for precision allocation. MODE remains a sensitivity-aware, ILP-based baseline in our evaluation. We will state these adaptations explicitly in the revised experimental setup. The additional matched GPTQ comparison on Qwen3-VL preserves MODE's GPTQ stage under the stated calibration and {2,3}\{2,3\} bit-budget constraints: MDMixQ + GPTQ achieves 97.85% Retain versus 96.52% for MODE + GPTQ. Retain and end-to-end calibration/quantization time are reported in the response to Reviewer Whd7.

Thank you for the clarification. The response to W2/Q3 resolved my concern. The response to W1/Q1 confirmed that the novelty is limited. Q2 did not point out any new operations; it merely restated the roles of the components. In summary, the main difference is the design choice of “using text-side saliency for assignment.” I believe my initial score was reasonable, and I will maintain my original main score.

Reviewer_Whd7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MDMixQ studies mixed-precision quantization for multimodal MoE models. It argues that combining routing statistics across modalities can underrepresent experts important to text tokens. The method separates modality-specific routing statistics, filters low-confidence routing events, and aggregates text-side router logits to globally rank experts for higher weight precision. An optional extension assigns higher activation precision to text than to non-text tokens within MoE layers.

Experiments on three multimodal MoE backbones report improved aggregate accuracy retention over the evaluated allocation methods at matched expert-weight budgets. The paper also evaluates text-only capabilities and compatibility with MXFP formats. Its main contribution is a routing-only importance estimator that avoids gradients and per-expert sensitivity profiling, aiming to simplify calibration while preserving quantized model quality.

Critical strengths
  1. The importance estimator is simple and potentially cheap. It uses router outputs collected during calibration, avoiding repeated sensitivity measurements for individual experts. This could make expert-level mixed precision easier to apply.

  2. The reported improvements are consistent across several backbones. Evaluation covers Qwen3-VL, Kimi-VL, and Qwen3-Omni, with image, video, audio, and additional text-only benchmarks. This provides broader support than an improvement on a single model or task.

  3. The experiments distinguish the main method from optional extensions. Non-text expert reservation and asymmetric activation quantization are evaluated separately.

Critical weaknesses

W1. Text-prioritized expert importance needs a more direct justification

Separating modality statistics is reasonable, but the default allocation uses only text-side scores. More concentrated text routing and greater visual-token redundancy do not directly establish that experts favored by text are more sensitive to quantization. The calibration ablation supports this choice in the tested setting, but offers limited explanation of when non-text-dependent experts also need protection. A targeted sensitivity analysis would strengthen the rationale.

Moreover, MODE[1] already separates modality-specific statistics. The distinctive filtering and rectification components therefore need a controlled comparison against simple text-only frequency and filtered probability aggregation to establish their additional value.

W2. Cross-layer comparability of the importance scores is not established

Expert routing operates within each layer. Allocating a global bit budget across layers can still be reasonable, but requires comparable importance scores. MDMixQ globally ranks accumulated rectified logits without establishing this comparability. Layer-dependent logit offsets or scales may affect allocation independently of quantization sensitivity. For example, adding a common constant to all expert logits for a token leaves their ordering and softmax probabilities unchanged, but can change the salience scores.

W3. Baseline adaptations and calibration costs are insufficiently documented

A common quantizer helps compare allocation signals, but MODE[1] includes sensitivity profiling and GPTQ calibration, while MBQ[2] has its own modality-balanced calibration procedure. How these components are preserved or adapted in the shared pipeline is unclear. Thus, the results do not fully establish superiority over the complete prior methods. A representative faithful comparison, together with calibration time and peak memory, would clarify the claimed accuracy–cost advantage.

W4. Practical efficiency of the core mixed-precision scheme is not demonstrated

Storage savings are analytical estimates, and the NPU timing experiment uses INT4 weights with mixed activation precision rather than the core 2/3-bit expert allocation. No end-to-end latency, throughput, or measured peak runtime memory is reported. Appendix A.9 acknowledges that optimized kernels for heterogeneous expert precision are unavailable in the evaluated backend, leaving deployment benefits unresolved. Calibration time and peak memory are also unmeasured, so the routing-only estimator's potential cost advantage remains unverified. These omissions substantially limit the practical validation of the compression method.

[1] Chen, Yuanteng, et al. "MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs." arXiv preprint arXiv:2606.17118. 2026.

[2] Li, Shiyao, et al. "MBQ: Modality-Balanced Quantization for Large Vision-Language Models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2025.

Questions for authors
  1. Does text-side salience predict actual quantization sensitivity better than modality-balanced statistics? A targeted analysis of experts with different modality preferences would help explain when text-prioritized allocation is appropriate.

  2. Are importance scores comparable across layers and why

  3. What do confidence filtering and rectification add beyond text-only frequency? Please isolate their contributions under the same calibration data, quantizer, and allocation budget.

  4. What practical efficiency gains does the core 2/3-bit expert allocation achieve? Please report end-to-end latency, throughput, and peak runtime memory against a uniform-precision reference, together with calibration time and peak memory relative to the closest baselines.

4 comments
AuthorsOpenReview ↗

W1 — Text-prioritized importance and SGFR controls

The calibration ablation supports text-prioritized allocation empirically. To complement the routing-concentration analysis, we report a modality-level expert-skipping sensitivity experiment using KL divergence. The text-side sensitivity is higher than the visual-side sensitivity across the tested skipping ratios from 12.5% to 100%, and the summary reports a median Text/Visual sensitivity ratio of 3.9×.

The KL result provides complementary modality-level evidence alongside the matched-budget calibration and allocation ablations. We will include the experimental setup and summary in the revised manuscript.

Text prioritization also allows non-text-important experts to be protected when beneficial: the separate Top-8 non-text reservation ablation improves Retain from 96.11% to 96.16% at the same budget. We therefore present text-side importance as an effective default in the tested setting, with non-text reservation as an optional extension.

We evaluated filtering and rectification under the shared Qwen3-VL calibration data, quantizer, 2.52-bit expert budget, and native routing configuration:

Importance estimatorRetain (%)
Fused activation frequency94.61
Text-only activation frequency95.82
Raw text-logit sum94.25
Filtering only, raw text-logit accumulation93.51
Rectification only95.28
Filtered local-softmax probability aggregation95.43
Full SGFR96.11

The filtered-probability condition is the Softmax result in Appendix A.6: it uses the same text tokens, native Top-K, and threshold-filtering mask as SGFR, replacing only the accumulated rectified logits with local-softmax probabilities. Full SGFR improves on text-only frequency and filtered probabilities by 0.29 and 0.68 Retain points, respectively. The two logit-processing stages are complementary: filtering adds 0.83 points when rectification is enabled, and rectification adds 2.60 points when filtering is enabled. Filtering alone reduces Retain by 0.74 points relative to raw accumulation, while filtering and rectification together reach 96.11% Retain.

AuthorsOpenReview ↗

W2 — Cross-layer comparability

We tested per-layer z-score normalization of final salience and token-centered logits to evaluate the effects of logit offsets and layerwise score scales on global precision allocation. Token centering is applied before rectification: g^t,il=gt,il−1Nl∑j=1Nlgt,jl.\hat g_{t,i}^{l}=g_{t,i}^{l}-\frac{1}{N_l}\sum_{j=1}^{N_l}g_{t,j}^{l}. All variants keep the native routed-expert set and filtering mask fixed. Centering removes common additive logit offsets; z-scoring final salience tests the effect of layerwise score normalization.

Scoring variantRetain (%)Δ Retain vs. original (points)3-bit assignment overlap (%)
Original SGFR96.110100
Per-layer z-scored salience95.84-0.2795
Token-centered logits95.79-0.3289

Overlap is the fraction of the original 3-bit expert set retained by the control, measured for matched calibration subsets. Both controls preserve most high-precision assignments and change Retain by at most 0.32 points. Original SGFR achieves the highest Retain. These results support the practical stability of our allocation under the tested Qwen3-VL controls.

AuthorsOpenReview ↗

W3 — Baseline adaptations and calibration cost

The shared-pipeline comparison keeps calibration data, weight quantizer, expert-weight budget, and evaluation fixed; methods differ in importance estimation and allocation.

MethodRetained or transferred informationAdaptation to the shared pipeline
MODEModality-decomposed frequencies, key-vision-token filtering, per-expert quantization sensitivity, and ILP allocationThese importance and allocation components are retained, with ILP choices restricted to {2,3}\{2,3\} as in MDMixQ. The original GPTQ stage is replaced by the shared quantizer.
MBQCalibration sensitivitySensitivity is converted into expert-level importance and used for precision assignment under the shared budget and quantizer.

The MODE baseline retains sensitivity-aware ILP allocation, and MBQ converts calibration sensitivity into expert importance for precision assignment. We will make these conversions explicit in the revised experimental setup. On Qwen3-VL-30B-A3B, we compared MDMixQ and MODE under both the shared quantizer and matched GPTQ settings. The GPTQ variants use the same calibration subsets, {2,3}\{2,3\} expert-bit choices, 2.52-bit average budget, and evaluation; each method retains its own importance estimator and allocation procedure. We report Retain and end-to-end calibration/quantization time for each pipeline.

MDMixQ's regular estimator uses forward-only router-statistics collection without per-expert quantization-sensitivity profiling. MODE estimates quantization sensitivity through repeated per-expert forward comparisons. We report Retain (%) and end-to-end calibration/quantization time (h). The time covers the complete preparation pipeline, including importance estimation, bit allocation, and weight quantization, with GPTQ included where used:

PipelineRetain (%)End-to-end calibration/quantization time (h)
MDMixQ, shared quantizer96.111.2
MODE, GPTQ replaced by shared quantizer94.932.1
MBQ sensitivity-based allocation, shared quantizer94.191.8
MDMixQ + GPTQ, paired reference97.857.5
MODE + GPTQ, paired reference96.528.1

Under the shared quantizer, MDMixQ improves Retain over MODE by 1.18 points and reduces preparation time from 2.1 h to 1.2 h, approximately 42.9% less time. Compared with MBQ, it gains 1.92 Retain points with approximately 33.3% less time. With GPTQ, MDMixQ's advantage persists: 97.85% versus 96.52% Retain, with 7.5 h versus 8.1 h, approximately 7.4% less time.

Applying GPTQ to MDMixQ raises Retain from 96.11% to 97.85% (+1.74 points), while increasing preparation time from 1.2 h to 7.5 h. This demonstrates compatibility with GPTQ for higher accuracy at a greater calibration cost. MDMixQ provides higher Retain and lower preparation time than MODE within each matched quantizer; its relative time saving is smaller when GPTQ is used.

Reviewer_Whd7OpenReview ↗

Response to the rebuttal

I appreciate the authors' response, which provides some small-scale empirical evidence; however, I still believe the paper lacks a solid theoretical foundation and rigorous analysis. Overall, it does not meet the acceptance criteria for ICLR and I will maintain my original score.

Reviewer_xaYbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose MDMixQ, a mixed precision quantization framework for multi-modal mixture-of-expert models. Mixing 2- and 3-bit experts, they achieve effective 2.5-bit compression at minor loss. They find that text-only calibration helps performance and identify different improvements for MoE quantization methods. MDMixQ achieves competitive performance compared to other methods.

Critical strengths

MDMixQ achieves strong performance over competitors.

Critical weaknesses

The paper is partially not easy to follow due to hyper-technical terms (f.e. "Salience-Guided Filtering and Rectification, which retains only confident text-side routing events and rectifies sign-inconsistent logits"). We encourage to authors to simplify their language.

The authors find that text-tokens typically activate a smaller set of of experts than other input data. Expert ranking using only one modality leads to a calibration-inference mismatch that should be further explored. Can the authors provide more intuition why it is enough to calibrate using text tokens only?

Questions for authors

The authors list a number of problem and their solutions in the abstract. They specifically say their method addresses failure modes "overlooked in other methods". It sounds like the three targeted components resolve these issues. Could the authors elaborate how/which related work has overlooked those?

4 comments
AuthorsOpenReview ↗

W1 — Presentation: technical terminology

We agree that the terminology can be simplified. We will revise the abstract and method overview to first explain the procedure in plain language: "We keep text and non-text routing statistics separate, discard low-confidence routing events, and use the resulting expert scores to distribute precision under a fixed budget." Mathematical definitions will follow this intuition.

AuthorsOpenReview ↗

W2 — Calibration-inference mismatch

Calibration inputs remain multimodal. "img0 + txt10k" means that images are still processed in the forward pass, while only text-token routing contributes to the importance scores. Thus, text-side statistics are collected from real multimodal activations.

Table 2(a) compares the statistics used for allocation:

Calibration statisticsText-token fractionRetain (%)
img10k + txt00%95.50
img5k + txt5k50%95.05
img2k + txt8k80%95.72
img0 + txt10k100%96.14

Text-only statistics achieve the highest Retain among these conditions. More concentrated text routing motivates this choice, while the ablation provides empirical support in the tested setting. A complementary expert-skipping experiment using KL divergence reports a median Text/Visual sensitivity ratio of 3.9×, supporting the modality-level rationale. Non-text-dependent experts can still benefit from protection: in the separate non-text-reservation ablation, reserving a small Top-8 non-text expert set improves Retain from 96.11% to 96.16% at the same budget. These results support text-prioritized allocation as the default, with optional non-text expert reservation.

AuthorsOpenReview ↗

Q1 — Which related work overlooked the targeted failure modes?

We will replace the broad wording "overlooked in other methods" with a precise description of the aggregation rules evaluated in our baselines:

  • Fused frequency (ActFreq baseline) aggregates routing events across modalities, so token-volume imbalance can affect expert priorities.
  • Uniform activation counting gives each selected event the same contribution, motivating the comparison against confidence-filtered statistics.
  • Raw-logit accumulation sums signed values, motivating the rectification ablation.

The revised paper will describe these baseline choices and their relation to our components method by method. The experiments isolate the value of modality separation, confidence filtering, and rectification in the tested quantization setting.

Thank you for the response. The clarification and additional experiments are helpful, but my concern about excluding non-text routing from precision allocation remains only partially addressed. I maintain my current score.

Reviewer_7Fg4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MDMixQ, a modality-decoupled mixed-precision quantization framework for multimodal MoE compression. Specifically, the authors propose a three-stage approach to improve routing-based expert importance estimation and bit allocation. The first stage separates text and non-text routing statistics during calibration. The second stage filters low-confidence routing events and rectifies signed router logits to obtain refined expert importance scores. The third stage allocates mixed-precision expert weights under a global bit budget based on these scores. The authors additionally explore modality-aware activation quantization within MoE layers.

Critical strengths
  1. The paper analyzes the differences in expert activation patterns across modalities and identifies several sources of bias in routing-based expert importance estimation, including cross-modal token imbalance, low-confidence routing events, and signed logit cancellation. These observations reveal the limitations of conventional routing-based importance estimation and motivate the proposed framework.
  2. The authors propose a lightweight mixed-precision quantization framework that combines modality-decoupled calibration, salience-guided filtering and rectification, and expert-level bit allocation. Compared with methods relying on gradient-based sensitivity estimation or Hessian information, MDMixQ directly constructs expert importance scores from routing statistics, avoiding expensive per-expert sensitivity profiling.
  3. The authors evaluate MDMixQ on three multimodal MoE models across image, video, and audio understanding tasks. The experimental results demonstrate consistent improvements over existing mixed-precision quantization methods under the same average expert-weight bit budget. Ablation studies on calibration strategies, expert allocation, and activation precision further validate several important design choices.
Critical weaknesses
  1. Recent work such as MODE has already explored modality-specific expert statistics for mixed-precision quantization of multimodal MoE models. In particular, MODE decomposes expert selection frequencies across modalities and filters redundant visual tokens before incorporating modality-specific quantization sensitivity for expert-level bit allocation. While MDMixQ adopts a distinct routing-only approach, the overlap in underlying motivations and overall design principles somewhat limits its methodological novelty.
  2. MDMixQ estimates expert importance by rectifying and accumulating raw router logits. However, unlike Softmax probabilities and Top-K routing, the proposed rectification function is not invariant to constant shifts in logits. For example, in a two-expert toy case, logits [2, 0] and [-8, -10] yield identical routing probabilities [0.881, 0.119], but the rectified score of the top expert changes from 2 to approximately 0.0003. Consequently, identical routing preferences can produce substantially different importance scores. Since MDMixQ ranks experts globally across layers for bit allocation, such sensitivity to logit offsets may distort expert rankings and lead to inconsistent precision assignments.
  3. Although SGFR is a central component of MDMixQ, the ablation studies do not separately evaluate the contributions of dynamic threshold filtering and nonlinear logit rectification. It would be helpful if the authors could provide independent ablations for these two components, along with a sensitivity analysis of the routing Top-K parameter.
Questions for authors

Please refer to the weaknesses above.

3 comments
AuthorsOpenReview ↗

W1 — Overlap with MODE

MDMixQ and MODE are concurrent works that share the principle of modality separation. MDMixQ uses confidence-filtered, rectified text-side router logits and global precision ranking. MODE combines modality-specific frequencies and quantization sensitivity with ILP allocation. Our MODE baseline retains these components and visual-token filtering, uses the same {2,3}\{2,3\} expert-bit choices as MDMixQ, and replaces GPTQ with the shared quantizer. On Qwen3-VL under the same 2.52-bit expert-weight budget, MDMixQ achieves 96.11% Retain versus 94.93% for MODE (Table 1). The advantage also holds with matched GPTQ: 97.85% versus 96.52% Retain. Full SGFR also improves on text-only frequency by 0.29 Retain points (W3).

We additionally investigate modality-decoupled activation quantization through the asymmetric NT4/T8 policy and provide operator implementation guidance supported by NPU latency measurements (Appendix A.4). MODE is already cited; we will expand the related work comparison in the revision.

AuthorsOpenReview ↗

W2 — Logit shift-invariance and cross-layer comparability

We evaluate the practical influence of logit offsets and layerwise score scales through controlled comparisons of precision assignments and accuracy.

The score-normalization and SGFR component ablations use Qwen3-VL, the same three independently re-sampled 50-example multimodal COCO calibration subsets, the shared quantizer, a 2.52-bit expert-weight budget, and 16-bit MoE activations. Inference routing and evaluation are fixed.

We compared the original salience with (i) per-layer z-score normalization of final expert salience before global ranking, and (ii) token-centered logits before rectification: g^t,il=gt,il−1Nl∑j=1Nlgt,jl.\hat g_{t,i}^{l}=g_{t,i}^{l}-\frac{1}{N_l}\sum_{j=1}^{N_l}g_{t,j}^{l}. Centering uses all experts at each token and layer. All three variants retain the backbone's native routed-expert set (Kr=8K_r=8 for Qwen3-VL) and the same softmax filtering mask. Token centering removes common additive logit offsets, mapping both [2, 0] and [-8, -10] in the reviewer's example to [1, -1]; z-scoring final salience tests the effect of layerwise score normalization.

Scoring variantRetain (%)Δ Retain vs. original (points)3-bit assignment overlap (%)
Original SGFR96.110100
Per-layer z-scored salience95.84-0.2795
Token-centered logits95.79-0.3289

Overlap is 100×∣E3original∩E3variant∣/∣E3original∣100\times|\mathcal E_3^{\mathrm{original}}\cap\mathcal E_3^{\mathrm{variant}}|/|\mathcal E_3^{\mathrm{original}}|, measured within each matched calibration subset and averaged across the three subsets.

Layerwise normalization and token centering preserve 95% and 89% of the original 3-bit expert set, while reducing Retain by 0.27 and 0.32 points, respectively. The largely preserved assignments and small Retain changes support the practical stability of our allocation under these two controls. Original SGFR achieves the highest Retain, providing empirical support for its global ranking in the evaluated Qwen3-VL setting.

AuthorsOpenReview ↗

W3 — Filtering/rectification ablation and native routing Top-K

We isolated threshold filtering and rectification using the first four conditions below. All four use the same native routed-expert pairs for text tokens (Kr=8K_r=8 for Qwen3-VL): disabling filtering removes only the confidence threshold, and disabling rectification uses raw logits in the accumulation. The final condition uses text-only expert activation frequency to isolate the gain beyond modality separation.

ConditionThreshold filteringRectificationRetain (%)
Raw-logit sumOffOff94.25
Filtering onlyOnOff93.51
Rectification onlyOffOn95.28
Full SGFROnOn96.11
Text-only activation frequency (MDC only)OffNot used95.82

The two stages are complementary in this setting. Full SGFR improves on filtering only and rectification only by 2.60 and 0.83 Retain points, respectively. Filtering alone underperforms raw-logit accumulation by 0.74 points, so its benefit depends on rectification in this comparison. Together, the two stages improve on raw-logit accumulation by 1.86 points. Full SGFR also exceeds text-only frequency by 0.29 points, showing an additional gain beyond modality separation alone.

Regarding routing Top-K, KrK_r follows each backbone's native routing configuration. SGFR collects importance statistics over the experts selected by the original router. We retain the native routing configuration in all experiments and isolate the contributions of filtering and rectification through the controlled ablations above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.