- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Owen-MoE to decide which experts should retain higher precision when quantizing MoE models. It groups experts with similar outputs, estimates their contributions using Owen values, and assigns them either 2-bit or 4-bit precision. It also protects experts that make substantial contributions within otherwise low-value coalitions. The authors evaluate the method on several MoE model checkpoints.
Critical strengths
This paper considers how experts work together when deciding which ones should retain higher precision. The comparison with flat Shapley attribution is particularly useful: Owen-MoE achieves higher accuracy with the same sampling and memory budgets, providing evidence that the coalition structure adds value.
Critical weaknesses
- The paper estimates expert contributions through masking in an expert availability game, but this does not directly measure the loss caused by quantizing an expert’s weights. An expert may be robust to low-bit quantization yet highly sensitive to being masked. The paper does not sufficiently establish how well availability-game contributions reflect quantization sensitivity.
- Comparisons with recent work are limited. Moreover, the matched-budget ranking comparisons require competing methods to use Owen-MoE’s per-layer 4-bit expert counts rather than their original bit allocators. These comparisons therefore do not establish whether Owen-MoE outperforms the complete methods at the same total expert-weight payload.
Questions for authors
Q1: Relationship between availability-game contributions and quantization sensitivity. Beyond the selected expert-pair interventions, is there broader evidence that contributions in the expert availability game reflect actual quantization sensitivity? Are there experts with large availability-game contributions that benefit little from retaining 4-bit rather than 2-bit precision? How common are such cases? Q2: Practical inference efficiency. The paper reports approximately 14 tokens/s on Jetson. Are there throughput and inference latency comparisons against uniform quantization or other mixed-precision methods on the same hardware under the same workload? Do these results show an actual speedup from the proposed bit allocation? Q3: Effects of calibration data and feature choice. Coalition construction and contribution estimates may depend on the calibration data distribution. Are accuracy results available using different calibration datasets? For coalition construction, have you compared expert input features, expert output features, and their combination? Feature choice and calibration data source are separate questions here. Q4: Comparisons with related methods. The MoPEQ comparisons are limited to the multimodal experiments in the appendix. Are further results available at matched expert-weight payloads? Could you explain why MODE was not included in the multimodal comparisons? For PreMoE, have you considered using its PEU scores in a ranking comparison with matched per-layer 4-bit expert counts to assess the additional benefit of Owen values? Q5: Performance on larger models. Are results available on larger MoE models, including accuracy gains and offline calibration costs? Do the observed advantages hold as model size increases?
7 comments
Response to Reviewer oRAr Part2
Q4. What changes with cross-layer allocation, and how do MoPEQ, MODE and PreMoE compare?
Cross-layer allocation. Giving Router-norm+MaxVar freedom to allocate precision across Qwen3 layers raises its accuracy by 2.69 and 3.42 points over inheriting Owen's layer quotas. Owen full retains a lead of 2.77 and 3.54 points at the corresponding budget targets. The first row gives the target expert-weight payload in bytes. The remaining rows give eight-task accuracy (%).
| Budget or allocation | 2.50 bits | 2.25 bits |
|---|---|---|
| Target expert-weight bytes | 9,059,696,640 | 8,153,726,976 |
| Router-norm+MaxVar, inherited quotas | 62.11 | 58.98 |
| Router-norm+MaxVar, cross-layer allocation | 64.80 | 62.40 |
| Owen full, submitted | 67.57 | 65.94 |
MoPEQ at a common reported model size. Following your suggestion, we re-evaluated Owen on DeepSeek-VL2-Tiny at the 2.367 GB model size reported for the MoPEQ Hessian-sensitivity (a) configuration. The matching quantity here is total model size as reported to three decimal places. The table gives each difference on the original benchmark scale.
| Task | Owen | MoPEQ | Owen minus MoPEQ |
|---|---|---|---|
| AI2D | 83.312 | 83.970 | −0.658 |
| DocVQA | 87.363 | 87.334 | +0.029 |
| InfoVQA | 60.403 | 61.645 | −1.242 |
| MME-Reasoning | 380.522 | 371.786 | +8.736 |
| MME-Perception | 1547.048 | 1561.112 | −14.064 |
| MMMU | 36.656 | 38.556 | −1.900 |
| RealWorldQA | 64.679 | 63.791 | +0.888 |
| ScienceQA | 87.550 | 87.172 | +0.378 |
| BLINK | 41.684 | 42.609 | −0.925 |
Owen scores higher on four of the nine tasks. For example, Owen gains 8.736 points on MME-Reasoning, while MoPEQ leads by 14.064 points on MME-Perception. DocVQA differs by just 0.029 points at this operating point.
Choice of multimodal baseline. Our main study concerns text-only MoE LLMs. For the multimodal extension, we chose MoPEQ because expert sensitivity can guide precision allocation using calibration data from either setting. MODE addresses a complementary question by estimating expert importance separately for visual and text tokens and filtering redundant visual tokens. Its visual calibration steps require visual inputs, so a text-only adaptation would omit part of the published method. We agree that MODE is relevant to our DeepSeek-VL2 evaluation. Its separate treatment of visual and text tokens suggests examining whether the same experts should be grouped together for both modalities.
PreMoE. Thank you for suggesting PreMoE's Predicted Expert Utility (PEU). It gave us a way to test how well router traces alone identify experts to protect. PEU fits directly into our allocation procedure because its ranking can fill the same four-bit slots in each layer. At 2.50 and 2.25 bits on Qwen3, PEU gives eight-task accuracies of 65.38% and 63.80%. Using Owen scores for the same slots gives 66.83% and 65.36%, gains of 1.45 and 1.56 points. For a deployment with recurring workloads, PEU could also provide an initial precision map from router traces collected during calibration.
If accepted, we will include a dedicated paragraph in the paper discussing both MODE and PreMoE. We will explain how MODE's modality-specific statistics could inform coalition construction and how PreMoE's workload-specific utility scores can be used within a fixed precision budget.
Q5. Do the gains persist on larger models, and what does calibration cost?
We agree that evaluating a larger checkpoint from the same model family would provide a more convincing test of how accuracy and calibration cost change with scale. Our current checkpoints differ in routing width, training and expert design as well as model size. At 2.75 expert bits, the eight-task gain over matched Router-norm is 0.39 points on Mixtral, 0.93 on Qwen1.5-MoE and 3.57 on Qwen3.
Mixtral calibration wall time (seconds) is shown below for both calibration pool sizes. MC denotes Monte Carlo trials.
| Calibration samples | MC = 8 | MC = 16 | MC = 32 | MC = 64 |
|---|---|---|---|---|
| 16 | 304 | 610 | 1216 | 2394 |
| 128 | 321 | 627 | 1227 | 2463 |
For Qwen3, counting the empty prefix and all successive prefixes gives game evaluations per seed.
The rebuttal period was too short to complete a larger-checkpoint evaluation. In future work, we will compare checkpoints within one family and report accuracy, calibration wall time and peak memory at matched expert-bit budgets.
Q1: Thank you for the additional analysis. It answers my question about high-score experts with small precision benefits, but does not fully resolve my concern. The median correlation of 0.29 and the small upgrade gains for 13 of the 32 sampled high-score experts suggest only a limited correspondence between availability contributions and individual quantization benefits. I understand that the method uses these contributions as a proxy and accounts for expert interactions, so these results do not invalidate the approach. However, I remain unconvinced that the evidence sufficiently supports the proxy as a reliable basis for precision allocation.
Q2: Thank you for the clarification. I agree that the Jetson result demonstrates deployment feasibility rather than a measured speedup. I also note that the DeepSeek-VL2-Tiny and Small results do not show a consistent accuracy advantage over the competing allocations at similar model sizes: Owen-MoE performs better on some tasks but falls behind on several others. Taken together, these results support feasibility, but leave the practical advantage of the proposed allocation on these multimodal models unclear.
Q3: Thank you for testing calibration data and feature choice separately. However, the second calibration corpus is not identified, and only averages are reported. This makes it difficult to judge whether the effects are consistent across tasks or whether gains on some tasks offset losses on others. The WikiText-output scores also differ from the main-table results, and it is unclear whether “top-quota selection” follows the original allocation procedure. These points leave me uncertain about how directly the additional results support the submitted method.
Response to Reviewer oRAr Part1
Q1. How often does a high availability score coincide with a small precision benefit?
Among 32 sampled high-score Qwen3 experts, 13 (40.6%) reduced layer reconstruction MSE by at most 1% when individually upgraded from two to four bits. Your question prompted us to test individual experts, which lets us estimate how often this case occurs beyond the selected disagreement pairs. Masking an expert allows the router to substitute another expert, whereas increasing its precision preserves its availability.
Across eight sampled layers, seven within-layer Spearman correlations between Owen score and individual upgrade gain were positive, with a median of 0.29. Median MSE reductions were 0.38%, 0.71% and 1.24% in the low-, middle- and high-score groups. Higher scores therefore were associated with larger median gains across these strata, with substantial variation among high-score experts. This complements the submitted intervention, in which the Owen-preferred upgrade won 75 of 96 selected expert-pair comparisons on all-token MSE and 80 of 96 on coactivated tokens.
Q2. Does the Jetson measurement establish a speedup?
Your question highlights the distinction between deployment feasibility and inference efficiency. DeepSeek-VL2-Tiny decoded at approximately 14 tokens/s on an 8 GB Jetson Orin Nano, where the 16-bit configuration exceeded available memory. The precision map is computed offline, and inference performance depends on the routing workload and low-bit kernels.
The rebuttal period was too short to complete further matched-workload measurements on Jetson. We regard these comparisons as an important direction for future work and will evaluate decode throughput and prefill latency against uniform and competing mixed-precision allocations on the same device under the same workload.
Q3. How sensitive is accuracy to the calibration corpus and the clustering features?
Following your suggestion, we varied the calibration corpus and the clustering features separately. Holding output features fixed, changing the corpus shifted eight-task accuracy by +0.96 and −0.19 points at 2.50 and 2.25 bits, respectively. We then compared the three feature choices on WikiText. All rows use Qwen3 top-quota selection and report eight-task mean accuracy (%).
On WikiText, input features give the highest score at 2.50 bits (+0.82 points over output). At 2.25 bits, input and output differ by just 0.04 points, with output slightly ahead. The feature ordering thus changes across the two budgets.