- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses mixed-precision quantization (MPQ) for multimodal MoE models and proposes Config-Owen. The core idea: replace the "scalar importance of expert E_i" with a "configuration-valued utility"—the same 2/4-bit local configuration can produce different end-to-end loss increments depending on the "outer context" (preceding expert configurations, co-activated experts at the same layer, subsequent expert configurations). The method comprises five objects: hierarchical coalition structure, configuration-valued utility, Owen-type layered attribution (compressed via endpoint differences), CASP + RPOS sampling estimator, and multiple-choice knapsack (MCKP) allocation with feasibility mask and effective-bit ledger.
Critical strengths
S1. Clear problem motivation Traditional MoE MPQ treats "expert importance" as a scalar independent of other experts' precision. The authors directly attack this assumption and use the Table 7 ablation chain (Expert-only < Projection-Owen < Local-Calib-MCKP < Config-Owen) to demonstrate that "hierarchy + context" are genuinely effective factors. This gradient-style improvement provides empirical support for the claim that "configuration-valued utility outperforms scalar importance."
S2. Good experimental coverage and ledger transparency; main results are internally consistent. Five models span 1B–7×8B parameters, unimodal/multimodal, and top-2/top-8/top-6 routing configurations. Appendix C Table 6 provides raw per-task scores for all eight methods;
Critical weaknesses
W1. Baseline coverage is thin, and there is no effectiveness closure between the extensive mathematical derivations and the actual algorithm.
All existing main comparisons (Uniform / Hessian / PMQ) are fundamentally the same paradigm—"compute a scalar score per unit → sort by score → allocate precision"—differing only in the scoring kernel. PMQ is Config-Owen's most direct "degenerate special case" (scalar vs. configuration-valued), Hessian is "another scalar," and Uniform is "no allocation at all." All three are strict subsets of Config-Owen's design space.
But MPQ is not limited to this single route:
- General PTQ + mixed precision: AWQ , SmoothQuant (difficulty migration to weights), OmniQuant (differentiable quantization parameter learning) can all naturally extend to per-expert/per-projection mixed-precision allocation. The authors cite these methods (§2) but ran none in experiments.
- Non-importance-based MoE quantization: KBVQ-MoE (vector quantization), MoEQuant (expert-balanced + routing reweighting), TileQ (low-rank MoE) are also entirely absent.
- Critical ablation missing: How much worse would a scheme with "AWQ scores + the same MCKP solver" be? If the gap is small, the complex attribution mechanism lacks independent justification.
Furthermore, the paper's mathematical volume (Appendices N–U, ~25 pages) far exceeds comparable papers, involving seven-term Fréchet perturbation decomposition, continuous–discrete integral correspondence, Walsh orthogonal spectrum, double Möbius tensor Γ contraction, Rényi-2 divergence and trace-variance optimal proposal, Lagrangian frontier and reduced-cost stability, multi-level product lattices, and calibration-to-population end-to-end guarantees. However:
- Most analytical tools do not appear in the algorithm path. Algorithm 1's core is endpoint-difference accumulation → aggregation → MCKP (i.e., Owen value estimation). The theoretical objects actually used are limited to the endpoint representation (Eq. 4) and the Owen definition (Eq. 6). Harsanyi coefficients ρ(S,T), the Γ tensor, Walsh spectrum, Rényi divergence, Fréchet derivatives, product-lattice DCP, etc. appear only in proofs and discussions, never participating in actual computation. This does not mean the theory is "useless"—the Owen framework genuinely provides conceptual justification for "why endpoint differences are valid scoring." However, the large volume of analysis unrelated to algorithmic implementation may mislead readers into believing these tools are necessary for the method's correctness.
- Theory inspired design choices, but approximation quality was never validated. Appendix Q.2 gives the trace-variance optimal proposal q*(π) ∝ p(π)‖Y(π)‖₂, and the main text explicitly states that router-mass RPOS is a "tractable approximation" to q*—this design was indeed theory-inspired. But the variance gap between q* and router-mass RPOS was never empirically measured, nor were Rényi-2 divergence or d_eff values reported, leaving readers unable to assess how tight the approximation is.
- Key constants in theoretical bounds were never empirically evaluated. Appendix S's β_{ik}, Appendix R's Γ_B, and Appendix Q's d_eff are all unmeasured. Readers cannot judge whether these bounds are "meaningful" or "vacuous decoration."
W2. Some main comparisons have Δ on the same order as noise
Paper §4.1 states "seed variation is reported separately rather than repeated in the main table." Appendix D.3 Table 4 gives 3-seed Avg. std as ±0.28 and ±0.31 for MolmoE-1B and DSVL2-L respectively. The following three Ours−PMQ cells fall within σ:
| Model | Avg bits | Ours | PMQ | Δ |
|---|---|---|---|---|
| MolmoE-1B | 2.30 | 46.21 | 46.12 | +0.09 |
| DSVL2-L | 2.57 | 70.97 | 70.60 | +0.37 |
| DSVL2-L | 2.30 | 70.38 | 70.25 | +0.13 |
These three cells are "bolded as best quantized" in the main table but are statistically indistinguishable from zero given the reported σ. The conclusion/abstract both claim "consistently improving the matched-budget average"—if three cells lack statistical significance, this statement needs qualification. Additionally:
- 5 models × 2 budgets = 10 cells, but only 2 models report σ; DSVL2-S/T and Mixtral seed variance is unreported.
- Table 1 shows DSVL2-S 2.30 MMB = 78.17 > 2.58's 77.24, a non-monotonicity ("tighter budget performs better") with no annotation indicating whether this is seed noise.
W3. CASP behavior contradicts the "bias-for-time" narrative; hyperparameter sensitivity is entirely absent.
Table 7 reports the effect of removing CASP at DSVL2-L @2.57 and DSVL2-S @2.58:
CASP simultaneously reduces time and improves accuracy. If it were merely a "bias term," removing it should yield equal or better Avg. A plausible explanation: avoiding collapse-distorted evaluations makes MCKP scoring more effective. The authors need to restate CASP's semantics—it is not merely "candidate reduction" but also "attribution quality enhancement."
Questions for authors
- What are the core advantages of your method, and does it have a precision advantage compared to methods like Omniquant and QTIP that do not require additional bits? Is there a significant advantage compared to methods like TileQ that also require additional bit positions?
- Baseline effectiveness: Can you run an ablation with "AWQ scores + Config-Owen's MCKP solver"? If the result is close to full Config-Owen (<1 pts gap), the core contribution may lie in MCKP's global allocation rather than the Owen endpoint-difference scoring.
- CASP causal mechanism: What explains CASP simultaneously reducing time and improving accuracy? Can you provide a micro-analysis showing "if CASP-pruned candidates were fully evaluated, the resulting allocation would degrade"?
- Hyperparameter sensitivity: Avg.% curves for τ ∈ {0.3, 0.5, 0.7, 0.9}? Does doubling M from 20 to 40 contract ε_r as the theory predicts?
13 comments
Part2
B. On the statistical reliability of the reported improvements.
Our original experiments use three random seeds for Config-Owen, but report standard deviations only for DeepSeek-VL2-S. We therefore extend the statistical evaluation to five seeds across all five models and both matched-budget settings.
Multi-seed comparison:
| Model | Bits | PMQ Reference Avg.% | Config-Owen (Mean ± Std) | Difference |
|---|---|---|---|---|
| MolmoE-1B | 2.60 | 47.67 | 48.94 ± 0.10 | +1.27 |
| MolmoE-1B | 2.30 | 46.12 | 46.20 ± 0.25 | +0.08 |
| DeepSeek-VL2-L | 2.57 | 70.60 | 71.01 ± 0.12 | +0.41 |
| DeepSeek-VL2-L | 2.30 | 70.25 | 70.43 ± 0.10 | +0.18 |
| DeepSeek-VL2-S | 2.58 | 63.66 | 67.93 ± 0.28 | +4.27 |
| DeepSeek-VL2-S | 2.30 | 62.73 | 67.41 ± 0.31 | +4.68 |
| DeepSeek-VL2-T | 2.59 | 59.46 | 60.57 ± 0.14 | +1.11 |
| DeepSeek-VL2-T | 2.30 | 58.57 | 60.19 ± 0.15 | +1.62 |
| Mixtral-8×7B | 2.54 | 67.50 | 68.88 ± 0.13 | +1.38 |
| Mixtral-8×7B | 2.30 | 65.16 | 66.51 ± 0.13 | +1.35 |
The results show that the improvements vary across models. The gains are more pronounced on DeepSeek-VL2-S, DeepSeek-VL2-T, and Mixtral, while some differences on MolmoE-1B and DeepSeek-VL2-L are relatively small. We therefore distinguish substantial gains from smaller differences rather than suggesting that the improvements are equally large across all models.
We also provide task-level results for each seed on all five models in Appendix E. These results allow us to examine the higher MMBench score at 2.30 bits on DeepSeek-VL2-S. Although this task shows a non-monotonic trend, the overall average remains higher at 2.58 bits. This is expected because our allocation objective optimizes calibration utility under a global memory budget rather than individual benchmark scores.
Part3
C. On the CASP mechanism and its threshold sensitivity.
We agree that CASP deserves further evaluation beyond its computational savings. It stops evaluating the remaining expert suffixes once calibration performance enters a collapsed regime. We therefore examine how different thresholds affect the final allocation and downstream performance.
CASP threshold sensitivity on DeepSeek-VL2-S:
| Threshold (tau) | MMB | MMStar | MMMU | AI2D | OCRBench | MME | Avg.% | Utility Time (min) |
|---|---|---|---|---|---|---|---|---|
| 0.3 | 75.62 | 55.14 | 44.71 | 76.32 | 79.25 | 1678.34 | 66.21 | 57 |
| 0.5 | 77.24 | 56.27 | 46.67 | 78.27 | 81.20 | 1697.07 | 67.93 | 43 |
| 0.7 | 76.10 | 55.63 | 45.82 | 77.12 | 79.84 | 1682.46 | 66.90 | 31 |
| 0.9 | 73.84 | 53.92 | 43.50 | 75.46 | 78.21 | 1658.72 | 64.99 | 22 |
The default threshold of 0.5 gives the best overall result. Lower thresholds require more computation, while higher thresholds lead to more aggressive pruning and larger performance drops. This supports our choice of 0.5 as a practical balance between computation and allocation quality.
Part1
We sincerely thank the reviewer for the careful reading and thoughtful comments. We especially appreciate the specific questions about our baselines, experimental reliability, and sampling strategy. These comments have helped us identify several aspects that need further validation and clearer explanation. We have therefore conducted additional experiments and revised our discussion accordingly. Our responses are provided below.
A.On baseline coverage and the choice of additional comparisons.
We thank the reviewer for suggesting several relevant quantization methods. After examining them, we find that they mainly address different problems from the precision-allocation problem studied in our work.
Specifically, SmoothQuant focuses on reducing activation quantization difficulty, while QTIP and KBVQ-MoE use vector-quantized weight representations. MoEQuant focuses on expert-aware quantization, whereas TileQ uses low-rank structures for compression. Although these methods are relevant to MoE quantization, their original formulations do not directly allocate different bit-widths to individual experts and their projections under a global memory budget. A direct comparison would therefore involve different quantization formats and optimization objectives.
Among the methods mentioned by the reviewer, AWQ and OmniQuant provide more suitable starting points for additional allocation baselines. We adapt their quantization sensitivity and reconstruction-error scores to projection-level utilities, while keeping the MCKP solver, GPTQ backend, and memory budgets identical to Config-Owen.
Additional comparisons on DeepSeek-VL2-S:
These comparisons help isolate the contribution of our utility estimation from that of the global optimizer.We will also expand the related-work discussion to include the five methods mentioned above and clarify how their quantization and compression techniques can complement our precision-allocation framework.