- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes IncMoE, which is a token-specific dynamic quantization scheme for expert weights of MoE models. It stores one copy of the weights and lets the precision vary based on the token, by incrementally loading most significant to least significant pairs of bits.
Here, each expert weight tile is quantized to 8 bits using a calibration-free, MSE optimized quantizer. Each of these 8 bits are split into 4 chunks of 2 bits. The most significant 2-bits are considered as bases, which are always loaded. The remaining 3 bit-pairs are loaded incrementally from most significant to least significant , based on necessity. This necessity is calculated based on these two scores: i) an offline, weight-only score (BitInfo) that ranks which tile’s next slice to load first. ii) the router’s gate weights, which give more slices to experts the router relies on more. In this way, based on the input token, the important/relevant expert tiles are loaded at higher precisions, compared to experts that do not contribute much for that token.
Experiments on this technique cover LLaMA-MoE-3.5B, Mixtral-8×7B, and Qwen3-30B-A3B: i) At an average 4-bit active-path budget, the authors claim that the quality is competitive with GPTQ, MxMoE, and MoEQuant. ii) Unseen-domain KL is lower than baselines calibrated only on GSM8K. iii) Batch-1 decoding latency falls between fixed INT4 and INT8.
Critical strengths
This paper provides a useful deployment strategy where one stored copy can be used to dynamically adjust precision at runtime: In this method, the same stored weights can be run at 2, 4, 6, or 8 bits, without keeping separate checkpoints. The precision can be changed per token. This lets a serving system trade speed for quality on the fly, which static quantization cannot do. The design is also simple and well specified.
Critical weaknesses
First, the comparisons provided in the paper are not matched for resource and does not fully justify the novelty of the paper:
i) IncMoE although on average reads and computes 4 bits per token, it stores the model in full 8-bit. However, Table 1 compares this model with other baselines which actually stores 4-bit representation of the model. Hence, the comparisons are not matched, as IncMoE has twice the resident memory. Moreover, even with half the resident memory, GPTQ-4 has better perplexity (5.60 vs. 5.68), comparable 6-task average on Mixtral-8x7B (Table 1), and presumably faster decoding speed (33.13 ms/token for INT4 vs. 37.62 ms/token for IncMoE-4, Table 4). Hence, the abstract’s claim of a “superior accuracy–memory Pareto frontier” is not supported.
ii) Did not provide comparison with other similar single-copy, multi-precision techniques such as Any-Precision LLM (Park et al., 2024), SliceMoE and DP-LLM (NeurIPS 2025), which were cited in the paper. Without these comparisons, the novelty reduces to the allocation rule (BitInfo+ router score). The experiments do not sufficiently isolate the benefit of that rule against any of these alternatives.
iii) A natural comparison baseline would have been to uniformly load 4 bits (first two 2bit chunks) from the same 8-bit representation. Table 4 reports latency for a fixed INT4 setting, but the quality of uniform truncation from the IncMoE representation is never reported to justify dynamic incremental loading. Table 5 reports a ‘Uniform’ variant (40.1 vs. 40.9 for BitInfo in the same panel).But it is not clearly defined, and it is evaluated only on a 50% TruthfulQA-Gen subset with ROUGE, from a single run.The paper needs perplexity and the full task suite for uniform truncation at 4, 5, and 6 bits on all three models. These comparisons would show whether this dynamic technique is worth its 13.6% latency overhead over INT4.
Second, empirical evidence presented for the proposed technique is weak:
i) All results come from single runs. Also, many reported margins are within noise. For example: IncMoE’s six-task mean is only 0.22 points higher than 4 bit MoEQuant on Mixtral. Moreover, several 4-bit models exceed FP16: IncMoE on TruthfulQA in LLaMA-MoE (26.19 vs 24.96) and Qwen3 (44.92 vs 39.05), and MoEQuant on Qwen3 GSM8K (89.31 vs 85.82). This suggests substantial evaluation variance.
ii) In order to show how dependence on calibration dataset can be harmful for samples of another domain, the baselines (AWQ, GPTQ, SmoothQuant) were calibrated on 128 GSM8K samples only. Then, they were tested on unseen domains. However, the standard practice is to calibrate on generic text (C4, WikiText), instead of calibrating on a dataset of a single domain. The paper acknowledges this, but the introduction’s claim that calibration shift “severely corrupt output distributions” rests on this setup. We can see in Table 1, the calibrated baselines perform similarly to IncMoE. So the practical harm of calibration is not evident in the main results (the calibration source is not stated).
Questions for authors
- Can you report the quality of uniform truncation (loading the same number of 2-bit chunks for every tile) at 4 and 6 bits. Moreover, can you also provide accuracy versus decoding latency for GPTQ/RTN at 4, 6, and 8 bits alongside IncMoE-4/6? If IncMoE gives better accuracy at the same latency, or clearly beats uniform truncation, that may justify the dynamic allocation.
- Can you compare with at least one cited single-copy, multi-precision method (Any-Precision LLM or SliceMoE) at the same average bit budget?
- How do AWQ, GPTQ, and SmoothQuant perform on the unseen domains in Table 3 when calibrated on generic text ( C4 or WikiText) instead of GSM8K only? If IncMoE still has lower KL, the calibration-robustness claim would be much stronger.
3 comments
3. Cross-Domain Robustness under General-Domain and Mixed-Domain Calibration
We introduce two additional calibration settings, C4 and Mixed-domain, to evaluate AWQ, GPTQ, and SmoothQuant. Mixed-domain calibration covers four categories of data: general text, mathematics, code, and commonsense reasoning. All methods use the same calibration budget (128 × 2,048 tokens), with consistent quantization configurations and evaluation protocols.
| Calibration Source | Best Calibrated Baseline (SmoothQuant) | IncMoE-4 | Relative KL Reduction |
|---|---|---|---|
| GSM8K-only | 1.541 | 1.404 | 8.9% |
| C4 | 1.516 | 1.404 | 7.4% |
| Mixed-domain | 1.494 | 1.404 | 6.0% |
These results show that changing the calibration dataset improves the cross-domain performance of existing methods, although the improvements remain relatively limited. SmoothQuant's average KL divergence across five domains decreases from 1.541 with GSM8K-only calibration to 1.516 with C4 and 1.494 with Mixed-domain calibration, corresponding to improvements of 1.6% and 3.0%, respectively. IncMoE requires no calibration and maintains an average KL divergence of 1.404, achieving reductions of 7.4% and 6.0% compared with the C4 and Mixed-domain calibration settings, respectively.
4. Statistical Reliability and Comparison with FP16
Quantized models occasionally outperforming FP16 on individual benchmarks is not uncommon in quantization research, and similar observations have been reported in prior studies. To address the reviewer's concerns regarding statistical reliability, we repeated the relevant evaluations using the original evaluation protocol. The reproduced results are largely consistent with those reported in the paper, and the main performance trends remain stable.
Importantly, we do not regard small improvements over FP16 or other baselines on individual benchmarks as primary evidence of our method's effectiveness. For example, IncMoE exceeds MoEQuant by only 0.22 percentage points in the six-task average on Mixtral. We do not interpret this small difference as a statistically significant or comprehensive advantage.
Response to the rebuttal
Thank you for the detailed response and the additional experiments. The uniform truncation results show that the allocation itself is useful, and the SliceMoE comparison and the C4 and mixed-domain calibration results address some of my concerns.
However, my main concern remains. The new latency table shows that GPTQ-4 is still better than IncMoE-4 in perplexity (5.60 vs. 5.68) and speed (36.18 vs. 37.62 ms/token), while using half the memory. At 6 bits, IncMoE-6 improves perplexity only slightly over GPTQ-6 (5.36 vs. 5.42) and is slower. So the practical benefit comes down to switching precision from one stored copy without calibration. The paper does not yet show a serving setting where this flexibility outweighs the extra memory and latency. The abstract’s “superior accuracy–memory Pareto frontier” claim is also still not supported, and the response does not say it will be revised.
In addition, the new comparisons are limited. The uniform truncation and SliceMoE results report only perplexity, the SliceMoE comparison uses a single model, and the repeated runs are described as “largely consistent” without any variance numbers. The calibration advantage also becomes smaller with broader calibration data (6.0% with mixed-domain).
For these reasons, I keep my score at 2.
1. Uniform Truncation and Quality–Latency Trade-off
We provide comparisons from two perspectives: the benefits of dynamic precision allocation under an identical weight representation, and the quality–latency trade-off against fixed-precision quantization methods.
(1) Uniform Truncation Comparison. We evaluate Uniform Truncation and IncMoE under exactly the same 8-bit weight representation, using identical quantized integers, scales, zero-points, tile partitioning, and decoding procedures. Uniform-4/6 consistently loads the first two or three 2-bit chunks for every tile, whereas IncMoE dynamically allocates the same total precision-increment budget according to the current token's router weights and BitInfo priorities.
For Mixtral, IncMoE reduces PPL from 5.84/5.44 to 5.68/5.36 under the 4-bit and 6-bit budgets, respectively. Similar trends are observed across all three models, indicating that the quality improvements arise not only from the nested quantization representation but also from router-conditioned, fine-grained allocation of precision increments.
(2) Quality–Latency Trade-off.
The primary objective of IncMoE is not to achieve the lowest latency at a single fixed precision, but to deliver model quality comparable to strong calibration-based quantization methods without requiring activation calibration, while supporting runtime-adjustable precision–latency trade-offs through a single nested weight representation.
We provide additional quality and decode latency results for Mixtral-8×7B on an NVIDIA H20. Under an average 4-bit execution budget, IncMoE achieves a PPL of 5.68 without calibration, approaching GPTQ-4's 5.60. Its average score across six tasks reaches 63.54, slightly exceeding GPTQ-4's 63.30. Increasing the execution budget to 6 bits further reduces IncMoE's PPL to 5.36, approaching FP16's 5.34, while achieving a six-task average of 65.04, nearly identical to FP16's 65.08.
Regarding execution efficiency, IncMoE-4/6 achieves measured decode latencies of 37.62/45.88 ms/token, respectively. Dynamic allocation introduces additional overhead from increment indexing, scheduling, and accumulation compared with highly optimized fixed low-bit execution. Nevertheless, IncMoE-6 still reduces latency by approximately 7.6% relative to fixed INT8 (49.66 ms/token), while maintaining near-FP16 model quality.
Together with the Uniform Truncation comparison under the identical 8-bit representation, these results primarily validate the efficiency of fine-grained dynamic allocation in utilizing a fixed execution budget. The key advantage of IncMoE is its ability to maintain competitive low-bit model quality without additional training or activation calibration, while dynamically allocating precision across tokens using a single resident representation, without maintaining or switching between multiple fixed-precision weight versions.
2. Comparison with Existing Single-Representation Multi-Precision Methods
We select SliceMoE as a direct baseline. SliceMoE constructs a nested weight representation using calibration-free AMAT and dynamically selects execution precision at the expert level based on router gating scores and cache states, primarily optimizing expert loading and caching efficiency on the device. In contrast, IncMoE combines weight-only BitInfo with fine-grained allocation of a limited precision budget across experts, projections, and tiles along each token's active execution path, improving execution-budget utilization.
We conduct the comparison on Qwen3-30B-A3B. SliceMoE uses its AMAT representation and DBSC-based expert-level precision selection, adjusting the proportion of 4-bit and 8-bit experts to match average active-path execution budgets of 5, 6, and 7 bits.
Compared with selecting between low and high precision at the expert level, fine-grained tile-level increment allocation further improves model quality under the same average execution budget. Particularly at intermediate precision budgets, IncMoE avoids uniformly increasing the precision of an entire expert and instead concentrates limited precision increments on regions with higher weight sensitivity.