acceptodds
Under review as a conference paper at ICLR 2027

IncMoE: Active-Path Precision Increment for Calibration-Free Mixture-of-Expert Quantization

Abstract

Mixture-of-Experts Models (MoE) achieve superior scaling efficiency, yet their massive parameter volume incurs prohibitive memory footprints and IO transfer overheads during serving. Mixed-precision quantization mitigates these bottlenecks by allocating non-uniform bit-widths across experts. However, existing paradigms suffer from two fundamental limitations: (i) static mixed-precision schemes enforce rigid allocations that fail to adapt to token-dependent routing dynamics, while multi-copy dynamic-precision approaches incur prohibitive memory duplication by storing multiple discrete weight copies; and (ii) conventional allocation heuristics rely on calibration activations that are vulnerable to runtime distribution shifts. To overcome these constraints, we propose IncMoE, a calibration-free, single-copy framework enabling runtime router-conditioned precision refinement. IncMoE reformulates precision from a static expert property into a dynamic trajectory of progressive refinements executed along routed paths. Specifically, each weight tensor is physically encoded once into nested, ordered bit slices comprising a compact base layer and progressively loadable increments, supporting variable precisions without parameter duplication. To identify critical increments without calibration data, we introduce BitInfo, which provides a data-free ranking score for individual bit slices directly from weight distributions to establish priority hierarchies. At runtime, a lightweight scheduler combines instantaneous gating confidences with offline priority lists, dynamically assigning prefix-consistent bit increments to active experts under a target bit budget. Experiments across three representative MoE architectures show that IncMoE approaches full-precision perplexity at an average 6-bit budget and preserves competitive downstream quality at 4 bits while strictly maintaining single-copy storage, establishing a superior accuracy-memory Pareto frontier for MoE serving.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture-of-Experts, Model Quantization, Dynamic Precision, Calibration-Free Quantization, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8zBkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IncMoE, which is a token-specific dynamic quantization scheme for expert weights of MoE models. It stores one copy of the weights and lets the precision vary based on the token, by incrementally loading most significant to least significant pairs of bits.

Here, each expert weight tile is quantized to 8 bits using a calibration-free, MSE optimized quantizer. Each of these 8 bits are split into 4 chunks of 2 bits. The most significant 2-bits are considered as bases, which are always loaded. The remaining 3 bit-pairs are loaded incrementally from most significant to least significant , based on necessity. This necessity is calculated based on these two scores: i) an offline, weight-only score (BitInfo) that ranks which tile’s next slice to load first. ii) the router’s gate weights, which give more slices to experts the router relies on more. In this way, based on the input token, the important/relevant expert tiles are loaded at higher precisions, compared to experts that do not contribute much for that token.

Experiments on this technique cover LLaMA-MoE-3.5B, Mixtral-8×7B, and Qwen3-30B-A3B: i) At an average 4-bit active-path budget, the authors claim that the quality is competitive with GPTQ, MxMoE, and MoEQuant. ii) Unseen-domain KL is lower than baselines calibrated only on GSM8K. iii) Batch-1 decoding latency falls between fixed INT4 and INT8.

Critical strengths

This paper provides a useful deployment strategy where one stored copy can be used to dynamically adjust precision at runtime: In this method, the same stored weights can be run at 2, 4, 6, or 8 bits, without keeping separate checkpoints. The precision can be changed per token. This lets a serving system trade speed for quality on the fly, which static quantization cannot do. The design is also simple and well specified.

Critical weaknesses

First, the comparisons provided in the paper are not matched for resource and does not fully justify the novelty of the paper:

i) IncMoE although on average reads and computes 4 bits per token, it stores the model in full 8-bit. However, Table 1 compares this model with other baselines which actually stores 4-bit representation of the model. Hence, the comparisons are not matched, as IncMoE has twice the resident memory. Moreover, even with half the resident memory, GPTQ-4 has better perplexity (5.60 vs. 5.68), comparable 6-task average on Mixtral-8x7B (Table 1), and presumably faster decoding speed (33.13 ms/token for INT4 vs. 37.62 ms/token for IncMoE-4, Table 4). Hence, the abstract’s claim of a “superior accuracy–memory Pareto frontier” is not supported.

ii) Did not provide comparison with other similar single-copy, multi-precision techniques such as Any-Precision LLM (Park et al., 2024), SliceMoE and DP-LLM (NeurIPS 2025), which were cited in the paper. Without these comparisons, the novelty reduces to the allocation rule (BitInfo+ router score). The experiments do not sufficiently isolate the benefit of that rule against any of these alternatives.

iii) A natural comparison baseline would have been to uniformly load 4 bits (first two 2bit chunks) from the same 8-bit representation. Table 4 reports latency for a fixed INT4 setting, but the quality of uniform truncation from the IncMoE representation is never reported to justify dynamic incremental loading. Table 5 reports a ‘Uniform’ variant (40.1 vs. 40.9 for BitInfo in the same panel).But it is not clearly defined, and it is evaluated only on a 50% TruthfulQA-Gen subset with ROUGE, from a single run.The paper needs perplexity and the full task suite for uniform truncation at 4, 5, and 6 bits on all three models. These comparisons would show whether this dynamic technique is worth its 13.6% latency overhead over INT4.

Second, empirical evidence presented for the proposed technique is weak:

i) All results come from single runs. Also, many reported margins are within noise. For example: IncMoE’s six-task mean is only 0.22 points higher than 4 bit MoEQuant on Mixtral. Moreover, several 4-bit models exceed FP16: IncMoE on TruthfulQA in LLaMA-MoE (26.19 vs 24.96) and Qwen3 (44.92 vs 39.05), and MoEQuant on Qwen3 GSM8K (89.31 vs 85.82). This suggests substantial evaluation variance.

ii) In order to show how dependence on calibration dataset can be harmful for samples of another domain, the baselines (AWQ, GPTQ, SmoothQuant) were calibrated on 128 GSM8K samples only. Then, they were tested on unseen domains. However, the standard practice is to calibrate on generic text (C4, WikiText), instead of calibrating on a dataset of a single domain. The paper acknowledges this, but the introduction’s claim that calibration shift “severely corrupt output distributions” rests on this setup. We can see in Table 1, the calibrated baselines perform similarly to IncMoE. So the practical harm of calibration is not evident in the main results (the calibration source is not stated).

Questions for authors
  1. Can you report the quality of uniform truncation (loading the same number of 2-bit chunks for every tile) at 4 and 6 bits. Moreover, can you also provide accuracy versus decoding latency for GPTQ/RTN at 4, 6, and 8 bits alongside IncMoE-4/6? If IncMoE gives better accuracy at the same latency, or clearly beats uniform truncation, that may justify the dynamic allocation.
  2. Can you compare with at least one cited single-copy, multi-precision method (Any-Precision LLM or SliceMoE) at the same average bit budget?
  3. How do AWQ, GPTQ, and SmoothQuant perform on the unseen domains in Table 3 when calibrated on generic text ( C4 or WikiText) instead of GSM8K only? If IncMoE still has lower KL, the calibration-robustness claim would be much stronger.
3 comments

1. Uniform Truncation and Quality–Latency Trade-off

We provide comparisons from two perspectives: the benefits of dynamic precision allocation under an identical weight representation, and the quality–latency trade-off against fixed-precision quantization methods.

(1) Uniform Truncation Comparison. We evaluate Uniform Truncation and IncMoE under exactly the same 8-bit weight representation, using identical quantized integers, scales, zero-points, tile partitioning, and decoding procedures. Uniform-4/6 consistently loads the first two or three 2-bit chunks for every tile, whereas IncMoE dynamically allocates the same total precision-increment budget according to the current token's router weights and BitInfo priorities.

ModelUniform-4IncMoE-4Uniform-6IncMoE-6FP16
LLaMA-MoE-3.5B10.8810.7510.6110.5610.48
Mixtral-8×7B5.845.685.445.365.34
Qwen3-30B-A3B12.2012.0311.8211.7211.57

For Mixtral, IncMoE reduces PPL from 5.84/5.44 to 5.68/5.36 under the 4-bit and 6-bit budgets, respectively. Similar trends are observed across all three models, indicating that the quality improvements arise not only from the nested quantization representation but also from router-conditioned, fine-grained allocation of precision increments.

(2) Quality–Latency Trade-off.

The primary objective of IncMoE is not to achieve the lowest latency at a single fixed precision, but to deliver model quality comparable to strong calibration-based quantization methods without requiring activation calibration, while supporting runtime-adjustable precision–latency trade-offs through a single nested weight representation.

We provide additional quality and decode latency results for Mixtral-8×7B on an NVIDIA H20. Under an average 4-bit execution budget, IncMoE achieves a PPL of 5.68 without calibration, approaching GPTQ-4's 5.60. Its average score across six tasks reaches 63.54, slightly exceeding GPTQ-4's 63.30. Increasing the execution budget to 6 bits further reduces IncMoE's PPL to 5.36, approaching FP16's 5.34, while achieving a six-task average of 65.04, nearly identical to FP16's 65.08.

Regarding execution efficiency, IncMoE-4/6 achieves measured decode latencies of 37.62/45.88 ms/token, respectively. Dynamic allocation introduces additional overhead from increment indexing, scheduling, and accumulation compared with highly optimized fixed low-bit execution. Nevertheless, IncMoE-6 still reduces latency by approximately 7.6% relative to fixed INT8 (49.66 ms/token), while maintaining near-FP16 model quality.

Together with the Uniform Truncation comparison under the identical 8-bit representation, these results primarily validate the efficiency of fine-grained dynamic allocation in utilizing a fixed execution budget. The key advantage of IncMoE is its ability to maintain competitive low-bit model quality without additional training or activation calibration, while dynamically allocating precision across tokens using a single resident representation, without maintaining or switching between multiple fixed-precision weight versions.

MethodPPL ↓Latency (ms/token) ↓
RTN-45.7635.92
GPTQ-45.6036.18
Uniform-45.8436.39
IncMoE-45.6837.62
RTN-65.4944.20
GPTQ-65.4244.71
Uniform-65.4444.43
IncMoE-65.3645.88

2. Comparison with Existing Single-Representation Multi-Precision Methods

We select SliceMoE as a direct baseline. SliceMoE constructs a nested weight representation using calibration-free AMAT and dynamically selects execution precision at the expert level based on router gating scores and cache states, primarily optimizing expert loading and caching efficiency on the device. In contrast, IncMoE combines weight-only BitInfo with fine-grained allocation of a limited precision budget across experts, projections, and tiles along each token's active execution path, improving execution-budget utilization.

We conduct the comparison on Qwen3-30B-A3B. SliceMoE uses its AMAT representation and DBSC-based expert-level precision selection, adjusting the proportion of 4-bit and 8-bit experts to match average active-path execution budgets of 5, 6, and 7 bits.

Avg. BitsMethodPPL ↓
5SliceMoE11.99
5IncMoE11.86
6SliceMoE11.82
6IncMoE11.72
7SliceMoE11.70
7IncMoE11.63
16FP1611.57

Compared with selecting between low and high precision at the expert level, fine-grained tile-level increment allocation further improves model quality under the same average execution budget. Particularly at intermediate precision budgets, IncMoE avoids uniformly increasing the precision of an entire expert and instead concentrates limited precision increments on regions with higher weight sensitivity.

3. Cross-Domain Robustness under General-Domain and Mixed-Domain Calibration

We introduce two additional calibration settings, C4 and Mixed-domain, to evaluate AWQ, GPTQ, and SmoothQuant. Mixed-domain calibration covers four categories of data: general text, mathematics, code, and commonsense reasoning. All methods use the same calibration budget (128 × 2,048 tokens), with consistent quantization configurations and evaluation protocols.

Calibration SourceBest Calibrated Baseline (SmoothQuant)IncMoE-4Relative KL Reduction
GSM8K-only1.5411.4048.9%
C41.5161.4047.4%
Mixed-domain1.4941.4046.0%

These results show that changing the calibration dataset improves the cross-domain performance of existing methods, although the improvements remain relatively limited. SmoothQuant's average KL divergence across five domains decreases from 1.541 with GSM8K-only calibration to 1.516 with C4 and 1.494 with Mixed-domain calibration, corresponding to improvements of 1.6% and 3.0%, respectively. IncMoE requires no calibration and maintains an average KL divergence of 1.404, achieving reductions of 7.4% and 6.0% compared with the C4 and Mixed-domain calibration settings, respectively.

4. Statistical Reliability and Comparison with FP16

Quantized models occasionally outperforming FP16 on individual benchmarks is not uncommon in quantization research, and similar observations have been reported in prior studies. To address the reviewer's concerns regarding statistical reliability, we repeated the relevant evaluations using the original evaluation protocol. The reproduced results are largely consistent with those reported in the paper, and the main performance trends remain stable.

Importantly, we do not regard small improvements over FP16 or other baselines on individual benchmarks as primary evidence of our method's effectiveness. For example, IncMoE exceeds MoEQuant by only 0.22 percentage points in the six-task average on Mixtral. We do not interpret this small difference as a statistically significant or comprehensive advantage.

Reviewer_8zBkOpenReview ↗

Response to the rebuttal

Thank you for the detailed response and the additional experiments. The uniform truncation results show that the allocation itself is useful, and the SliceMoE comparison and the C4 and mixed-domain calibration results address some of my concerns.

However, my main concern remains. The new latency table shows that GPTQ-4 is still better than IncMoE-4 in perplexity (5.60 vs. 5.68) and speed (36.18 vs. 37.62 ms/token), while using half the memory. At 6 bits, IncMoE-6 improves perplexity only slightly over GPTQ-6 (5.36 vs. 5.42) and is slower. So the practical benefit comes down to switching precision from one stored copy without calibration. The paper does not yet show a serving setting where this flexibility outweighs the extra memory and latency. The abstract’s “superior accuracy–memory Pareto frontier” claim is also still not supported, and the response does not say it will be revised.

In addition, the new comparisons are limited. The uniform truncation and SliceMoE results report only perplexity, the SliceMoE comparison uses a single model, and the repeated runs are described as “largely consistent” without any variance numbers. The calibration advantage also becomes smaller with broader calibration data (6.0% with mixed-domain).

For these reasons, I keep my score at 2.

Reviewer_LFmiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IncMoE, a calibration-free method for dynamic precision allocation in MoE models. It stores expert weights in a shared 8-bit representation and adjusts execution precision by selecting ordered 2-bit increments. The proposed BitInfo score combines increment energy and symbol surprisal to rank refinements.

Experiments on three MoE architectures show competitive performance at a 4-bit execution budget. Additional results demonstrate that the method can adjust quality and decoding latency without changing the stored representation.

Critical strengths
  • The proposed representation and allocation procedure are clearly formulated, with sound reconstruction and budget-feasibility arguments.

  • The experiments cover three MoE architectures and show promising performance without calibration data.

Critical weaknesses
  1. The novelty compared with existing methods is not clear. D²MoE already supports dynamic expert precision and nested weight storage, but this work is not discussed in the paper. IncMoE mainly differs in its calibration-free design and finer allocation granularity, but experimental results do not clearly separate the benefits of these two aspects.

  2. The accuracy–memory advantage are overclaiming. Table 1 compares methods at the same execution bit budget, while IncMoE still stores the complete 8-bit representation. The storage comparison in Table 4 mainly includes FP16 and multiple separate checkpoints.

  3. Limited validations of BitInfo needs. Table 5 only reports results on a TruthfulQA subset using Mixtral, and the improvement over the energy-only baseline is small. The exact weight-error reduction described in the paper can also be computed without calibration data.

  4. Missed implementation details. The stated truncation procedure also introduces a one-sided error relative to the 8-bit reconstruction. It is unclear whether the implementation applies any correction to this error.

Questions for authors
  • What are the main technical differences from D²MoE, and how much benefit comes from projection- and tile-level allocation compared with expert-level allocation?

  • Does the domain-shift advantage remain when the baselines use an equally sized general-domain or mixed-domain calibration set?

4 comments

1. Distinction from D²MoE and the Contribution of Fine-Grained Allocation

D²MoE already supports dynamic expert precision selection and nested weight storage, but the two methods differ fundamentally in their precision decision mechanisms and allocation granularity. D²MoE relies on an additionally trained Bit-width Router and activation calibration to select execution precision at the expert level. In contrast, IncMoE directly reuses the original MoE Router and incorporates weight-only BitInfo, requiring neither additional training nor activation calibration, while extending precision allocation to projections and tiles. The key insight is that different weight regions within the same expert exhibit different quantization sensitivities, making uniform expert-level precision allocation insufficient to fully utilize a limited budget. Through Router-conditioned Allocation and local increment prioritization, IncMoE preferentially recovers more valuable weight information while satisfying prefix dependencies and budget constraints. To isolate the contribution of fine-grained allocation, we keep the weight representation, BitInfo criterion, and average 4-bit execution budget identical, varying only the allocation granularity.

Allocation GranularityLLaMA-MoE-3.5BMixtral-8×7BQwen3-30B-A3B
FP16 Reference10.485.3411.57
Expert-only10.905.8712.28
Expert + Projection10.825.7712.15
Full IncMoE (+ Tile)10.755.6812.03

PPL consistently decreases across all three models as allocation granularity is refined from experts to projections and tiles. Compared with Expert-only allocation, full IncMoE reduces PPL by 0.15, 0.19, and 0.25 on LLaMA-MoE, Mixtral, and Qwen3, respectively. IncMoE's core innovation is not merely enabling dynamic precision or nested storage, but achieving token-routing-aware, fine-grained incremental precision allocation without additional training or activation calibration. Unlike D²MoE's Expert-level Precision Routing and I/O–Compute Scheduling, IncMoE focuses on improving precision-budget utilization under a fixed execution budget. The two approaches therefore address different technical objectives.

2. Cross-Domain Robustness under General-Domain and Mixed-Domain Calibration

To address this concern, we introduce two additional calibration settings, C4 and Mixed-domain, and compare AWQ, GPTQ, and SmoothQuant. Mixed-domain calibration covers general text, mathematics, code, and commonsense reasoning. All methods use the same calibration budget (128 × 2,048 tokens) and consistent quantization configurations.

Our results show that changing the calibration dataset improves baseline cross-domain performance, although the gains remain relatively modest. The best baseline, SmoothQuant, reduces its average KL divergence across five domains from 1.541 with GSM8K-only calibration to 1.516 with C4 and 1.494 with Mixed-domain calibration, corresponding to improvements of 1.6% and 3.0%, respectively. In contrast, calibration-free IncMoE achieves an average KL divergence of 1.404, remaining 7.4% and 6.0% lower than the two additional calibration settings, respectively. Broader calibration-domain coverage can partially mitigate distribution bias, but does not eliminate IncMoE's relative advantage.

3. Clarification of the Accuracy–Memory Advantage

We clarify that IncMoE's storage advantage does not come from compressing the model to 4 bits, but from supporting multiple runtime execution precisions through a single nested weight representation, without storing multiple independent quantized copies. Our paper consistently distinguishes resident storage from the active-path execution budget: 4-bit refers to the average executed precision, while the complete 8-bit weight information and associated metadata are retained.

For Mixtral, IncMoE's complete expert representation occupies 44.46 GiB, including weights, quantization metadata, and priority indices. In contrast, independently storing 4/6/8-bit quantized versions requires 95.98 GiB, representing a 53.7% storage reduction. Moreover, by adjusting only the runtime increment allocation, IncMoE supports 4/5/6-bit execution using the same weight representation, achieving PPL values of 5.68/5.49/5.36, respectively, without requantization, switching model weights, or maintaining additional copies.

Therefore, IncMoE's core storage advantage is providing multi-precision execution capability at a storage cost close to that of a single INT8 representation, rather than claiming lower memory consumption than a single fixed INT4 deployment. Table 1 compares model quality under identical execution budgets, whereas Table 4 independently quantifies the storage savings of a single nested representation over multi-copy deployment. These tables evaluate different dimensions, and a 4-bit execution budget is never equated with 4-bit storage.

4. Necessity and Effectiveness of BitInfo

BitInfo aims to provide effective increment priorities for fine-grained precision allocation without activation calibration. Although the exact reduction in weight reconstruction error delta E can also be computed offline, it only measures the Frobenius error in weight space and cannot directly capture the effects of input activations and subsequent nonlinear computations on output quality. BitInfo combines increment energy with sign self-information, incorporating statistical structure from the weights to construct a more effective weight-only ranking criterion. As shown in Table 5, BitInfo achieves an average score of 40.9, outperforming Energy-only (40.3), Magnitude (39.8), and Random (39.3).

Regarding the reviewer's concern that Table 5 evaluates only a subset of TruthfulQA: the combinatorial search space for ablations over allocation criteria (Energy, Information, Magnitude, and Random) is substantial. We therefore selected a 50% subset of TruthfulQA, as this benchmark is particularly representative of and sensitive to quantization-induced degradation in generative reasoning, allowing us to control computational costs while maintaining reliable conclusions. To provide more comprehensive validation, we will include ablation results on the full TruthfulQA set and additional tasks, including GSM8K, in the appendix of the final revision.

5. Truncation Bias and Its Treatment

We clarify that IncMoE does not independently requantize weights to different bitwidths at runtime. Instead, it directly partitions the 8-bit integers obtained from a single quantization procedure into ordered 2-bit chunks, enabling dynamic precision through prefix truncation and incremental accumulation.

For any b-bit prefix, the reconstruction error relative to the complete 8-bit representation satisfies:

W[8]−W[b]=s(Q mod 28−b)∈[0,s(28−b−1)].W^{[8]}-W^{[b]}=s(Q\bmod 2^{8-b})\in[0,s(2^{8-b}-1)].

This property is already established in Equation (7) of the paper. As subsequent bit slices are progressively restored, the truncation error decreases monotonically and becomes exactly zero at 8-bit precision.

Omitting Midpoint Correction is a deliberate design choice aligned with IncMoE's incremental execution mechanism. We incorporate the zero-point only into the base representation, allowing subsequent precision refinement to proceed through direct accumulation of unsigned increments from the original 2-bit chunks, without requantization or constructing separate weight representations for different bitwidths. Although Midpoint Correction can reduce the worst-case truncation error bound, it introduces bitwidth-dependent compensation terms, alters the original direct bit-level prefix reconstruction, and does not guarantee improvements in downstream model quality. Therefore, IncMoE preserves exact Bit-slice Prefix Reconstruction and selectively recovers important increments through Router-conditioned Allocation and BitInfo.

Reviewer_LFmiOpenReview ↗

Response to the authors

Thank you for the detailed rebuttal and additional experiments. The new results help clarify the differences from D²MoE and address my concerns about calibration robustness. However, I still have reservations regarding the overall novelty, the practical accuracy–memory trade-off, and the limited validation of BitInfo. I appreciate the authors' efforts but will maintain my original rating.

Thank you for your follow-up. We would like to emphasize IncMoE’s central contribution: a bit-wise quantized representation that makes precision progressively refinable, coupled with calibration-free, token-conditioned allocation of those refinements. This turns precision into a resource allocated along each token’s active path, allowing one stored representation to support adaptive quality–latency trade-offs without recalibration or requantization. We believe this representation–allocation framework offers meaningful methodological novelty and practical value, and respectfully ask you to reconsider the original rating in light of the clarifications and additional evidence.

Reviewer_XP1AOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes IncMoE that tackles the memory and bandwidth costs of MoE inference while seeking flexible runtime precision without multiple weight copies or calibration data. It stores expert weights as nested bit slices and dynamically allocates refinement increments using routing weights and a weight-only BitInfo score. A single resident representation supports several execution budgets without storing separate checkpoints. Experiments evaluate quality, domain sensitivity, storage, and single-sequence decoding latency across multiple MoE models.

Critical strengths

The combination of one stored representation and token-dependent precision control is interesting. Allocating increments within experts, across projections and tiles, gives the method more flexibility than a fixed expert-level bit assignment.

The method maintains competitive quality across several MoE architectures without calibration data. This makes it appealing when representative deployment data are difficult to obtain.

Critical weaknesses

Four-bit execution still requires the resident 8-bit representation plus indices, so its advantage depends on needing multiple operating points. The latency evidence is also limited to single-sequence decoding, where conflicting refinement requests within a batch do not arise.

The robustness argument relies heavily on baselines calibrated only on GSM8K. Section 4.3 appropriately acknowledges this limitation, but the result leaves open whether the advantage reflects avoiding a narrow calibration source or a more general benefit of the calibration-free allocation.

Questions for authors
  1. Under the same total resident-memory limit, where does IncMoE offer a better quality-latency trade-off than fixed-precision deployment?

  2. When a small batch routes several tokens to the same expert but requests different refinement depths, how much extra fetching or dispatch overhead results? This seems central to whether the single-sequence latency gains carry over to serving workloads.

  3. Does the unseen-domain advantage in Table 3 persist when the existing baselines use a modest mixed-domain calibration set instead of GSM8K alone?

1 comment

1. Quality–Latency Trade-off under the Same Resident Memory Footprint

IncMoE's advantage lies in its flexible trade-off between model quality and execution cost. For Mixtral, increasing the average execution precision from 4 to 6 bits reduces PPL from 5.68 to 5.36, approaching FP16's 5.34, while decode latency increases from 37.62 to 45.88 ms/token. Compared with fixed INT8 (49.66 ms/token), IncMoE supports different runtime precision levels using the same resident representation to accommodate varying quality and latency requirements across requests.

Therefore, we do not claim that IncMoE achieves a superior quality–latency trade-off under every fixed-precision setting. Its primary contribution is enabling fine-grained, token-dependent runtime precision allocation without storing multiple quantized weight copies or redeploying the model. This provides flexibility unavailable in fixed-precision deployments, particularly for serving scenarios with heterogeneous quality and latency requirements.

2. Execution Overhead of Different Refinement Depths within a Batch

For tokens routed to the same expert, IncMoE shares an identical offline increment-priority list within each projection. Therefore, different refinement requests naturally follow a nested prefix structure.

For small batches, we adopt a Longest-Prefix Promotion strategy, in which all tokens within the same expert/projection use the longest requested prefix. This avoids additional token grouping or grouped kernel dispatches and requires no extra weight copies, thereby preserving the efficiency of batched matrix operations.

On Mixtral-8×7B with a target average execution precision of 4 bits, we observe that Batch=4/8 increases the actual executed bitwidth by only approximately 6%–8%, corresponding to an average execution precision of 4.24–4.32 bits. These results indicate that the additional execution cost introduced by longest-prefix alignment remains limited under the tested small-batch workloads. We will include the corresponding experimental settings and results in the revised paper and clarify that these measurements do not constitute a general upper bound on the overhead for arbitrary batch sizes.

3. Cross-Domain Robustness under Mixed-Domain Calibration

We introduce two additional calibration settings, C4 and Mixed-domain, and evaluate AWQ, GPTQ, and SmoothQuant. Mixed-domain calibration covers general text, mathematics, code, and commonsense reasoning. All methods use the same calibration budget (128 × 2,048 tokens), with consistent quantization configurations and evaluation protocols, and no overlap between calibration and evaluation samples.

Calibration SourceBest Calibrated Baseline (SmoothQuant)IncMoE-4Relative KL Reduction
GSM8K-only1.5411.4048.9%
C41.5161.4047.4%
Mixed-domain1.4941.4046.0%

As shown above, more diverse calibration data improve the cross-domain performance of existing baselines, but the gains remain relatively modest. SmoothQuant achieves improvements of 1.6% and 3.0% under C4 and Mixed-domain calibration, respectively, compared with GSM8K-only calibration. In contrast, calibration-free IncMoE maintains an average KL divergence of 1.404, outperforming the best calibrated baseline by 7.4% and 6.0% under the two additional settings.

These results demonstrate that broader calibration-domain coverage can partially improve the cross-domain robustness of existing quantization methods, but does not eliminate the performance gap relative to IncMoE.

Comments on the paper

2 comments
Area_Chair_GPvQOpenReview ↗

Please check author responses

Dear reviewers,

We’re about halfway through the discussion period. This is a friendly reminder to check other's reviews and the author responses. If the authors have answered questions or responded to critiques in reviews, please leave a comment for the authors!

Best, AC

Dear Area Chair,

Thank you for your time and effort in overseeing the review process.

We have provided detailed rebuttal responses addressing the reviewers' concerns, including additional experimental evidence on practical inference efficiency, GPU memory usage, calibration overhead, and cross-model applicability.

As the discussion period is approaching its end, we noticed all reviewers have not yet responded to our clarifications and new results.

We would greatly appreciate it if you could kindly encourage the reviewers to revisit our responses and provide any further feedback before the discussion deadline. We would be happy to address any remaining questions or concerns.

Thank you very much for your consideration and assistance.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.