acceptodds
Under review as a conference paper at ICLR 2027

BitsMoE: Cost-Aware Bit Allocation in Spectral Space for MoE LLM Quantization

Abstract

Mixture-of-Experts (MoE) large language models incur substantial memory costs due to their large expert parameter counts. Mixed-precision quantization reduces these costs by allocating different bit-widths to experts or linear blocks according to their importance. However, assigning a single precision within each expert or linear block overlooks its internal structural heterogeneity. This limitation motivates two key questions: (1) how to define a fine-grained unit for quantization within a linear transformation; and (2) how to characterize the quantization cost of each unit under actual activation patterns and different bit-widths. To address these two questions, we propose BitsMoE, a cost-aware mixed-precision quantization framework built on two complementary techniques: (1) Shared-basis Spectral Decomposition (SSD) separates expert weights into a shared basis and expert-specific spectral components, defining structural quantization units while exploiting cross-expert redundancy. (2) Factorized Quantization Cost Modeling (FQCM) estimates component-wise costs from output reconstruction loss by combining intrinsic spectral importance, activation-dependent importance, and bit-width-dependent distortion. Using these component-wise costs, we formulate bit allocation as an integer linear program (ILP) that minimizes total modeled quantization cost under a fixed memory budget. On Qwen3-30B-A3B at 2-bit, BitsMoE achieves 64.29% average accuracy over seven downstream tasks, outperforming the evaluated MoE-specific methods, including those using ILP-based bit allocation, and exceeding GEMQ by 2.80 percentage points. Under the same setting, it achieves a end-to-end offline quantization speedup over GEMQ. It also achieves up to the decode throughput of GPTQ.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Post-Training Quantization, Mixed-Precision Quantization, Spectral Decomposition, Efficient LLM Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_P3p6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

BitsMoE reduces the memory cost of MoE large language models by assigning different bit-widths to individual spectral components within expert weights. It introduces Shared-basis Spectral Decomposition (SSD) to separate shared structure across experts from expert-specific components, and Factorized Quantization Cost Modeling (FQCM) to estimate their quantization costs and guide bit allocation under a fixed memory budget. On Qwen3-30B-A3B at 2-bit, BitsMoE achieves 64.29% average accuracy over seven downstream tasks, exceeding GEMQ by 2.80 percentage points. It also achieves a 16.47× offline quantization speedup over GEMQ and up to 6.46× the decode throughput of GPTQ.

Critical strengths

1 BitsMoE defines fine-grained quantization units within MoE experts through Shared-basis Spectral Decomposition. It separates shared structure across experts from expert-specific spectral components and assigns heterogeneous bit-widths under a fixed bit budget.

2 FQCM combines intrinsic spectral importance, activation-dependent importance, and bit-width-dependent distortion to guide bit allocation. The ablation results show that incorporating activation and routing information substantially improves accuracy over allocation based only on spectral energy.

3 The experiments report both accuracy and efficiency gains. At 2-bit on Qwen3-30B-A3B, BitsMoE achieves 64.29% average accuracy over seven downstream tasks, exceeding GEMQ by 2.80 percentage points. It also achieves a 16.47× end-to-end offline quantization speedup over GEMQ and up to 6.46× the decode throughput of GPTQ in the evaluated settings.

Critical weaknesses

1、Table 4 shows a modest improvement from independent SVD to SSD under uniform allocation, from 24.86% to 25.27%. Could the authors also evaluate independent SVD with FQCM allocation under the same storage budget? Additional comparisons and implementation details would help clarify the contribution of SSD.

2、Could the authors provide more details on the ILP solution procedure, including the solver tolerances, time limits, and CPU and thread configuration?

3、Equation (8) combines intrinsic spectral importance, activation-dependent importance, and bit-width-dependent distortion through multiplication. How is the relative influence of these factors determined, and is any additional weighting or normalization needed?

4、In Table 3, the gains over GEMQ are concentrated in MQ and GS, while some other tasks show lower accuracy. Could the authors explain why the method benefits these mathematical tasks more than the others?

Questions for authors

The interpretation of Table 4 is my main concern.

3 comments
AuthorsOpenReview ↗

Response to Reviewer P3p6 Part 1

We sincerely appreciate your constructive feedback. We provide point-by-point responses below.

Weakness 1: Independent SVD with FQCM and the interpretation of Table 4.

Thank you very much for your professional feedback.

(1) Decomposition comparison

Non-Basis with FQCM achieves 49.98% average accuracy at a target bit-width of 2-bit on Qwen3-30B-A3B. Non-Basis applies SVD independently to each expert, without SSD. Average accuracy is computed over the seven downstream tasks.

BasisBitsAllocatorAverage
Non-Basis−-Uniform24.86
Non-Basis−-FQCM49.98
SSD4-bitFQCM62.78
SSD8-bitFQCM64.29

(2) Task-level results

BasisBitsHSAEMQMMOBWGGSAvg.
Non-Basis−-51.2672.1044.1540.8335.2063.1443.2149.98

(3) Effectiveness of FQCM without SSD

In the Non-Basis setting, FQCM improves average accuracy from 24.86% under uniform allocation to 49.98%, demonstrating its effectiveness without SSD.

We have added the Non-Basis ablation to the revised manuscript in Section 4.3 (Tab. 4) and Appendix E.2 (Tab. 12).

Weakness 2: ILP solution procedure, tolerances, and hardware configuration.

Thank you very much for your professional feedback.

We use Gurobi 12.0.0 to solve one global ILP per projection type, allocating bits jointly across all MoE layers. The solver settings are listed below.

ParameterSetting
MIPGap0.001
FeasibilityTol1e-6
IntFeasTol1e-5
OptimalityTol1e-6
TimeLimitNo explicit limit

We use 48 solver threads for all three models. A representative host has two AMD EPYC 7763 CPUs (128 physical cores and 256 hardware threads) and NVIDIA A100 80GB PCIe GPUs.

We have added the solver settings and host configuration to the revised manuscript in Appendix A (Tab. 7).

We sincerely appreciate your professional feedback.

AuthorsOpenReview ↗

Response to Reviewer P3p6 Part 2

Weakness 3: Relative influence and normalization of the FQCM factors.

Thank you very much for your professional feedback.

The multiplicative form follows from the reconstruction-loss derivation in App. C under the stated approximations. The three factors are multiplied directly in Eq.(8) & Eq. (40), without additional weighting or normalization. The resulting component-wise quantization costs are summed to obtain the global FQCM loss in Eq. (9), with no further normalization.

Weakness 4: Larger gains on MathQA and GSM8K.

Thank you very much for your professional feedback.

GEMQ's expert-level allocation preserves some experts at higher precision but can perturb important directions in lower-precision experts. BitsMoE can retain higher precision for important spectral directions within each expert, which may help preserve the numerical distinctions needed for mathematical tasks. Although MathQA uses candidate-likelihood scoring, selecting the correct numerical answer still depends on the quantitative relationships in the question. GSM8K additionally requires generating a solution, where errors in key numerical values or intermediate operations can affect subsequent generation and the final answer.

We sincerely appreciate your professional feedback.

Thank you for your response. I realize I misunderstood this point earlier: Non-Basis refers to the SVD results. Most of my concerns have been addressed, and I will keep my positive rating.

Reviewer_8hU8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents BitsMoE, a fine-grained mixed-precision post-training quantization framework for Mixture-of-Experts (MoE) large language models. The authors argue that existing expert-level or linear-block-level bit allocation ignores the quantization sensitivity of different spectral directions within weight matrices. To address this issue, the paper proposes Shared-basis Spectral Decomposition (SSD), which decomposes weights into shared spectral bases and expert-specific spectral components via cross-expert joint SVD, with the latter treated as independent quantization units. Afterwards, Factorized Quantization Cost Modeling (FQCM) estimates the quantization cost of different spectral directions by combining spectral energy, routing-weighted activation importance and bit-width-dependent distortion, and performs bit-width allocation under a fixed memory budget using integer linear programming (ILP). The authors further implement bit-plane packing and GPU kernels to enable efficient inference under the factorized representation. Experiments are conducted on three MoE models. On Qwen3-30B-A3B, BitsMoE achieves an average accuracy of 64.29% across seven tasks at a 2-bit target precision, outperforming GEMQ by 2.80 percentage points, and delivers a 16.47× speedup for offline quantization compared with GEMQ.

Critical strengths

Strength 1: The paper puts forward a clear problem definition targeting the internal structural differences of experts in MoE quantization. SSD constructs fine-grained quantization units via shared spectral bases across experts, while FQCM integrates the inherent importance of spectral directions, activation patterns and bitwidth-related distortion to provide an interpretable cost model for mixed-precision allocation. Compared with assigning uniform bitwidth directly by expert or linear block, this design can leverage the quantization sensitivity of different spectral directions more elaborately.

Strength 2: The paper demonstrates promising experimental performance across multiple MoE architectures under ultra-low bitwidth settings, achieving notable improvements especially on mathematical reasoning tasks. Under the 2-bit setting of Qwen3-30B-A3B, BitsMoE attains a GSM8K accuracy of 71.04%, a 24.94 percentage point increase over GEMQ’s 46.10%; on Qwen3-Next-80B-A3B, its average accuracy across seven tasks reaches 70.70%, outperforming HQQ’s 67.65%. The paper also presents performance analysis for 1.4–2.0-bit, ablation studies of different FQCM components, and correlation analysis between FQCM loss and perplexity under 170 bitwidth allocation schemes, strengthening the credibility of experimental results.

Strength 3: Beyond quantization accuracy, the paper provides practical GPU inference implementation and comprehensive efficiency evaluation. The authors realize direct computation on quantized spectral directions through shared spectral basis computation reuse, bitwidth-based grouping and bit-plane packing. On Qwen3-30B-A3B, BitsMoE reports a 4.70× offline quantization speedup compared with GPTQ, and achieves an improvement in batch-1 streaming decode throughput from 3.08 tok/s to 7.19 tok/s on a single RTX A6000. These results demonstrate that the method has certain potential for real-world deployment.

Critical weaknesses

Weakness 1: The novelty of the core method and its distinction from prior work remain insufficiently clear. MoE-SVD [1] leverages weight redundancy by sharing SVD bases across experts, while Sub-MoE [5] further adopts Experts Union Decomposition to perform joint SVD on concatenated expert weights for extracting shared orthogonal bases and expert-specific factors. Its core decomposition idea is highly similar to the SSD in this paper. Although Sub-MoE targets expert merging rather than quantization, this paper has not adequately clarified the independent innovation of SSD in the decomposition mechanism nor discussed its direct relationship with Sub-MoE. In addition, Delta-CoMe [2] has assigned different bit widths according to the importance of singular vectors, and MxMoE [3] and GEMQ [4] have investigated mixed-precision allocation based on sensitivity and global optimization. Therefore, the main contribution of BitsMoE appears to be the combination of existing shared spectral decomposition, spectral direction importance estimation, and ILP bit-width allocation. While such combination has certain practical value, the paper lacks sufficient novelty.

Weakness 2: The theoretical derivation of FQCM relies on strong approximation assumptions, and its accuracy has not been fully validated. Eq. (6)–(7) approximates the full output reconstruction loss as the sum of costs in each direction by ignoring quantization-error cross terms between different spectral components. However, under deterministic low-bit quantization, quantization errors across different directions are not necessarily independent. This assumption may become unreasonable especially when some directions use 1-bit quantization or direct 0-bit eviction. For down projection, Appendix C.2 also assumes that the quantized shared basis is approximately orthogonal and uses the activation importance of original spectral directions instead of the sensitivity of actual quantization error directions.

Weakness 3: There remains a gap between the routing-aware cost and the actual MoE output error. FQCM employs the routing-weighted Hessian (Hg,e=XeGeXe⊤H_{g,e}=X_eG_eX_e^\top), where routing weights enter the local reconstruction objective with a power of one. Nevertheless, the real MoE output is jointly formed by weighted results of multiple experts, and its squared reconstruction error typically contains squared terms of routing weights as well as cross error terms among different experts. The current objective is closer to a routing-weighted per-expert reconstruction surrogate rather than the full MoE output error. Furthermore, the authors normalize activation importance to enable cross-layer allocation, which may alter the importance differences between different experts arising from absolute token usage frequency. The paper has not fully explained how these modeling choices affect the final allocation quality.

Weakness 4: The modeling of bit-width-dependent distortion is overly simplified. For 2-bit, 3-bit and 4-bit, FQCM uses the same empirical distortion coefficient (κb\kappa_b) for all spectral components. Hence, across these bit widths, the component-wise cost is actually dominated by (αe,k2βe,k\alpha_{e,k}^2\beta_{e,k}) and does not explicitly reflect the inherent quantization difficulty of different spectral directions. Moreover, these coefficients are estimated via single-scale Gaussian random unit vectors, while the actual quantization uses a group size of 128; the error statistics of the two may not be fully consistent.

Weakness 5: The baseline selection for inference efficiency may affect the generalizability of speedup conclusions. The authors report a maximum 6.46× decode speedup on Qwen3-30B-A3B, but explicitly note that GPTQ uses the Triton backend because Marlin and ExLlamaV2 in their adopted implementation do not support the corresponding 2-bit checkpoints. In contrast, BitsMoE uses specially designed bit-plane kernels and performs shared spectral basis computation via Marlin W8A16. Therefore, the current speed comparison incorporates differences in both quantization representation and kernel implementation, and cannot sufficiently prove that BitsMoE retains the same magnitude of advantage under a unified optimization level. Although the paper has reported results for batch-1, various batch sizes and TTFT, it lacks independent overhead analysis for shared-basis computation, bit-plane unpacking, indexed execution and component magnitude processing, and does not separately report prefill throughput.

[1] Li et al. MoE-SVD: Structured Mixture-of-Experts LLMs Compression via Singular Value Decomposition. ICML 2025. [2] Ping et al. Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models. NeurIPS 2024. [3] Duanmu et al. MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design. ICML 2025. [4] Deng et al. GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs. ICML 2026. [5] Li et al. Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging. AAAI 2026.

Questions for authors

Q1: MoE-SVD [1] and Sub-MoE [5] have investigated cross-expert joint SVD and shared spectral basis decomposition. Among them, the Experts Union Decomposition in Sub-MoE bears a high resemblance to the mathematical formulation of SSD in this paper. Delta-CoMe [2] has adopted spectral-direction-level mixed-precision allocation, while MxMoE [3] and GEMQ [4] have studied bit-width allocation based on importance estimation and global optimization. Could the authors further clarify the independent technical innovations of SSD and FQCM compared with these existing methods?

Q2: In Equations (6)–(7), FQCM ignores quantization-error cross terms between different spectral components. Meanwhile, Appendix C.2 introduces the approximate orthogonality of shared bases and error-direction approximation for down projection. In addition, the current routing-weighted reconstruction loss uses routing weights to the first power, whereas the squared error of the full MoE output also involves weighted error interactions across different experts. Could the authors further elaborate on the theoretical justifications for these approximations, and verify the consistency between the FQCM estimated cost and the actual output reconstruction error on gate, up and down projections respectively? In particular, are these assumptions still valid under 1-bit quantization and 0-bit eviction?

Q3: For 2-bit, 3-bit and 4-bit settings, FQCM uses Gaussian distortion coefficients shared across the model, without explicitly accounting for the actual quantization difficulty of different spectral directions. At the same time, single-scale Gaussian random vectors are adopted for coefficient estimation, while group size 128 is used in practical quantization. Could the authors compare shared coefficients, projection-specific coefficients and actual component-wise quantization MSE on the three main models, and report the corresponding accuracy–quantization-time trade-off?

Q4: The 1.4–3.0-bit configuration reported in Appendix C.4 appears to correspond to the static equivalent storage budget of all routed expert weights, rather than the active BPW of the real token routing path. Considering that the shared spectral basis is stored in 8-bit format and only a small number of experts are activated per token, after amortizing its overhead over Top-K active experts, the active-path equivalent BPW may be substantially higher than the reported static bit width. Could the authors separately report static storage BPW, active-path equivalent BPW and actual memory traffic, and explicitly include FP16 magnitudes, quantization scales, component indices and packing metadata?

Q5: SSD requires performing joint SVD on the concatenated expert weight matrices within each MoE layer. For Qwen3-Next-80B-A3B, each layer contains 512 routed experts, so the joint decomposition may involve large matrices. Appendix F.2 reports offline quantization on a single NVIDIA A100 PCIe 80GB GPU, and the SVD stage in Table 15 only takes 4.17 minutes. Could the authors further describe the specific implementation of joint SVD, including the decomposition algorithm used, CPU/GPU execution location, layer-wise processing or blocking strategy, and peak CPU/GPU memory requirements? If CPU offloading is involved, could the CPU model be provided and clarify whether data transfer time is included in the reported SVD runtime, to support the reproducibility of this efficiency result?

Q6: The authors report a maximum 6.46× decode speedup for Qwen3-30B-A3B. However, the GPTQ baseline uses the Triton backend, whereas BitsMoE employs specially optimized bit-plane kernels and Marlin W8A16 shared-basis execution. Therefore, the current speedup results are affected by both differences in quantization representation and kernel implementation. Could the authors supplement a fairer optimized-kernel comparison, and separately report prefill throughput, decode throughput and end-to-end latency under batch-1 and larger batch sizes? Furthermore, could the independent overheads of shared-basis computation, bit-plane unpacking, component indexing and low-bit GEMV be further quantified, to identify which technical components contribute most to the practical speedup?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 8hU8 Part 1

We sincerely appreciate your constructive feedback. We provide point-by-point responses below.

Weakness 1 & Question 1: Technical contributions of SSD and FQCM relative to prior work.

Thank you very much for your professional feedback.

We agree that MoE-SVD and Sub-MoE study shared spectral bases across experts, Delta-CoMe applies spectral mixed-precision quantization, and MxMoE and GEMQ study importance-based and global bit allocation. Our contributions concern how SSD defines quantization units and how FQCM models their costs.

(1) SSD defines expert-specific quantization units for FQCM. We do not claim to be the first to use shared-basis decomposition. SSD exploits cross-expert redundancy by extracting a shared spectral basis while retaining component magnitudes and unit-norm expert-specific spectral directions. These directions serve as fine-grained quantization units, allowing FQCM to distinguish their quantization distortion from their importance to expert outputs and use these differences to guide bit allocation.

(2) FQCM models component-wise quantization costs for adaptive bit allocation. Delta-CoMe allocates precision according to singular-value ordering, without explicitly modeling each component's reconstruction cost at each candidate bit-width. App. C derives FQCM in Eq. (40) from the expected routing-weighted output reconstruction loss under the stated approximations, combining intrinsic spectral importance, activation-dependent importance, and bit-width-dependent distortion. This provides a cost for quantizing each expert-specific spectral direction to each candidate bit-width.

MxMoE and GEMQ estimate candidate costs from output errors evaluated on calibration data. Our symmetric-quantizer analysis yields the computationally efficient distortion surrogate in Eq. (71). Combined with precomputed activation and routing statistics, it enables direct component-wise cost construction without repeated candidate output-error evaluation, balancing estimation fidelity against offline quantization time.

Tab. 18 reports offline quantization speedups over MxMoE and GEMQ, reflecting savings in both cost construction and quantization. Figs. 5 and 10 support the surrogate through the ranking of fixed-budget allocations by WikiText-2 PPL and the layer-wise ranking of output reconstruction error, respectively; Tabs. 1 and 3 demonstrate downstream task accuracy gains.

We sincerely appreciate your professional feedback.

AuthorsOpenReview ↗

Response to Reviewer 8hU8 Part 2

Weaknesses 2 & 3 and Question 2: FQCM approximations, routing weights, and reconstruction-error validation.

Thank you very much for your professional feedback.

(1) Tractable component-wise allocation

In App. C.2, we omit cross-component error terms so that precomputed component-wise quantization costs form a linear ILP objective. Retaining these interactions would couple allocation decisions; the additive surrogate trades modeling fidelity for tractable global allocation.

For down projection, we treat the 8-bit shared basis as approximately orthonormal (Eq. (35)) and use each original expert-specific spectral direction's activation-dependent importance as a proxy for error-direction sensitivity. This favors preserving spectral directions that are important under calibration activations and avoids recomputing sensitivity for every candidate bit-width.

These choices balance ILP tractability and quantization performance, and the results in Figs. 5 and 10 and Tabs. 1 and 2 provide empirical support for the effectiveness of our approximations and surrogate.

(2) Routing weights and normalization

Inspired by MoEQuant, we use routing weights to the first power to reflect each expert's importance for each token. This is a heuristic reconstruction objective rather than the full MoE squared error, which contains squared routing weights and cross-expert error terms. Incorporating routing weights improves average accuracy from 60.45% to 64.29% on Qwen3-30B-A3B at 2-bit (Tab. 4).

Normalization uses unweighted activation statistics to make scales comparable across layers while retaining routing-weight information. The resulting activation-dependent importance emphasizes token–expert relevance rather than absolute usage frequency.

(3) Reconstruction error and allocation ranking

  • Layer-wise reconstruction (App. D.4, Fig. 10). Across 48 layers of Qwen3-30B-A3B at 2-bit, FQCM costs have Spearman correlations of 0.815, 0.853, and 0.895 with layer averages of normalized, routing-weighted per-expert output reconstruction error for gate, up, and down projections, respectively.
  • Global allocation (Fig. 5). Across 170 allocations under the same 2-bit budget, total FQCM loss has a Spearman correlation of 0.8846 with WikiText-2 PPL; the ILP solution has the lowest measured PPL among these allocations.

The reconstruction experiment includes 0-bit eviction and 1-bit quantization and supports layer-wise ranking under these settings, without separately validating each approximation.

We have added the layer-wise reconstruction-error analysis to the revised manuscript in Appendix D.4 (Fig. 10).

Weakness 4 & Question 3: Shared distortion coefficients and the accuracy–quantization-time trade-off.

Thank you very much for your professional feedback.

(1) Shared versus component-wise distortion

In our formulation, κb\kappa_b depends only on bit-width and does not vary across tensors or projection types. It is estimated by averaging the squared errors from MSE-optimal quantization of Gaussian random unit vectors, independently of model weights. We compare the estimated coefficients (Estimated κb\kappa_b) with actual component-wise quantization MSE (Actual MSE) measured with group size 128 at a target bit-width of 2-bit. Accuracy is the average over the seven downstream tasks (%).

Average accuracy (%)

ModelEstimated κb\kappa_bActual MSE
DeepSeek-V2-Lite47.4446.73
Qwen3-30B-A3B64.2964.15
Qwen3-Next-80B-A3B70.7070.60

Distortion construction time (sec)

ModelEstimated κb\kappa_bActual MSE
DeepSeek-V2-Lite0.9463.52
Qwen3-30B-A3B2.60226.74
Qwen3-Next-80B-A3B12.30909.16

(2) Measurement settings and trade-off

Shared κb\kappa_b provides a useful approximation of quantization error for allocation: downstream task accuracy remains nearly identical to that obtained using actual component-wise quantization MSE, while reducing offline quantization time. The following table compares the end-to-end offline quantization time of the two strategies.

End-to-end offline quantization time (min)

ModelEstimated κb\kappa_bActual MSE
DeepSeek-V2-Lite21.3222.36
Qwen3-30B-A3B63.4167.15
Qwen3-Next-80B-A3B267.04281.99

We have added the comparison between estimated distortion coefficients and actual component-wise quantization MSE to the revised manuscript in Appendix D.1 (Tab. 9).

We sincerely appreciate your professional feedback.

AuthorsOpenReview ↗

Response to Reviewer 8hU8 Part 3

Question 4: Static storage, active-path BPW, and metadata.

Thank you very much for your professional feedback.

The target bit-width specifies a static routed-expert weight budget, including the shared basis. For the 2-bit checkpoints, the storage BPW below includes the 8-bit shared bases, FP16 magnitudes, quantization scales, component indices, and packing metadata.

ModelMoE static BPWMoE active-path BPW
DeepSeek-V2-Lite2.204.03
Qwen3-30B-A3B2.244.81
Qwen3-Next-80B-A3B2.305.52

Static and active-path BPW are normalized by the original weight counts of all and selected routed experts, respectively. Both include shared bases; active-path BPW is averaged over 16 decoding steps with a batch size of 1.

Question 5: Joint SVD implementation, runtime, and peak memory.

Thank you very much for your professional feedback.

(1) Decomposition and execution strategy

SSD uses FP32 SVD on the GPU, implemented with torch.linalg.svd(..., full_matrices=False). It processes one layer and one projection at a time, without additional blocking. Qwen3-Next-80B-A3B has 48 MoE layers and 512 routed experts per layer; concatenating the expert weights gives matrices of shape 262,144×2,048262{,}144\times2{,}048 for gate and up projections and 2,048×262,1442{,}048\times262{,}144 for down projection.

At the start of calibration, we load the entire model into CPU memory using Transformers (e.g., AutoModelForCausalLM.from_pretrained(model_path, device_map="cpu")). We then move one layer at a time to the GPU for expert-weight concatenation and SVD, keeping the active layer's parameters on the GPU during its processing. The host has two AMD EPYC 7763 CPUs. The end-to-end offline quantization time reported in Tab. 17 includes model loading, data transfers, and initialization.

(2) Runtime and memory

For Qwen3-Next-80B-A3B, the 4.17-minute stage comprises 225.72 seconds for joint SVD and factor construction plus 24.49 seconds for factor reconstruction during packing. Transfers are included in total offline quantization time and excluded from this stage. These timings were measured on a single NVIDIA A100 80GB PCIe GPU.

To account for the CPU memory occupied by offloaded model weights, we profile Qwen3-30B-A3B and Qwen3-Next-80B-A3B. For each model, we keep the full model resident in CPU memory and perform SVD on the first MoE layer using a single NVIDIA A100 80GB PCIe GPU. The CPU process peak includes model loading and full-model CPU residency. The measured memory usage is reported below.

Memory metricQwen3-30B-A3B (GiB)Qwen3-Next-80B-A3B (GiB)
CPU RSS after full-model loading58.111150.354
Peak CPU RSS (including model loading and offloaded model weights)60.824153.491
Sampled peak CPU RSS during SVD58.206150.405
Peak GPU allocated memory5.80515.372
Peak GPU reserved memory7.27518.904

These memory profiles cover full-model loading and SVD of the first MoE layer; they exclude calibration, ILP, quantization, and packing.

We sincerely appreciate your professional feedback.

Reviewer_tUrkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BitsMoE, a new post-training quantization framework for MoE-based LLM combining shared spectral bases, component-level bit allocation, and specialized inference kernels. Simulation results show the advantage of proposed methods compared to conventional PTQ methods.

Critical strengths
  • The paper is well presented, with a clear structure and easy-to-follow explanations.
  • The illustrations effectively explain the proposed method and its main components.
  • The paper provides extensive experiments, including comparisons, ablations, and efficiency evaluations.
Critical weaknesses
  • The theoretical contribution is limited. The paper mainly develops a cost model and an optimization formulation, with few theoretical results characterizing the proposed method’s properties or performance. Stronger analysis would help establish when and why the method is effective.
  • The novelty relative to prior spectral allocation methods needs clearer discussion. PrinMix (“Principled SVD-based Delta Compression via Quantization Error Minimization”) already combines SVD-based quantization-error model with ILP-based bit allocation. The paper should distinguish its cost model and MoE-specific design from this closely related approach.
  • The ablations do not isolate the benefit of the shared spectral basis under a strong allocator. Independent SVD is tested only with uniform allocation. Comparing independent SVD and the proposed decomposition using the same cost-aware allocation would clarify which component drives the gains.
  • The decoding-efficiency evaluation is limited to GPTQ using the Triton backend. Comparisons with MoE-specific PTQ methods, such as MiLo and MxMoE, are missing, making it difficult to assess the practical inference advantage over existing MoE quantization methods.
Questions for authors

See weakness.

6 comments
AuthorsOpenReview ↗

Response to Reviewer tUrk Part 1

We sincerely appreciate your constructive feedback. We provide point-by-point responses below.

Weakness 1: Theoretical contribution and justification of FQCM.

Thank you very much for your professional feedback.

Our contribution is the reconstruction-based derivation and efficient evaluation of the component-wise quantization cost in Eq. (8) for SSD's expert-specific spectral directions. Although the individual factors have precedents, App. C develops their combination from the expected routing-weighted output reconstruction loss in Eq. (27) under the stated approximations. Intrinsic spectral importance is given by squared component magnitudes, and activation-dependent importance is computed from calibration activation and routing statistics. For bit-width-dependent distortion, we analyze the symmetric quantizer to obtain the computationally efficient distortion model in Eq. (71). This provides a mathematical basis for the allocation criterion and balances estimation fidelity against offline quantization time.

Our formulation is not tied to a particular MoE architecture, expert count, or expert projection shape, making it directly applicable to mainstream MoE LLMs with standard feed-forward experts. Its effectiveness as an allocation criterion is supported empirically: on Qwen3-30B-A3B at 2-bit, Fig. 5 reports a Spearman correlation of 0.8846 between FQCM loss and WikiText-2 PPL across 170 allocations. Fig. 10 reports Spearman correlations of 0.815, 0.853, and 0.895 for gate, up, and down projections across 48 layers, comparing FQCM loss with layer averages of normalized routing-weighted per-expert output reconstruction error. Tab. 4 and Tabs. 1 and 3 further support the component contributions and ultra-low-bit task accuracy, respectively.

We have added the layer-wise reconstruction-error analysis to the revised manuscript in Appendix D.4 (Fig. 10).

Weakness 2: Novelty relative to PrinMix and other spectral allocation methods.

Thank you very much for your professional feedback.

We agree that PrinMix and BitsMoE share SVD-based error modeling and ILP-based bit allocation. PrinMix compresses fine-tuning deltas by combining singular-value scaling with activation-weighted quantization errors to select bit-widths. The differences lie in the MoE representation and the construction of candidate quantization costs.

(1) Shared structure with expert-specific allocation. SSD jointly decomposes MoE expert weights to extract a shared spectral basis while retaining expert-specific spectral directions and component magnitudes. Storing the shared spectral basis once leaves more budget for expert-specific spectral directions in the ultra-low-bit regime. These directions serve as fine-grained quantization units. At the same 2-bit budget on Qwen3-30B-A3B, SSD with FQCM and an 8-bit basis achieves 64.29% average accuracy, compared with 49.98% for Non-Basis with FQCM (Tab. 4).

(2) Model-based distortion estimation. PrinMix simulates candidate quantization on calibration data to construct its costs. FQCM separates activation-dependent importance from bit-width-dependent distortion and estimates the latter through Eq. (71), with a detailed theoretical derivation based on the symmetric quantizer in App. C. This avoids repeated candidate simulation for cost construction as the number of experts and candidate bit-widths grows; Qwen3-Next-80B-A3B, for example, has 512 routed experts per layer. Tab. 18 reports 49.00×49.00\times and 16.47×16.47\times end-to-end offline quantization speedups over MxMoE and GEMQ on Qwen3-30B-A3B at 2-bit, reflecting savings in both cost construction and quantization.

We sincerely appreciate your professional feedback.

AuthorsOpenReview ↗

Response to Reviewer tUrk Part 2

Weakness 3: Independent SVD with cost-aware allocation.

Thank you very much for your professional feedback.

(1) Decomposition comparison

Non-Basis with FQCM achieves 49.98% average accuracy at a target bit-width of 2-bit on Qwen3-30B-A3B. Non-Basis applies SVD independently to each expert, without SSD. Average accuracy is computed over the seven downstream tasks.

BasisBitsAlloc.Avg.
Non-Basis−-Uniform24.86
Non-Basis−-FQCM49.98
SSD4-bitFQCM62.78
SSD8-bitFQCM64.29

(2) Task-level results

BasisBitsHSAEMQMMOBWGGSAvg.
Non-Basis−-51.2672.1044.1540.8335.2063.1443.2149.98

(3) Effectiveness of FQCM without SSD

In the Non-Basis setting, FQCM improves average accuracy from 24.86% under uniform allocation to 49.98%, demonstrating its effectiveness without SSD.

We have added the Non-Basis ablation to the revised manuscript in Section 4.3 (Tab. 4) and Appendix E.2 (Tab. 12).

We sincerely appreciate your professional feedback.

AuthorsOpenReview ↗

Response to Reviewer tUrk Part 3

Weakness 4: Comparison with MoE-specific quantization methods.

Thank you very much for your professional feedback.

(1) Additional inference comparison

MiLo's reported INT3 backend does not support our 2-bit setting. We therefore compare the fused MxMoE W2A16 kernel with BitsMoE at the same target bit-width of 2-bit on an RTX A6000, using group size 128 and identical inputs and generation lengths for both methods.

BitsMoE achieves 1.131.13–1.25×1.25\times decode speedup at batch sizes 1–32 on Qwen3-30B-A3B.

Qwen3-30B-A3B

BSMxMoE (tok/s) ↑\uparrowBitsMoE (tok/s) ↑\uparrowSpeedup
19.4511.301.20×1.20\times
216.0919.031.18×1.18\times
431.0237.481.21×1.21\times
861.2675.381.23×1.23\times
16114.96143.661.25×1.25\times
32227.78257.181.13×1.13\times

We have added the MxMoE decode-throughput comparison to the revised manuscript in Section 4.4 (Fig. 7).

We sincerely appreciate your professional feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.