acceptodds
Under review as a conference paper at ICLR 2027

BitLift: Unified Mixed-Precision Quantization and Caching for Efficient MoE Inference

Abstract

Mixture-of-Experts (MoE) models reduce computation through sparse activation, but moving expert weights can dominate decoding when the model exceeds device memory. Dynamic mixed precision and DRAM-caching can reduce this cost, yet their effectiveness depends on how weight representation, precision allocation, and cache residency interact. We introduce , a training-free framework that co-designs these components. A router-rank policy dynamically allocates a fixed nominal precision budget across selected experts. Each expert is encoded as a 2-bit base and two 1-bit refinements, jointly optimized for the target policy through weighted scale fitting and error compensation. A two-pool cache separates base and refinement planes and applies cache-resident precision refinement without fetching additional weights for an upgrade. Across three MoE model families, rank-assigned execution approaches independently optimized 3-bit quality at nominal budgets of 2.333–2.375 bits per routing slot, and outperforms the evaluated 2.5-bit baselines. On Qwen3-30B-A3B, cache-resident refinement matches uniform 4-bit WikiText-2 perplexity with a cache holding 50% of the 4-bit routed-expert footprint. At smaller matched capacities of 6.25–25%, it improves on uniform 3-bit perplexity. In this flash-limited regime, our analytical hardware model estimates a throughput gain of – over uniform 3-bit and – over uniform 4-bit inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-experts, large language models, post-training quantization, dynamic mixed precision, progressive quantization, expert caching, memory-constrained inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ktTFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BitLift, a training-free framework for mixed-precision MoE inference under limited memory. It combines router-rank precision allocation with a shared 2-bit base and two 1-bit refinement planes, optimized across precision levels. A two-pool cache separates base and refinement data, and resident refinement planes can increase execution precision without additional fetches for the upgrade. Experiments evaluate predictive quality and weight traffic, while an analytical model estimates throughput.

Critical strengths
  1. BitLift improves mean downstream accuracy by 0.93%–1.30% over the D²MoE representation under the same precision policies across three MoE models.

  2. Using resident refinement planes to improve precision is a practical idea. The evaluation shows improved predictive quality with nearly unchanged weight-transfer traffic.

Critical weaknesses
  1. The contribution over D²MoE appears limited. D²MoE already supports the same 2+1+1-bit nested representation, dynamic expert precision, and bit-aware memory management. BitLift adds policy-weighted quantization and cache-resident refinement, but the reported improvements do not clearly establish a substantial advance over this existing framework. Although the paper includes scale-fitting and cache ablations, the representation comparison does not fully separate the benefit of policy-aware optimization from differences in error compensation.

  2. The evaluation does not establish an advantage over the complete D²MoE system. Its representation is evaluated under BitLift’s precision policy, which is useful for isolating representation quality but excludes D²MoE’s learned controller and scheduling mechanisms. Moreover, the reported throughput gains are analytical estimates based on assumed bandwidth and compute capacity, rather than measured end-to-end performance. The practical advantage of BitLift’s combined quantization and caching design therefore remains unclear.

Questions for authors
  1. How much of the improvement over D²MoE comes from policy-aware optimization when error compensation is otherwise matched?

  2. How does BitLift compare with the complete D²MoE system in accuracy and measured end-to-end performance under the same hardware and memory budget?

5 comments

Response for Reviewer ktTF (1/2)

We thank the reviewer for their time and feedback. Below, we address the concerns raised and provide supporting clarifications and additional experimental evidence.

W1. The contribution over D²MoE appears limited. D²MoE already supports the same 2+1+1-bit nested representation, dynamic expert precision, and bit-aware memory management. BitLift adds policy-weighted quantization and cache-resident refinement, but the reported improvements do not clearly establish a substantial advance over this existing framework.

We disagree and believe there are fundamental differences between BitLift and D²MoE. The shared 2+1+1 encoding only describes the storage layout; BitLift’s contribution lies in the joint design of representation fitting, precision allocation, and cache-dependent execution.

  1. Policy-aware joint fitting addresses the limitations of a base-heavy representation. D²MoE optimizes the 2-bit base first and fits successive residuals with that base fixed. This prioritizes base accuracy even when execution predominantly uses higher precisions, constraining the reconstruction quality available at those precisions.

    BitLift jointly fits the prefixes to provide a trade-off suited to the intended precision policy. Figure 3 demonstrates why this flexibility matters: 2-bit-heavy execution favors (0.70,0.15,0.15), whereas 3/4-bit-heavy execution favors (0.10,0.45,0.45).

    Figure 7 compares the resulting reconstruction grids for the same weight group: BitLift permits a 10.7% increase in base error while reducing 3-bit and 4-bit errors by 26.2% and 50.0%, respectively.

    Table 2 demonstrates the model-level benefit under the same precision schedule, reducing Qwen3 WikiText-2/C4 perplexity from 9.33/15.56 to 9.01/14.82.

    Our tied scaling, in comparison to D2MoE's separate scaling, reduces FP16 scale metadata from 6 to 2 bytes per group, with fewer scale loads and multiplications during reconstruction.

  2. A training-free precision prior with expert skipping. D²MoE trains per-expert bitwidth controllers. BitLift instead assigns {0,2,3,4}-bit requests by router rank under an exact nominal budget, skipping low-ranked experts to allocate more bits to higher-ranked experts. This simple prior coordinates precision allocation, representation fitting, and cache allocation without additional training.

  3. Policy-guided cache allocation. D²MoE schedules expert loading and execution. BitLift allocates separate base and refinement pools according to their sizes and policy-dependent demand, coupling cache capacity to the representation and precision prior. At D=48 on Qwen3, selecting the policy-dependent split (18,12) instead of (16,16) reduces traffic by a further 1.4–1.6% (Table 3).

  4. Cache-resident refinement improves quality at nearly unchanged traffic. BitLift-R upgrades precision and revives skipped experts using cached prefixes, without extra fetches or replacement-state updates. It improves perplexity and accuracy over BitLift-A across all tested models and budgets, gaining 0.57–1.37 MMLU points at D=128.

These benefits extend beyond the representation-only comparison: Part 2/2, W2/Q2 also reports lower perplexity and higher mean zero-shot accuracy on Mixtral-8×7B compared with D²MoE-V1’s reported results using its own learned controller.

Response for Reviewer ktTF (2/2)

W1(continued). Although the paper includes scale-fitting and cache ablations, the representation comparison does not fully separate the benefit of policy-aware optimization from differences in error compensation.

Q1. How much of the improvement over D²MoE comes from policy-aware optimization when error compensation is otherwise matched?

Our D²MoE representation baseline is labeled “Sequential residual” in Table 2 and “Sequential fitting” in Figure 7. Both this baseline and BitLift use column-wise GPTQ compensation. The following table distinguishes the matched settings from the representation changes evaluated in Table 2:

ComponentD²MoE-style sequential baselineBitLiftComparison
Calibration128 WikiText-2 training sequences, 2,048 tokens, seed 0SameMatched
Group/block size128/128SameMatched
Hessian damping0.05SameMatched
Column reorderingGARSameMatched
Evaluation precision scheduleFixed rank-based scheduleSameMatched
Column-wise GPTQ compensationEnabled, including intra-block updatesEnabled, including intra-block updatesMatched
Representation initializationFit the INT2 base firstInitialize from an INT4-anchored gridMethod difference
Scale parameterizationSeparate base and residual scalesTied scales (8τ, 2τ, τ)Method difference
Scale fittingClosed-form Hessian-weighted fitting of each residual stageClosed-form policy-weighted joint fitting across prefixesMethod difference
Propagated errorStage-specific reconstruction errorPolicy-weighted combination of prefix errorsMethod difference

Policy-aware error propagation is part of the joint optimization, rather than compensation introduced only for BitLift, as also highlighted in Algorithm 1.

With column-wise compensation enabled in both methods, Table 2 reports WikiText-2/C4 perplexity improvements of 9.33/15.56 → 9.01/14.82 on Qwen3 and 6.65/10.12 → 6.55/9.80 on DeepSeek. This quantifies the sequential (D2MoE)-versus-joint (BitLift) representation improvement.

W2. The evaluation does not establish an advantage over the complete D²MoE system. Its representation is evaluated under BitLift’s precision policy, which is useful for isolating representation quality but excludes D²MoE’s learned controller and scheduling mechanisms.

Q2. How does BitLift compare with the complete D²MoE system in accuracy and measured end-to-end performance under the same hardware and memory budget?

We agree that evaluating D²MoE’s representation under BitLift’s schedule isolates representation quality but does not establish an advantage over its complete system. D²MoE’s code and trained precision controller are not publicly available; therefore matched-hardware runtime comparison is not feasible.

To address the accuracy component, we extend BitLift-A to Mixtral-8×7B at 3 average bits and compare with D²MoE-V1’s reported results on the same backbone and five downstream tasks under same budget. Unlike the representation-only comparison in Table 2, these D²MoE-V1 results use its own learned precision controller rather than BitLift’s precision schedule.

MethodWiki-2 PPLPIQAARC-EBoolQHSWGAvg.
D2MoE-V14.2881.3281.5278.0582.8875.5279.86
BitLift-A4.1982.5480.2685.9981.7776.6481.44

BitLift-A achieves lower WikiText-2 perplexity and 1.58 percentage points higher mean zero-shot accuracy.

W2(continued). Moreover, the reported throughput gains are analytical estimates based on assumed bandwidth and compute capacity, rather than measured end-to-end performance. The practical advantage of BitLift’s combined quantization and caching design therefore remains unclear.

We agree that analytical ceilings alone do not establish speedups, however, these hardware-informed cost modeling and simulation are established approaches for studying LLM inference performance, as used in Dynamic Input Pruning (Federici et al., 2025), LLM in a Flash (Alizadeh et al., 2024), and LLMCompass (Zhang et al., 2024). Our analytical evaluation follows this approach by combining observed expert-weight traffic with bandwidth assumptions grounded in published hardware specifications.

We have now measured full-model decoding on an A100. On OLMoE at batch size 1 and matched cache capacity, BitLift-R achieves 1.18× and 1.21× decoding speedups over uniform 3-bit inference, with storage bandwidths of 4 and 2 GB/s, respectively. The experimental setup and complete latency and throughput results are provided in our response to Reviewer MhwE, Part 1/2, W1.

AuthorsOpenReview ↗

Follow-up on Rebuttal and Additional Results

Dear Reviewer ktTF,

As the discussion period approaches its end, we would appreciate your feedback on our rebuttal.

We clarified BitLift's methodological differences from D²MoE and addressed the concerns regarding error compensation, while adding comparisons with D²MoE's learned precision controller. We also provided measured end-to-end decoding results on hardware.

We would appreciate your assessment of whether these results and clarifications address your concerns, along with any remaining points that may require further discussion.

Thank you for your time and consideration.

Reviewer_MhwEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BitLift targets MoE decoding when expert weights exceed fast-memory capacity and must be fetched from slower storage. It jointly designs dynamic precision allocation, weight representation, and expert caching. A router-rank policy assigns execution precisions, while each expert is stored as a 2-bit base and two 1-bit refinements, jointly fitted for the intended precision policy. At runtime, additional cached refinements can improve execution precision without fetching extra weights for those upgrades.

Experiments across three MoE families show favorable predictive quality and expert-weight traffic compared with the evaluated baselines. The main contribution is the interaction between policy-aware quantization and cache-resident refinement: cached weights provide additional quality at little change in backing-storage traffic. Throughput benefits are estimated analytically and remain unverified on an actual deployment platform.

Critical strengths
  1. The components interact in a meaningful way. Comparisons under identical precision schedules and the shared-scale ablation provide evidence that fitting the representation for its execution policy improves quality. The contribution goes beyond independently applying quantization and caching.

  2. Resident refinement has a clear cache-state contract. Optional reads use the pre-token cache state and do not update replacement state. The identical-traffic property is appropriately restricted to a fixed nominal request trace, while experiments account for subsequent routing changes caused by refined outputs.

  3. Quality and traffic are evaluated across several models and tasks. Three MoE backbones, two perplexity datasets, downstream benchmarks, and cache-component ablations support the observed trade-offs.

Critical weaknesses

W1. Actual inference efficiency is not demonstrated — Major.

Despite implementing packed Triton kernels, the paper provides no measured end-to-end latency or throughput. The resource ceilings and serialized analysis use reference hardware rates and omit measured unpacking, metadata access, dispatch, and other model costs. Ratios of analytical ceilings do not establish actual speedups. This matters because the serialized model's allowance for additional overhead falls to only 2.4 ms/token at one reported cache budget. Representative execution and storage-access measurements are needed to establish whether the traffic savings survive implementation costs.

W2. Comparisons with complete cache-aware systems are missing — Major.

Table 2 provides useful representation comparisons with MatGPTQ[1] and SliceMoE-style AMAT[2]. However, runtime comparisons primarily use uniform GPTQ with caching, rather than the complete precision and caching policies of SliceMoE[2] or HOBBIT[3]. D2MoE is also evaluated with BitLift's schedule instead of its learned controller. These controls isolate representation quality but leave the advantage over existing integrated systems unresolved. A representative comparison with a complete prior policy, or a controlled exchange of representations and runtime policies, would better establish the system contribution.

[1] Kleinegger, Maximilian, Elvir Crnčević, and Dan Alistarh. "MatGPTQ: Accurate and Efficient Post-Training Matryoshka Quantization." arXiv preprint arXiv:2602.03537v1. 2026.

[2] Choi, Yuseon, et al. "SliceMoE: Bit-Sliced Expert Caching under Miss-Rate Constraints for Efficient MoE Inference." arXiv preprint arXiv:2512.12990v1. 2025.

[3] Tang, Peng, et al. "HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference." arXiv preprint arXiv:2411.01433v1. 2024.

Questions for authors
  1. What latency and throughput does the implementation achieve relative to uniform 3-bit inference? Please measure unpacking, dispatch, refinement, and storage-access costs, and compare the additional overhead with the reported break-even allowances.

  2. How does BitLift compare with complete SliceMoE or HOBBIT policies? A comparison on one backbone at matched cache capacity and quality would help distinguish representation gains from runtime-policy gains.

  3. How are reconstruction weights, precision schedules, and cache splits selected? Please provide the data splits, candidate sets, objectives, and total calibration cost needed to reproduce the final configuration.

5 comments

Response for Reviewer MhwE (1/2)

We thank the reviewer for recognizing the meaningful interaction among BitLift’s components. Below, we address the concerns about measured efficiency, comparisons with prior systems, and configuration selection.

W1. Despite implementing packed Triton kernels, the paper provides no measured end-to-end latency or throughput. The resource ceilings and serialized analysis use reference hardware rates and omit measured unpacking, metadata access, dispatch, and other model costs. Ratios of analytical ceilings do not establish actual speedups. This matters because the serialized model's allowance for additional overhead falls to only 2.4 ms/token at one reported cache budget. Representative execution and storage-access measurements are needed to establish whether the traffic savings survive implementation costs.

We added full-model decoding with our packed Triton kernels, using the controlled-runtime approach of SpecMD (Hoang et al., 2026): the real model runs on an A100 with a software-limited GPU expert cache and experts held in CPU memory, and expert transfers are rate-limited to 4 or 2 GB/s (UFS 4.0 / UFS 3.0) by holding each real copy until the corresponding sequential read time has elapsed. We compare BitLift-R with uniform 3-bit GPTQ (Q3) on OLMoE-1B-7B-0125, batch 1, in one shared runtime (CUDA graphs, compiled cache controllers) with the same packed-weight cache budget at D = 48. Each missed component (a Q3 expert, or a BitLift base or refinement piece) is copied to the GPU through a pinned buffer, and decoding waits until the later of the real copy and the sequential read time (weights plus metadata). We decode eight held-out WikiText windows (64-token prompt, 64 greedy tokens), averaging two runs, reversing the order of the methods and runtime variants in the second run.

Flash BandwidthMethodTotal (ms/token)Flash (ms/token)Throughput (tok/s)
4 GB/sQ342.1535.4323.7
4 GB/sBitLift-R35.8028.9427.9
2 GB/sQ378.2170.8312.8
2 GB/sBitLift-R65.1357.9115.4

This establishes BitLift-R is faster than Q3 in all eight windows at both bandwidths, with paired latency savings of 6.35 ± 1.87 ms/token (1.18×) at 4 GB/s and 13.08 ± 3.75 ms/token (1.21×) at 2 GB/s. The traffic savings survive implementation costs: the flash-time savings (6.49 and 12.92 ms/token) match the savings predicted from transferred bytes alone at the imposed bandwidth (6.48 and 12.95 ms/token) within 0.03 ms. The remaining non-flash difference is only +0.14 and −0.16 ms/token.

Q1. What latency and throughput does the implementation achieve relative to uniform 3-bit inference? Please measure unpacking, dispatch, refinement, and storage-access costs, and compare the additional overhead with the reported break-even allowances.

Against Q3 (uniform 3-bit), BitLift-R is 1.18× faster at 4 GB/s and 1.21× at 2 GB/s (W1 table). The per-token costs are:

  • Unpacking: included in the packed expert kernels, which take 0.733/0.768 ms/token for BitLift-A/R vs. 0.674 for Q3. Total GPU kernel work is 4.294/4.322 vs. 4.268 ms/token, including about 3.4 ms of attention.
  • Dispatch and other runtime costs: BitLift-R’s non-flash time (total minus flash) is 6.86 vs. 6.72 ms/token for Q3 at 4 GB/s, an excess of only 0.14 ms/token; at 2 GB/s, it is 7.22 vs. 7.37 ms/token.
  • Refinement: BitLift-R is only 0.18 ms/token (4 GB/s) and 0.08 ms/token (2 GB/s) slower than BitLift-A.
  • Storage access: BitLift-R reads 115.7 vs. 141.6 MB/token (−18.3%, metadata included). Flash is 81-91% of each token (W1 table), and BitLift-R cuts it by 6.49 ms/token at 4 GB/s and 12.92 ms at 2 GB/s.

Response for Reviewer MhwE (2/2)

W2. Table 2 provides useful representation comparisons with MatGPTQ[1] and SliceMoE-style AMAT[2]. However, runtime comparisons primarily use uniform GPTQ with caching, rather than the complete precision and caching policies of SliceMoE[2] or HOBBIT[3]. D2MoE is also evaluated with BitLift's schedule instead of its learned controller. These controls isolate representation quality but leave the advantage over existing integrated systems unresolved. A representative comparison with a complete prior policy, or a controlled exchange of representations and runtime policies, would better establish the system contribution.

Q2. How does BitLift compare with complete SliceMoE or HOBBIT policies? A comparison on one backbone at matched cache capacity and quality would help distinguish representation gains from runtime-policy gains.

We compare BitLift-R with the precision and cache policies of SliceMoE and HOBBIT on Qwen3-30B-A3B and DeepSeek-V2-Lite, at matched cache capacity (D=64) and matched quality: each baseline's thresholds and hit policy are selected on validation data as the lowest-traffic setting within 1% of BitLift-R's validation perplexity. All methods use group size 128; each baseline keeps its own quantizer. We decode the WikiText-2 test split with batch size 1 and windows of up to 2,048 tokens, each starting from an empty cache. Traffic counts expert weights plus scales and zero points.

MethodQwen3 PPL ↓Qwen3 MB/token ↓DeepSeek PPL ↓DeepSeek MB/token ↓
Uniform Q38.95204.76.54220.8
SliceMoE 4/28.79479.26.45437.4
SliceMoE 6/39.10246.36.56306.1
HOBBIT9.06427.86.52408.0
BitLift-R8.92172.96.52180.1

At comparable quality, BitLift-R reduces traffic by 1.42×/1.70× and 2.77×/2.43× relative to SliceMoE 6/3 and 4/2, and by 2.47×/2.27× relative to HOBBIT (Qwen3/DeepSeek).

D²MoE’s learned precision controller is not publicly available, and reproducing it would require implementing and training additional per-expert networks. We therefore extend BitLift-A to Mixtral-8×7B at 3 average bits, matching D²MoE-V1’s nominal average precision, and compare predictive quality on the same backbone and five downstream tasks. Unlike the representation-only comparison in Table 2, these reported D²MoE-V1 results use its own learned precision controller rather than BitLift’s schedule. BitLift-A achieves lower WikiText-2 perplexity (4.19 vs. 4.28) and higher mean zero-shot accuracy (81.44% vs. 79.86%).

MethodWiki-2 PPLPIQAARC-EBoolQHSWGAvg.
D2MoE-V14.2881.3281.5278.0582.8875.5279.86
BitLift-A4.1982.5480.2685.9981.7776.6481.44

Q3. How are reconstruction weights, precision schedules, and cache splits selected? Please provide the data splits, candidate sets, objectives, and total calibration cost needed to reproduce the final configuration.

We fix the three choices in sequence (schedule → reconstruction weights → cache split), freezing each before the next.

Precision schedule π0Reconstruction weights λCache split (C_B, C_R)
Candidates18 rank-monotone schedules over {0, 2, 3, 4} bits, averaging 2–4 bits19 settings of (λ₂, λ₃, λ₄), provided in Appendix F; five are shown in Figure 35 splits per budget with 2C_B + C_R = D (e.g., C_B ∈ {12, 16, 18, 20, 24} at D = 48)
Selection dataWikiText-2 and C4 validation, 20 × 2,048-token windows each (Qwen3, OLMoE)WikiText-2 and C4 validationWikiText-2 validation (129 windows) and C4 validation (256 windows)
ObjectiveLowest perplexity at matched budgetLowest perplexity under π0Lowest BitLift-A traffic, Eq. (10)
CostOne evaluation pass per schedule, model, and corpusOne quantization run per candidateOne routing-trace pass per model and corpus, then 130 CPU-only cache replays
AuthorsOpenReview ↗

Follow-up on Rebuttal and Additional Results

Dear Reviewer MhwE,

As the discussion period approaches its end, we would appreciate your feedback on our rebuttal.

We have provided additional experiments addressing your major concerns, including measured end-to-end decoding latency and throughput on hardware. We also added comparisons with complete SliceMoE and HOBBIT policies at matched cache capacity and quality, along with detailed clarifications of our configuration-selection procedures.

We would appreciate your assessment of whether these additional results and clarifications address your concerns, along with any remaining points that may require further discussion.

Thank you for your time and consideration.

Reviewer_1hn9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a unified mixed-precision quantization and caching strategy for efficient MoE inference. Specifically, the paper leverages the router's top-k selection to vary bitwidth across activated experts for a given token. As the same experts can be represented by different precision for the computation of different tokens, the paper avoids loading/offloading multiple copies of these experts by representing them by a combination of base and refinement bits. The paper proposes an optimization strategy to jointly minimize the quantization loss for the base and the refinement representations. Additionally, the paper proposes a calibration-data-based capacity allocation for the base and refinement experts' weights. The proposed method is evaluated on large MoE models on benchmark LLM tasks.

Critical strengths
  1. The proposed method is well-designed and intuitive

  2. Substantial empirical evidence is provided to show that the method outperforms other static/dynamic mixed-precision MoE inference baselines. The paper also demonstrates an acceptable amount of DRAM traffic reduction.

Critical weaknesses
  1. The dependence on top-k rank for mixed-precision seems too simple to me. For example, based on the intuition that not all tokens are equally important in a layer and/or not all tokens require an equal level of bit-width precision for their experts, one can design a token-wise dynamic mixed-precision strategy that may improve performance significantly. Indeed, the relative performance improvement compared to static mixed-precision strategies in Table 1 seems marginal to me. Moreover, the current method is heavily calibration-data-dependent, which may lead to suboptimal performance gains. The paper also skips comparing other recent mixed-precision MoE setups, e.g., [1], [2].

  2. The paper's system-level performance evaluation is limited. For example, the paper does not compare throughput with other mixed-precision or relevant efficient MoE inference baselines. Additionally, it would be appropriate to measure the system-level performance in terms of throughput, latency, and time-to-first-token (TTFT) under practical inference frameworks, e.g., vLLM or SGLang, to realize the actual gain of the proposed method.

[1] Huang, W., Liao, Y., Liu, J., He, R., Tan, H., Zhang, S., ... & Qi, X. (2025, May). Mixture compressor for mixture-of-experts llms gains more. In International Conference on Learning Representations (Vol. 2025, pp. 68628-68648).

[2] Chowdhury, M. N. R., El Maghraoui, K., Tsai, H., Wang, N., Burr, G., Liu, L., & Wang, M. (2026, April). Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees. In International Conference on Learning Representations (Vol. 2026, pp. 76345-76380).

Questions for authors

Please see the weaknesses

3 comments

Response for Reviewer 1hn9 (1/2)

We thank the reviewer for their time and feedback, and for recognizing BitLift’s design and empirical strengths. Below, we address the concerns raised with clarifications and additional experiments on precision allocation, calibration sensitivity, baseline comparisons, and end-to-end inference.

W1. The dependence on top-k rank for mixed-precision seems too simple to me. ... one can design a token-wise dynamic mixed-precision strategy that may improve performance significantly.

We agree that token-wise precision allocation could improve quality. However, the top-k schedule specifies a precision-policy prior, not the final execution policy. It provides a training-free, empirically strong starting point under an exact budget (see Figure 1 and Section 3.1). BitLift-R adapts this prior at runtime: the precision actually executed depends jointly on router-derived expert importance and the available cached components.

Precision allocation is one axis of our co-design, coupled with progressive representation fitting and cache-aware execution. Improving allocation quality in isolation may therefore not yield a better system-level outcome. Our focus is the overall quality–cost trade-off achieved through their coordinated design.

In comparison, D2MoE uses a learned token-dependent precision controller, yet BitLift-A, with a joint design of the representation and a simple rank-based precision prior, achieves lower Mixtral WikiText-2 perplexity (4.19 vs. 4.28) and higher mean accuracy across five zero-shot tasks (81.44% vs. 79.86%) than D2MoE’s results (please see our response to Reviewer ktTF, part 2/2, W2/Q2, for the full comparison table).

We also evaluated an offline oracle that allocates precision budgets across tokens and layers under an overall 2.5-bit average, guided by BF16 MoE-output reconstruction error. On Qwen3, WikiText-2/C4 perplexity improves from 9.0854/14.8174 to 8.9574/14.5723. The oracle reduces perplexity by approximately 1.4% on WikiText-2 and 1.7% on C4, showing that our simple prior remains competitive with this more flexible allocation.

W1 (continued). Indeed, the relative performance improvement compared to static mixed-precision strategies in Table 1 seems marginal to me.

We respectfully disagree with characterizing the improvements over static mixed-precision methods as marginal. At the reported 2.5-bit settings, BitLift-A improves mean zero-shot accuracy over GEMQ, a state-of-the-art static mixed-precision baseline, by 4.09 percentage points on Qwen3 and 1.79 points on DeepSeek-V2-Lite, while reducing WikiText-2 and C4 perplexity on both models by up to 6.8%. For context, GEMQ itself reports improvements over MC-MoE[1] of 3.73 and 1.13 points, respectively, on these same models. These gains represent consistent advances over a strong baseline. Our additional comparisons with MC-MoE[1] and Chowdhury et al.[2], discussed below, further support this trend.

Given these consistent gains, could the reviewer clarify which results in Table 1 they consider marginal?

W1(continued). Moreover, the current method is heavily calibration-data-dependent, which may lead to suboptimal performance gains.

We do not agree that BitLift is “heavily calibration-data-dependent.” Its reported gains already extend beyond the calibration corpus: the WikiText-2-calibrated models are evaluated on C4 and seven zero-shot downstream tasks without task-specific calibration.

To directly test sensitivity to calibration choice, we additionally evaluated Qwen3-30B-A3B at 2.5 average bits using four calibration corpora: WikiText-2, C4, PTB, and DCLM-Edu. We held the precision policy, objective weights, calibration budget, and quantization settings fixed.

Calibration CorpusWiki-2C4PTBDCLM-Edu
Wiki-29.014414.822017.713913.0178
C49.111714.652217.343412.9335
PTB9.266714.909717.222013.0664
DCLM-Edu9.296614.930017.795613.0840
MetricWiki-2 calibrationC4 calibrationPTB calibrationDCLM-Edu calibration
Seven-task zero-shot average69.7869.2568.7569.81

Across these calibration choices, perplexity varies by at most 3.33% within any evaluation corpus, and mean zero-shot accuracy spans 1.06 percentage points. The C4, PTB, and DCLM-Edu-calibrated models also outperform GEMQ’s reported seven-task mean accuracy by 3.06-4.12 percentage points. Together with the results already reported in Table 1, this evidence supports robustness across the tested calibration distributions and does not support characterizing the dependence as “heavy.”

Could the reviewer clarify the empirical basis for characterizing BitLift as “heavily calibration-data-dependent”?

Response for Reviewer 1hn9 (2/2)

W1(continued). The paper also skips comparing other recent mixed-precision MoE setups, e.g., [1], [2].

Our submitted evaluation includes GEMQ, a state-of-the-art static mixed-precision baseline that already reports improvements over MC-MoE[1]. To address the reviewer's concern explicitly, we extend the comparison to MC-MoE[1] and Chowdhury et al. [2], reporting perplexity and all seven zero-shot tasks. All methods use average 2.5-bit expert settings.

ModelMethodWiki-2 ↓C4 ↓ARC-C ↑ARC-E ↑HS ↑PIQA ↑WG ↑MathQA ↑MMLU ↑Avg. ↑
OLMoEMC-MoE8.2215.2343.7769.6571.3374.2165.4328.5446.6257.08
OLMoEChowdhury et al.9.4015.3639.5167.0071.3275.1464.8825.8043.7055.33
OLMoEBitLift-A7.4913.0847.6173.7875.7578.8466.6128.2450.0760.13
Qwen3-30B-A3BMC-MoE10.9417.0145.9968.4372.1376.6166.2236.9267.3961.96
Qwen3-30B-A3BChowdhury et al.10.5518.3947.6168.3971.6378.1367.0147.1070.9864.41
Qwen3-30B-A3BBitLift-A9.0114.8254.8678.7975.9779.6068.9854.2776.0069.78

We additionally evaluated BitLift-A on Mixtral-8×7B at an average precision of 2.5 bits.

MethodWiki-2 ↓C4 ↓ARC-C ↑ARC-E ↑HS ↑PIQA ↑WG ↑MathQA ↑MMLU ↑Avg. ↑
MC-MoE[1]5.109.2151.2874.8779.1880.3673.9534.9455.7864.34
GEMQ5.039.0251.7975.9379.5781.0773.7234.1059.7465.13
Chowdhury et al.[2]——54.4478.4179.3680.7974.3536.2858.2365.98
BitLift-A4.387.9855.0379.1781.6282.5976.1639.1065.1368.40

W2. The paper's system-level performance evaluation is limited. For example, the paper does not compare throughput with other mixed-precision or relevant efficient MoE inference baselines. Additionally, it would be appropriate to measure the system-level performance in terms of throughput, latency, and time-to-first-token (TTFT) under practical inference frameworks, e.g., vLLM or SGLang, to realize the actual gain of the proposed method.

Thank you for suggesting this evaluation. Our primary target is low-batch autoregressive decoding in memory-constrained deployments, which we assess through decode latency and throughput. TTFT characterizes initial-response latency and is outside the scope of this decoding comparison. We have integrated BitLift-A's packed kernels into vLLM for GPU-resident inference and added full-model decoding measurements against BF16, uniform INT4 with Marlin, uniform INT3 with Humming, and a static 2.5-bit Humming baseline.

To isolate execution performance from weight-transfer costs, we evaluate OLMoE-1B-7B-0125 on an NVIDIA A100-SXM4-80GB at batch size 1, with all weights resident in GPU memory. All methods use vLLM 0.17.1 with identical engine settings, FlashAttention and full decode CUDA graphs, with 512 input tokens, 128 generated tokens, three warmups and ten timed trials. Quantized methods use INT4 Marlin attention projections. Decode timing excludes the first generated token.

MethodDecode latency (ms/token) ↓Decode throughput (tok/s) ↑
BF163.025330.54
Uniform INT4 / Marlin2.100476.10
Uniform INT3 / Humming2.105475.17
Static 2.5-bit / Humming2.084479.87
BitLift-A 2.5-bit1.832545.77

At batch 1, BitLift-A achieves 1.15× the decode throughput of both uniform INT4 and uniform INT3, and 1.14× that of static 2.5-bit.

Integrating our cache controller into vLLM requires changes to its CUDA-graph execution. We therefore evaluate caching in a separate shared runtime at matched cache capacity; end-to-end latency and throughput are reported in our response to Reviewer MhwE, Part 1/2, W1.

AuthorsOpenReview ↗

Follow-up on Rebuttal and Additional Results

Dear Reviewer 1hn9,

As the discussion period approaches its end, we would appreciate your feedback on our rebuttal.

We have provided additional experiments addressing your concerns, including comparisons with additional baselines and a learned token-wise dynamic precision allocation method (D²MoE), calibration sensitivity across four datasets, oracle precision allocation, and measured decoding throughput in vLLM.

We would appreciate your assessment of whether these results address your concerns, along with your thoughts on the specific clarification questions raised in our rebuttal.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.