- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BitLift, a training-free framework for mixed-precision MoE inference under limited memory. It combines router-rank precision allocation with a shared 2-bit base and two 1-bit refinement planes, optimized across precision levels. A two-pool cache separates base and refinement data, and resident refinement planes can increase execution precision without additional fetches for the upgrade. Experiments evaluate predictive quality and weight traffic, while an analytical model estimates throughput.
Critical strengths
-
BitLift improves mean downstream accuracy by 0.93%–1.30% over the D²MoE representation under the same precision policies across three MoE models.
-
Using resident refinement planes to improve precision is a practical idea. The evaluation shows improved predictive quality with nearly unchanged weight-transfer traffic.
Critical weaknesses
-
The contribution over D²MoE appears limited. D²MoE already supports the same 2+1+1-bit nested representation, dynamic expert precision, and bit-aware memory management. BitLift adds policy-weighted quantization and cache-resident refinement, but the reported improvements do not clearly establish a substantial advance over this existing framework. Although the paper includes scale-fitting and cache ablations, the representation comparison does not fully separate the benefit of policy-aware optimization from differences in error compensation.
-
The evaluation does not establish an advantage over the complete D²MoE system. Its representation is evaluated under BitLift’s precision policy, which is useful for isolating representation quality but excludes D²MoE’s learned controller and scheduling mechanisms. Moreover, the reported throughput gains are analytical estimates based on assumed bandwidth and compute capacity, rather than measured end-to-end performance. The practical advantage of BitLift’s combined quantization and caching design therefore remains unclear.
Questions for authors
-
How much of the improvement over D²MoE comes from policy-aware optimization when error compensation is otherwise matched?
-
How does BitLift compare with the complete D²MoE system in accuracy and measured end-to-end performance under the same hardware and memory budget?
5 comments
Response for Reviewer ktTF (2/2)
W1(continued). Although the paper includes scale-fitting and cache ablations, the representation comparison does not fully separate the benefit of policy-aware optimization from differences in error compensation.
Q1. How much of the improvement over D²MoE comes from policy-aware optimization when error compensation is otherwise matched?
Our D²MoE representation baseline is labeled “Sequential residual” in Table 2 and “Sequential fitting” in Figure 7. Both this baseline and BitLift use column-wise GPTQ compensation. The following table distinguishes the matched settings from the representation changes evaluated in Table 2:
| Component | D²MoE-style sequential baseline | BitLift | Comparison |
|---|---|---|---|
| Calibration | 128 WikiText-2 training sequences, 2,048 tokens, seed 0 | Same | Matched |
| Group/block size | 128/128 | Same | Matched |
| Hessian damping | 0.05 | Same | Matched |
| Column reordering | GAR | Same | Matched |
| Evaluation precision schedule | Fixed rank-based schedule | Same | Matched |
| Column-wise GPTQ compensation | Enabled, including intra-block updates | Enabled, including intra-block updates | Matched |
| Representation initialization | Fit the INT2 base first | Initialize from an INT4-anchored grid | Method difference |
| Scale parameterization | Separate base and residual scales | Tied scales (8τ, 2τ, τ) | Method difference |
| Scale fitting | Closed-form Hessian-weighted fitting of each residual stage | Closed-form policy-weighted joint fitting across prefixes | Method difference |
| Propagated error | Stage-specific reconstruction error | Policy-weighted combination of prefix errors | Method difference |
Policy-aware error propagation is part of the joint optimization, rather than compensation introduced only for BitLift, as also highlighted in Algorithm 1.
With column-wise compensation enabled in both methods, Table 2 reports WikiText-2/C4 perplexity improvements of 9.33/15.56 → 9.01/14.82 on Qwen3 and 6.65/10.12 → 6.55/9.80 on DeepSeek. This quantifies the sequential (D2MoE)-versus-joint (BitLift) representation improvement.
W2. The evaluation does not establish an advantage over the complete D²MoE system. Its representation is evaluated under BitLift’s precision policy, which is useful for isolating representation quality but excludes D²MoE’s learned controller and scheduling mechanisms.
Q2. How does BitLift compare with the complete D²MoE system in accuracy and measured end-to-end performance under the same hardware and memory budget?
We agree that evaluating D²MoE’s representation under BitLift’s schedule isolates representation quality but does not establish an advantage over its complete system. D²MoE’s code and trained precision controller are not publicly available; therefore matched-hardware runtime comparison is not feasible.
To address the accuracy component, we extend BitLift-A to Mixtral-8×7B at 3 average bits and compare with D²MoE-V1’s reported results on the same backbone and five downstream tasks under same budget. Unlike the representation-only comparison in Table 2, these D²MoE-V1 results use its own learned precision controller rather than BitLift’s precision schedule.
| Method | Wiki-2 PPL | PIQA | ARC-E | BoolQ | HS | WG | Avg. |
|---|---|---|---|---|---|---|---|
| D2MoE-V1 | 4.28 | 81.32 | 81.52 | 78.05 | 82.88 | 75.52 | 79.86 |
| BitLift-A | 4.19 | 82.54 | 80.26 | 85.99 | 81.77 | 76.64 | 81.44 |
BitLift-A achieves lower WikiText-2 perplexity and 1.58 percentage points higher mean zero-shot accuracy.
W2(continued). Moreover, the reported throughput gains are analytical estimates based on assumed bandwidth and compute capacity, rather than measured end-to-end performance. The practical advantage of BitLift’s combined quantization and caching design therefore remains unclear.
We agree that analytical ceilings alone do not establish speedups, however, these hardware-informed cost modeling and simulation are established approaches for studying LLM inference performance, as used in Dynamic Input Pruning (Federici et al., 2025), LLM in a Flash (Alizadeh et al., 2024), and LLMCompass (Zhang et al., 2024). Our analytical evaluation follows this approach by combining observed expert-weight traffic with bandwidth assumptions grounded in published hardware specifications.
We have now measured full-model decoding on an A100. On OLMoE at batch size 1 and matched cache capacity, BitLift-R achieves 1.18× and 1.21× decoding speedups over uniform 3-bit inference, with storage bandwidths of 4 and 2 GB/s, respectively. The experimental setup and complete latency and throughput results are provided in our response to Reviewer MhwE, Part 1/2, W1.
Follow-up on Rebuttal and Additional Results
Dear Reviewer ktTF,
As the discussion period approaches its end, we would appreciate your feedback on our rebuttal.
We clarified BitLift's methodological differences from D²MoE and addressed the concerns regarding error compensation, while adding comparisons with D²MoE's learned precision controller. We also provided measured end-to-end decoding results on hardware.
We would appreciate your assessment of whether these results and clarifications address your concerns, along with any remaining points that may require further discussion.
Thank you for your time and consideration.
Response for Reviewer ktTF (1/2)
We thank the reviewer for their time and feedback. Below, we address the concerns raised and provide supporting clarifications and additional experimental evidence.
We disagree and believe there are fundamental differences between BitLift and D²MoE. The shared 2+1+1 encoding only describes the storage layout; BitLift’s contribution lies in the joint design of representation fitting, precision allocation, and cache-dependent execution.
Policy-aware joint fitting addresses the limitations of a base-heavy representation. D²MoE optimizes the 2-bit base first and fits successive residuals with that base fixed. This prioritizes base accuracy even when execution predominantly uses higher precisions, constraining the reconstruction quality available at those precisions.
BitLift jointly fits the prefixes to provide a trade-off suited to the intended precision policy. Figure 3 demonstrates why this flexibility matters: 2-bit-heavy execution favors (0.70,0.15,0.15), whereas 3/4-bit-heavy execution favors (0.10,0.45,0.45).
Figure 7 compares the resulting reconstruction grids for the same weight group: BitLift permits a 10.7% increase in base error while reducing 3-bit and 4-bit errors by 26.2% and 50.0%, respectively.
Table 2 demonstrates the model-level benefit under the same precision schedule, reducing Qwen3 WikiText-2/C4 perplexity from 9.33/15.56 to 9.01/14.82.
Our tied scaling, in comparison to D2MoE's separate scaling, reduces FP16 scale metadata from 6 to 2 bytes per group, with fewer scale loads and multiplications during reconstruction.
A training-free precision prior with expert skipping. D²MoE trains per-expert bitwidth controllers. BitLift instead assigns {0,2,3,4}-bit requests by router rank under an exact nominal budget, skipping low-ranked experts to allocate more bits to higher-ranked experts. This simple prior coordinates precision allocation, representation fitting, and cache allocation without additional training.
Policy-guided cache allocation. D²MoE schedules expert loading and execution. BitLift allocates separate base and refinement pools according to their sizes and policy-dependent demand, coupling cache capacity to the representation and precision prior. At D=48 on Qwen3, selecting the policy-dependent split (18,12) instead of (16,16) reduces traffic by a further 1.4–1.6% (Table 3).
Cache-resident refinement improves quality at nearly unchanged traffic. BitLift-R upgrades precision and revives skipped experts using cached prefixes, without extra fetches or replacement-state updates. It improves perplexity and accuracy over BitLift-A across all tested models and budgets, gaining 0.57–1.37 MMLU points at D=128.
These benefits extend beyond the representation-only comparison: Part 2/2, W2/Q2 also reports lower perplexity and higher mean zero-shot accuracy on Mixtral-8×7B compared with D²MoE-V1’s reported results using its own learned controller.