acceptodds
Under review as a conference paper at ICLR 2027

BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Acceleration

Abstract

Speculative decoding accelerates autoregressive generation by using a lightweight draft to propose multiple tokens for parallel verification. However, existing methods often require an additional draft model or weight representation, introducing non-negligible memory overhead on resource-constrained devices. Self-speculative approaches reduce this overhead, yet still face trade-offs between draft quality, target quality, and storage efficiency. We propose BitNest, a bit-nested speculative decoding framework that embeds a low-precision draft directly into the higher-precision target representation. Instead of deriving a draft from a predefined target, BitNest first constructs a strong low-precision base and then recovers the higher-precision target through residual refinement, enabling both models to share a single physical weight representation. BitNest further extends this progressive-precision design to the KV cache for long-context inference. Across multiple 7B–8B edge friendly LLMs and diverse workloads, BitNest achieves an average speculative acceptance rate of 95.2% while closely preserving higher-precision model quality, and delivers 1.48–1.61 end-to-end speedup over FP16 autoregressive decoding. On the LLaMA models supported by all representative self-speculative baselines, BitNest also achieves consistently competitive or higher decoding speedup.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Speculative Decoding, LLM Quantization, KV Cache Quantization, Large Language Model Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZwAFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BitNest, a self-speculative decoding framework that constructs a low-precision draft and a higher-precision target within a shared weight representation. The method stores base and refinement codes in independently accessible planes, extends this design to the KV cache, and evaluates inference quality, speed, memory usage, and edge-device performance.

Critical strengths
  1. The representation and storage designs fit the stated memory-efficiency objective: drafting can access the base without loading refinement bits, while verification uses both planes.

  2. The evaluation includes actual edge hardware and reports energy and memory alongside decoding speed, providing useful deployment evidence beyond a desktop-GPU benchmark.

Critical weaknesses
  1. The methods use different draft–target precisions, and QuantSpec generates 64 tokens whereas the default evaluation generates 256, so the reported speedup ranking also reflects different operating conditions.

  2. In the OpenVLA experiment, speculative decoding reduces decode time but increases total per-action latency from 201.5 ms to 212.7 ms, illustrating a material prefill limitation of the current implementation

Questions for authors
  1. Could the authors report BitNest’s speedup over autoregressive decoding with the same nested W8A8 target across all main workloads, separating quantization gains from speculative-decoding gains?

  2. Could the comparisons use a common output length and include absolute latency or throughput, in addition to speedups normalized within each inference engine?(“QuantSpec follows its official 64-token setting”)

  3. The authors conclude that using KV4 only for drafting while retaining KV8 for verification better preserves target fidelity than using KV4 for both stages. However, Table 11 reports different GSM8K accuracies for W8A8+KV8 autoregressive decoding (82.4%), KV8-draft→KV8-target speculation (82.2%), and KV4-draft→KV8-target speculation (81.6%); given the stated default of greedy decoding, could the authors explain these differences?

6 comments
AuthorsOpenReview ↗

Weakness 1

We sincerely thank the reviewer for the positive assessment, in particular for recognizing that the dual-plane design fits BitNest's memory-efficiency goal and that our edge-hardware evaluation provides deployment evidence beyond a desktop-GPU benchmark.

W1. Different draft–target precisions and output lengths across methods Thank you for this comment. We believe the differing precisions may have been read as an inconsistent setup, and we clarify this below; the output-length mismatch is answered under Q2&W1. The precisions differ because the methods themselves differ. A self-speculative method obtains its draft by lowering precision somewhere in the target, and where it lowers precision is that method's design:

  • QSpec lowers activation precision on shared 4-bit weights (W4A4 draft, W4A16 target).
  • QuantSpec keeps 16-bit weights and lowers only the KV-cache precision (KV4 draft, KV8 target). It beats FP16 only when --wq adds 4-bit draft MLP weights; without them it runs at 0.81–0.89× of FP16.
  • BitNest lowers weight precision inside one nested model (W4 base plane as the draft, both planes as the W8 target), and also nests the KV cache at long context. No prior method nests the draft inside the target's quantized weights, so there is no shared precision setting under which all three would still be the original methods. Aligning their precisions would remove the very gap each one draws its speedup from. We therefore keep every baseline at its official code and configuration; our QSpec measurement, 1.26×, matches the 1.24× it reports. We compare the methods on outcomes that do not depend on matching precisions:

Quality of the target being verified. Speculative decoding outputs exactly the target's tokens, so the target's precision sets the delivered quality. All rows are evaluated on the same tokens (QSpec GSM8K from Table 2):

TargetLLaMA-2 WikiText-2 PPL ↓LLaMA-2 code PPL ↓LLaMA-3 WikiText-2 PPL ↓LLaMA-3 GSM8K ↑
FP165.472.6076.1450.1
BitNest (W8A8)5.512.6126.2349.8
QSpec (W4A16)5.652.6426.6537.0

BitNest's target stays within 0.09 PPL and 0.3 GSM8K points of FP16. QSpec's 4-bit-weight target loses 13 GSM8K points on LLaMA-3. QuantSpec's target keeps 16-bit weights, so its speedup comes at FP16 weight bandwidth.

-Absolute decode throughput. On identical prompts and hardware, BitNest exceeds QuantSpec by 1.09–1.26× in all 12 cells (Q2&W1).

-The speculative gain on its own. Speculation alone gives 1.21–1.31× over the same nested W8A8 target decoded autoregressively (Q1).

AuthorsOpenReview ↗

Question 2 and Weakness 1

Q2&W1. Mismatched output lengths and missing absolute throughput.

Thank you for this constructive suggestion. QuantSpec was evaluated at 64 generated tokens because its public code hits a device-side assert at 128 or more generated tokens in our setup, so it cannot be extended to 256. We therefore re-ran BitNest at QuantSpec's official 64-token setting, on the same 2,048-token input prompts and the same GPU. The table reports both views at this common length. The first three columns give absolute decode throughput in tok/s. The last two give the speedup normalized within each engine, i.e., over that engine's own FP16 autoregressive decoding:

Model / domainBitNest (tok/s)QuantSpec +wq (tok/s)BitNest ÷ QuantSpecBitNest ÷ own FP16QuantSpec ÷ own FP16
LLaMA-2 wiki261.754.31.14×1.49×1.16×
LLaMA-2 GSM8K63.354.91.15×1.57×1.17×
LLaMA-2 code62.857.51.09×1.51×1.23×
LLaMA-2 ShareGPT64.158.11.10×1.55×1.24×
LLaMA-2 longdoc64.355.91.15×1.56×1.19×
LLaMA-2 PG-1960.250.91.18×1.46×1.09×
LLaMA-3 wiki255.644.91.24×1.40×1.08×
LLaMA-3 GSM8K57.650.21.15×1.46×1.21×
LLaMA-3 code60.648.81.24×1.53×1.17×
LLaMA-3 ShareGPT61.653.81.15×1.55×1.29×
LLaMA-3 longdoc58.246.11.26×1.47×1.11×
LLaMA-3 PG-1956.545.21.25×1.42×1.09×

Both views give the same ranking in every cell. In absolute terms BitNest is 1.09–1.26× faster than QuantSpec. Within each engine BitNest reaches 1.40–1.57× over FP16, against 1.08–1.29× for QuantSpec. The gap does not come from a faster inference engine on our side. QuantSpec's own FP16 baseline (gpt-fast, 41.5–46.9 tok/s) is faster than BitNest's (39.6–41.5 tok/s). BitNest thus starts from the slower 16-bit baseline and still ends with the higher speculative throughput. The difference comes from what each draft and target read per step. At batch size 1, decoding is bound by memory reads, and at a 2K context the weights dominate the bytes read. QuantSpec's target reads 16-bit weights, while BitNest's target reads 8-bit weights and its draft reads only the 4-bit base plane. Shortening the output does not change BitNest's behavior either. At 64 tokens BitNest remains 1.15–1.31× faster than its own W8A8 autoregressive decoding, matching the 1.21–1.31× measured at 256 tokens (Q1).This table will be added to the revision.

AuthorsOpenReview ↗

Question 1 and Weakness 2

Q1. Speedup should be measured against the same nested W8A8 autoregressive model. Thank you for this constructive suggestion. The reference below is BitNest's own W8A8 target decoded autoregressively. It uses the same stored weight planes, the same kernels and the same engine, and only the draft is removed, so the ratio isolates the speculative gain. All entries use 2,048-token input prompts and 256 generated tokens:

Spec ÷ W8A8-ARwiki2GSM8KcodeShareGPTlongdocPG-19
LLaMA-2-7B1.21×1.27×1.27×1.29×1.28×1.25×
LLaMA-3-8B1.25×1.28×1.31×1.31×1.29×1.25×
Qwen2-7B1.23×1.26×1.27×1.28×1.27×1.27×
Qwen2.5-7B1.26×1.29×1.29×1.29×1.29×1.27×

The end-to-end gain over FP16 splits into quantization and speculation, which multiply:

ModelQuantization (W8A8-AR ÷ FP16)Speculation (Spec ÷ W8A8-AR)Total (Spec ÷ FP16)
LLaMA-2-7B1.24–1.25×1.21–1.29×1.51–1.61×
LLaMA-3-8B1.20–1.22×1.25–1.31×1.49–1.59×
Qwen2-7B1.20–1.22×1.23–1.28×1.49–1.55×
Qwen2.5-7B1.20–1.24×1.26–1.29×1.52–1.59×

Speculation contributes about as much as quantization, and both come from a single stored model. The measured speculative gain also matches what the acceptance rate predicts. With per-token acceptance α ≈ 0.94–0.95 and a draft/target step-cost ratio c ≈ 0.60–0.66, the analytic speedup is 1.21–1.27×, in line with the measured 1.21–1.31×.

W2. Limited end-to-end speedup on OpenVLA due to prefill overhead.

Thank you for this constructive comment. BitNest is designed for LLM generation. We include OpenVLA as a transferability test: it checks whether nested self-speculation carries over to a non-text domain, and whether it can help beyond LLM generation. Per-action breakdown on LIBERO (A6000, median over 100 frames, γ=7):

Per actionPrefill (288 tok, ms)Decode (6 tok, ms)Total (ms)AcceptanceActions done in one round
FP16 AR65.9135.6201.5––
BitNest W8A8 AR111.7105.3216.9––
BitNest spec111.7100.8212.70.9385 / 100

Prefill currently limits end-to-end latency. Each action needs a fresh, compute-bound 288-token prefill, and our implementation dequantizes the planes for this step. The 6 decode tokens per action also give speculation only a single round over which to amortize its cost.

The more important finding is that nesting transfers to action tokens. The W4A8 draft's acceptance is 0.93, on par with text generation. In 85 of the 100 actions, the 4-bit plane alone produces exactly the action of the 8-bit target. We believe this points to a promising direction for accelerating robot policies. In this setting, the nested planes need not be used for speculative decoding at all. They can instead serve as the foundation for dynamic bit-width selection, where the policy chooses 4 or 8 bits at each step from the same stored weights.

  • Step-level precision switching. Most control steps do not need 8 bits, but some do: running the whole trajectory at 4 bits lowers LIBERO success from 86% to 80%. Easy steps can run on the 4-bit plane alone with no verification, at about 0.6 of the cost of an 8-bit step. Hard steps use the full 8-bit target. The draft–target agreement we measure is a natural signal for deciding which steps need 8 bits.
  • No extra memory on the robot. Robots typically run on memory-constrained edge modules such as the Jetson Orin we evaluated. Unlike a separate draft model or a second checkpoint, BitNest changes precision without loading anything, because one stored model serves both levels.
  • Decode-dominant VLAs. Policies that generate action chunks, or reason before acting, decode many tokens per prefill, so the decode gain carries through. With 140 action tokens per prefill, the W8A8 plane alone is already 1.24× faster than FP16 per chunk.

Combined with an int8 prefill GEMM that reads the planes directly, we see BitNest's nesting as a basis for dynamic-precision robot inference. In the revision, we will present OpenVLA as a transferability result with this end-to-end caveat, and add this discussion as future work.

Reviewer_ZHzyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces BitNest, a self-speculative decoding framework that embeds a 4-bit draft model within an 8-bit target representation using residual quantization and dual-plane weight storage. The approach also extends to KV-cache quantization for long-context inference. Experiments on 7B–8B models demonstrate high draft acceptance rates, reduced memory usage, and decoding speedups, including evaluations on edge devices.

Critical strengths
  1. The shared weight representation reduces the memory overhead of maintaining separate draft and target models, which is particularly useful for memory-constrained devices.

  2. The paper provides evaluations across multiple models and workloads, including long-context inference and edge-device measurements of throughput, memory usage, and energy consumption.

Critical weaknesses
  1. Limited methodological novelty. The approach primarily combines established quantization techniques, residual error encoding, and bit-plane storage to construct a low-precision drafter and higher-precision verifier. Although the integration is useful, the paper does not sufficiently establish the novelty of these individual components or demonstrate a substantial methodological advance beyond existing quantization-based self-speculative decoding.

  2. Incomplete comparison and attribution of speedups. While QSpec, QuantSpec, and Draft & Verify are evaluated, the experimental comparison omits recent approaches such as Speculative Streaming and DFlash. More importantly, the reported speedups are primarily relative to FP16 autoregressive decoding, making it difficult to isolate the benefits of speculative decoding from those of quantization. Comparisons with matched W8A8 autoregressive execution and stronger contemporary decoding approaches would better establish the practical contribution.

Questions for authors
  1. What is the fundamental methodological novelty of the base-first residual construction compared with existing progressive or residual quantization techniques?

  2. Can the authors provide a clearer comparison with recent methods such as Speculative Streaming and DFlash, including throughput versus total memory footprint under comparable hardware and model settings?

  3. How much of the observed speedup is attributable specifically to speculative decoding versus quantized inference and kernel optimizations? Can this be isolated through matched-precision ablations?

4 comments
AuthorsOpenReview ↗

Question 1 and Weakness 1

Q1&W1. Limited novelty relative to progressive / residual quantization.

Thank you for this insightful question, which helps us sharpen our core contribution. The key difference lies in how the bit-widths are optimized. Progressive and nested methods, such as Matryoshka Quantization (MatQuant) and its recent post-training variant MatGPTQ, quantize all bit-widths simultaneously; MatGPTQ, for instance, selects each code against all tiers in one GPTQ pass and propagates a single averaged error, (e8+e4)/2(e_8+e_4)/2, to the remaining columns. Residual quantization proceeds stage by stage, but each stage is quantized independently with its own scale or codebook, and the stages are reconstructed and summed separately.

BitNest instead quantizes the tiers sequentially, base first (Sec. 1, 3.2). This follows from our observation that the tiers are not equally hard: 8-bit weights tolerate rounding error, whereas 4-bit quality comes almost entirely from optimization. BitNest therefore spends the optimization on the base. After the rotations R1/R2R_1/R_2 are folded into the weights (in our implementation they are learned under the W4A8 draft configuration), the W4 base (q4,s4)(q_4, s_4) receives GPTQ's full error feedback and is then fixed. The remaining error ΔW=W′−q4s4\Delta W = W' - q_4 s_4 is rounded in closed form to a 4-bit refinement with the derived scale sref=s4/16s_{\mathrm{ref}} = s_4/16, giving Wtarget=(16q4+qr) ,srefW_{\mathrm{target}} = (16q_4 + q_r)\ ,s_{\mathrm{ref}} (Appendix A.1). Unlike residual quantization, base and refinement form one standard INT8 code with no extra scale metadata, executed by a single W8A8 GEMM (Sec. 3.3), and the W4 draft is an exact subset of the W8 target, a relationship that independent quantization would break (Sec. 1).

We add two experiments to support this. First, we change only which tier receives the GPTQ error feedback:

Error feedback goes toTarget PPLDraft PPLAgreement
8-bit tier (joint code selection)5.4726.5090.1%
Both, averaged (MatGPTQ)5.5675.81392.5%
4-bit tier, 8-bit closed form (BitNest)5.5125.59296.0%

LLaMA-2-7B; rotation, A8, group size and calibration fixed. FP16 WikiText-2 PPL is 5.471 (Table 2). Agreement is the draft's teacher-forced top-1 agreement with the target on GSM8K.

Giving the feedback to the 8-bit tier improves the target by only 0.04 PPL over BitNest, but degrades the draft by 0.9 PPL. Giving it to the 4-bit tier costs the target only 0.04 PPL. Averaging it, as MatGPTQ does, leaves both tiers under-compensated. Reweighting the code-selection objective does not help: with the feedback on the 8-bit tier, λ4∈{0,1,16}\lambda_4 \in \{0, 1, 16\} keeps the draft PPL at 6.50–6.56.

Second, we compare the joint and base-first constructions on LLaMA-3-8B, both storing one W8 copy and changing only the construction:

ConstructionTarget / Draft GSM8KAgreement % (wiki2 / GSM8K)Proj. speedup (wiki2 / GSM8K)
Joint (MatGPTQ)46.3 / 40.383.6 / 90.81.10 / 1.27×
Base-first (BitNest)49.8 / 41.491.0 / 94.21.27 / 1.36×

Projected speedup over W8A8 AR uses E/(γc+1)E/(\gamma c + 1), with γ=4\gamma = 4 and measured c=0.57c = 0.57.

The joint construction weakens both tiers. MatGPTQ was designed for multi-precision deployment rather than speculation, and on all four models we evaluated it trails BitNest in target PPL, draft PPL and agreement. Storing two separately quantized GPTQ models (a W4 draft and a W8 target) avoids this coupling and reaches 50.0 / 41.4 GSM8K and 90.5 / 94.0% agreement (Table 5(b)), but it needs 1.5× the weight memory (9.38 vs. 6.22 GiB on LLaMA-2-7B, Sec. 4.4). Of these options, only base-first obtains a W8-level target and a GPTQ-W4-level draft from a single stored copy. Its agreement even exceeds that of the separate pair, because the target's high nibble is bit-identical to the draft.

We will add these comparisons, a discussion of progressive and residual quantization, and the rotation training setting to the revision.

AuthorsOpenReview ↗

Weakness 1 and Question 1 (part b)

Methodological advance beyond quantized self-speculative decoding.

Beyond the quantizer (Part A), we clarify the methodological advance of BitNest as a whole. As stated in our contributions (Sec. 1), BitNest introduces Bit-Nested Speculative Decoding, a self-speculative paradigm in which the draft and the target are co-encoded as nested bit planes within a single physical representation. Prior quantization-based self-speculation either constrains the target to a shared low-bit representation (QSpec), or nests only the KV cache while keeping a separate low-precision copy of the draft weights (QuantSpec) (Sec. 2.2). Rotation-based PTQ optimizes individual quantized configurations rather than organizing multiple precision views into a nested representation (Sec. 2.3).

Three components realize the nesting, and each is designed for it:

  1. Base-first construction (Sec. 3.2). It makes the base a strong draft (Part A).
  2. Dual-plane weight storage (Sec. 3.3, Figure 3). It packs the base and the refinement into two independently accessible planes. Because memory is byte-addressable, storing both in one byte would make the draft fetch the refinement bits and lose its bandwidth advantage: if the draft read contiguous INT8 weights and truncated them on the fly, it would cost as much as the target (c≈1(c \approx 1), and the projected speedup on LLaMA-3-8B would fall to 0.83–0.89×.
  3. Context-aware KV nesting (Sec. 3.4, Figure 4, Appendix A.2). It applies the same base-plus-refinement code to the KV cache. Each entry is stored once, and the draft reads it as KV4 while verification reads it as KV8. KV states generated during drafting are recomputed during verification; accepted entries are retained and the rejected suffix is discarded. Short contexts use KV8 for both stages, and long contexts switch to KV4 drafting. To make the 4-bit key plane usable, our implementation applies an online per-head Hadamard rotation to queries and keys, which leaves attention scores exactly unchanged but flattens key outliers; without it, the KV4 draft's agreement on Qwen2.5-7B falls to 0.64 at 16K, and with it, it stays at 0.86–0.92 up to 32K.

To our knowledge, BitNest is the first self-speculative method in which one stored copy of both the weights and the KV cache serves both the draft and the target. Figure 5 and Secs. 4.3–4.4 show the effect at long context:

Method (LLaMA-2-7B-32K)4K32KPeak mem. at 32K (GiB)
BitNest, KV8 draft → KV8 target1.57×1.38×21.0
BitNest, KV4 draft → KV8 target1.53×1.66×21.0
QuantSpec (KV nesting + W4 draft copy)1.15×1.20×25.8
QSpec (shared W4 weights)1.42×1.11×24.0

Speedup over each method's own 16-bit AR in the same engine (Figure 5(a)); decode-time peak memory. QuantSpec uses its best γ and residual window per column.

When the draft reads the full KV8 cache, the speedup fades as KV traffic grows (1.57→1.38×). Reading only the KV4 base during drafting raises it to 1.66× at 32K from the same storage, at the lowest peak memory. At standard contexts, BitNest achieves 1.48–1.61× over 16-bit AR across four models and six workloads (Table 1). Its target stays close to FP16 quality: on LLaMA-3-8B it reaches 49.8 GSM8K (FP16: 50.1), vs. 37.0 for QSpec's W4A16 target (Table 2). At 2K context with a unified 64-token output, BitNest also has the highest absolute decode throughput against QSpec and QuantSpec in all 12 model–domain cells.

We will state these contributions more explicitly in the Introduction and Method sections, and add the rotation details to Appendix A.

AuthorsOpenReview ↗

Weakness 2 and Question 2

W2&Q2. Missing comparison with trained-drafter methods (Speculative Streaming, DFlash).

Thank you for this excellent question and the helpful suggestion. Our paper discusses DFlash-style auxiliary drafters in the Introduction, Background (Sec. 2.1) and Limitations (Sec. 6), noting that they may accelerate more when the extra draft-model overhead is acceptable. One distinction matters here: DFlash's drafter is not an autoregressive LM but a block-diffusion LM that proposes a whole block of tokens in one parallel forward pass, whereas BitNest, like other self-speculative methods, drafts autoregressively. Its speed advantage on short inputs therefore comes from a different drafting mechanism, orthogonal to how BitNest obtains its draft.

Setup. All methods run on one RTX A6000 with batch size 1 and greedy decoding, and each speedup is measured against 16-bit AR in the same engine. DFlash uses the official code and LLaMA-3.1-8B-Instruct drafter, in HF and vLLM. Speculative Streaming only provides Qwen3 code, so we ported it to LLaMA-3.1-8B-Instruct and trained it in lossless mode (≈5.6 GPU-hours). BitNest uses LLaMA-3-8B for short inputs (Tables 1, 3) and LLaMA-2-7B-32K for long inputs (Fig. 5).

Short inputs. On GSM8K (≈110-token prompts, 256 generated tokens):

MethodExtra paramsTrainingDecode tok/s (÷ own AR)Peak mem. (GiB)
16-bit AR0–37.8 / 39.115.03 / 15.50
Speculative Streaming12.6Mper target56.7 (1.49×)15.26
DFlash1.05B (≈13% of target)per target125.2 (3.27×)17.00
BitNest0PTQ only60.5 (1.55×)9.13

DFlash is the fastest here, as Sec. 6 anticipates, and vLLM gives the same 3.27×. BitNest outperforms Speculative Streaming, requires only post-training quantization, and uses 54% of DFlash's memory.

Memory under a fixed budget. Besides its 1.95 GiB of drafter weights, DFlash caches target hidden states and the drafter's KV for every context token (≈240 KiB per token, versus 128 KiB for the target's own KV), so its overhead over AR grows from 2.2 GiB at 1K to 9.5 GiB at 32K. We measured peak memory on PG-19 prefixes of increasing length (64 generated tokens) and the longest context that fits in 16 GiB:

Peak GiB1K4K16K32KMax context under 16 GiB
16-bit AR15.2015.9218.7922.614K
Speculative Streaming16.5822.09––none (OOM at 1K)
DFlash17.3918.8124.4932.06none
BitNest8.949.4611.9015.1932K

Speculative Streaming prefills four parallel streams; we use SDPA, which favors it over its official eager attention. BitNest beyond its native 8K window is measured for capacity only. Under 16 GiB, neither auxiliary method fits even a 1K context, while BitNest serves 32K. On a Jetson Orin NX 16GB, BitNest likewise extends LLaMA-2-7B-32K from 2K to 10K context (Sec. 4.5). The same holds for serving: in vLLM with a 42.2 GiB budget, DFlash reduces KV capacity by 23.5%, lowering the maximum concurrency at 2K from 100.9 to 77.2 sequences, whereas BitNest's KV8 plane (64 KiB per token) would hold about 2.5× as many sequences as 16-bit AR (projected).

Long inputs. We then measured the speedup on PG-19 prefixes from 1K to 32K (10 prefixes per length, 256 generated tokens). For DFlash in vLLM, we report the best of inference block sizes 4, 6, 8 and 10 at each length:

Context1K2K4K8K16K32K
DFlash (vLLM, best block)1.87×1.94×1.38×1.23×1.15×1.01×
DFlash (official HF)1.78×1.70×1.14×0.85×0.72×0.56×
BitNest1.63×1.59×1.57×1.51×1.56×1.66×

BitNest drafts with KV8 up to 4K and KV4 beyond; 1K and 2K are new, the rest are from Fig. 5(a). DFlash's gain shrinks as the context grows, and on these inputs it accepts only 1.8–2.6 tokens per verification round. BitNest drafts with the target itself and accepts 4.4–4.8 tokens per round at every length (γ=4). It also keeps no state beyond the target's KV cache, and its draft reads only the 4-bit KV base (Sec. 3.4). As a result, BitNest holds 1.51–1.66× throughout and overtakes DFlash from 4K onward in both engines.

Summary. BitNest needs no auxiliary drafter or extra parameters, only post-training quantization. On short inputs it uses 54% of DFlash's memory; under a 16 GiB budget it serves 32K context where neither auxiliary method fits 1K; and it holds 1.51–1.66× up to 32K, overtaking DFlash from 4K. We will add these results to Sec. 2.1 and the experiments. Sec. 6 will discuss diffusion-based drafters such as DFlash as future work: their parallel block drafting is orthogonal to BitNest, and extending BitNest to them while keeping its nested weights and KV cache is promising for memory-bound devices.

AuthorsOpenReview ↗

Question 3

Q3. Unclear attribution of the speedup between speculation and quantization. Thank you for this question. We now report speedups over autoregressive decoding of the same nested W8A8 model (W8A8 AR). W8A8 AR uses the same weight planes, the same kernels and the same engine as BitNest, but no draft. The ratio between BitNest and W8A8 AR therefore contains no quantization or kernel difference, and it isolates the gain due to speculation alone.

Decode tok/s, GSM8K, 2,048-token prompts, 256 generated tokens, batch 1, RTX A6000. The totals match the GSM8K column of Table 1.

ModelFP16 ARW8A8 ARBitNestQuantization (W8A8 AR ÷ FP16)Speculation (BitNest ÷ W8A8 AR)Total
LLaMA-2-7B41.151.465.21.25×1.27×1.59×
LLaMA-3-8B39.147.260.51.21×1.28×1.55×
Qwen2-7B41.150.162.91.22×1.26×1.53×
Qwen2.5-7B41.650.965.71.22×1.29×1.58×

RTX A6000, 2K context, all six workloads: quantization contributes 1.20–1.25×, and speculation contributes 1.21–1.31×.

Jetson Orin NX 16GB (Table 4): on LLaMA-2-7B-32K, W8A8 AR reaches 7.8 tok/s versus 6.0 for FP16 AR (~1.30×), and BitNest reaches 9.0–9.3 tok/s (~1.50×). Speculation therefore adds 1.15–1.19× on top of quantization.

Kernel optimizations: they do not inflate the baseline. Our FP16 AR runs at 39–42 tok/s, on par with vLLM (38–43 tok/s) and gpt-fast (42–47 tok/s) in the same setting. Kernel improvements speed up both configurations alike: in Appendix A.6, the optimized GEMV kernel raises W8A8 AR from 6.0 to 7.8 tok/s and BitNest from 7.4–7.9 to 9.0–9.3 tok/s.

The core of BitNest is that the quantization gain and the speculation gain stack, and both come from one stored model. A separate W4 draft on top of the W8 target would need 9.38 GiB instead of 6.22 GiB on LLaMA-2-7B (Sec. 4.4). We will add the W8A8 AR column to Table 1.

We thank the reviewer again for the constructive feedback. If our responses have addressed your concerns, we would be grateful if you would consider raising your score. We are happy to provide any further details.

Reviewer_X45MOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper proposes BitNest, a bit-nested self-speculative decoding framework that embeds a low-precision draft representation into a higher-precision target representation. Specifically, the method first constructs a W4 base using GPTQ and then quantizes the residual error into an additional 4-bit refinement plane, yielding a W4 draft and W8 target from the same physical weight storage. The authors implement dual-plane storage and customized kernels to avoid fetching refinement bits during drafting. The draft KV cache can also use 4-bit under long-context scenarios.

Critical strengths

The bit-level disaggregation for draft and target model is innovative and effective. The dual-plane storage design is also well motivated: storing the base and refinement separately allows the draft to avoid fetching the refinement bits. The experiments are generally extensive and informative, covering multiple key aspects such as acceleration performances, memory consumption, task evaluation, etc. The presentation is clear and readable, especially for the figures.

Critical weaknesses
  1. The experiments of BitNest are conducted under W4A8 → W8A8, whereas the original model weights are typically in BF16. Therefore, BitNest inevitably introduces additional quantization-related task-performance degradation compared with the original, unquantized model.
  2. The comparison of BitNest against other baselines is somehow unfair, as the quantization strategies are different (e.g., QSpec W4A4 v.s. BitNest W4A8). I think BitNest is adaptable to various strategies in principle, while the experiments are missing.
  3. Section 3.4 states that the KV caches of the draft and target models use 4-bit and 8-bit quantization, respectively, thereby reducing memory overhead. However, it is unclear how the verification process operates when the target model uses W8A8 while the KV cache is quantized to 4 bits.
Questions for authors
  1. Is BitNest applicable to other quantization strategies than W4A8 and W8A8 for draft and verification? The comparison with other baselines could be more convincing if the quantization methods are the same. Furthermore, the autoregressive baseline should also use the quantized model rather than bf16.
  2. Please clarify how the 4-bit KV cache is consumed during verification and whether any dequantization or additional computation is required.
5 comments
AuthorsOpenReview ↗

Weakness 1 and Question 1(b)

W1. Accuracy degradation introduced by quantization.

Thank you for raising this question. BitNest is designed for memory-constrained deployment, such as edge devices, where quantization is already the standard practice rather than an option. For example, LLaMA-2-7B needs 13.5 GB for its BF16 weights, which do not fit on an 8 GB Jetson Orin NX, whereas its 8-bit version needs 6.7 GB and its 4-bit version about 3.5 GB. Accordingly, llama.cpp, a widely used framework for on-device inference, provides more than 20 quantization formats from about 1.5 to 8 bits per weight. Speculative decoding can further accelerate decoding, but existing methods usually require an extra draft model, which adds memory exactly where memory is scarce. BitNest integrates quantization into speculative decoding: the draft is the 4-bit high plane of the target's own 8-bit weights, so a single quantized model fits within the memory budget and is also accelerated. In this setting, our target remains nearly equivalent to BF16. On all six domains of Table 2 (GSM8K: accuracy; others: PPL):

Wiki2 ↓GSM8K ↑Code ↓ShareGPT ↓LongDoc ↓PG-19 ↓
LLaMA-2 BF165.47113.62.6072.2214.3349.659
BitNest5.51213.82.6122.2324.3549.762
LLaMA-3 BF166.13950.12.7132.6465.25911.836
BitNest6.22649.82.7332.6645.31211.948
Qwen2 BF167.13479.42.2112.3285.24813.765
BitNest7.17677.62.2272.3405.27813.863
Qwen2.5 BF166.85182.92.0982.3965.51012.881
BitNest6.90682.02.1122.4065.53812.998

PPL stays within 1.4% of BF16 on every model and domain. On OpenVLA-7B, task success is 89% vs. 86% for BF16. An 8-bit target is therefore already sufficient for near-lossless quality. BitNest could also nest the draft in a higher-precision target, such as 16 bits, but this would double the memory budget with little room left for quality gains. On top of this target, our speculative acceleration is lossless: every drafted token is verified, so the output is exactly the target's greedy output, �0��0�. In the revision, we will clarify that BitNest is lossless with respect to its W8A8 target, whose quality is nearly equivalent to BF16.

Q1. Breakdown Speedup comparison with quantized baseline than FP16. Thank you for this constructive suggestion. Below,we report speedups over autoregressive decoding of the same nested W8A8 model (KV8), which uses the same weight planes, kernels and engine, with no draft. The gain of BitNest over FP16 therefore splits into quantization (W8A8 AR ÷ FP16 AR) and speculation (BitNest ÷ W8A8 AR):

SettingFP16 AR (tok/s)W8A8 AR (tok/s)BitNest (tok/s)Quant.Spec.
LLaMA-2-7B41.0–41.551.4–51.562.5–66.11.24–1.25×1.21–1.29×
LLaMA-3-8B38.8–39.246.9–47.558.4–62.21.20–1.22×1.25–1.31×
Qwen2-7B41.1–41.649.3–50.161.8–63.71.20–1.22×1.23–1.28×
Qwen2.5-7B41.5–41.850.3–51.363.5–65.91.20–1.24×1.26–1.29×

The decode throughput result was tested on A6000, GSM8K, 2K-token prompts; across all six domains, speculation gives 1.21–1.31×. Speculation contributes at least as much as quantization. As noted in Q1, the two gains stack from a single stored model. We will add this decomposition to the main text.

AuthorsOpenReview ↗

Weakness 2 & Question 1(a)

W2&Q1. Unfair comparison due to different quantization strategies with baseline.

Thank you for raising this point. We believe there may be a misunderstanding, which we clarify below. To our knowledge, BitNest is the first self-speculative decoding method that nests the draft inside the target's own quantized weights and KV cache, so that a single stored model serves as both. QSpec, which you mentioned, is the most related work, but its mechanism is different: it keeps one set of 4-bit weights and only switches activation precision, using a W4A4 draft and a W4A16 target, while its weights are not nested and its KV cache stays in FP16.

This difference matters for comparison. In self-speculation, the draft is cheaper only because it runs at a lower precision than the target, so the draft–target precision gap is the very source of the speedup. QSpec places this gap on activations, whereas BitNest places it on weights (W4 vs. W8 planes) and, at long context, also on the KV cache (KV4 vs. KV8 planes). Forcing both methods into one precision setting would remove the gap that one of them is built on, so the comparison would no longer evaluate the original methods. We therefore run each method at its published setting (our QSpec run reproduces its reported 1.24×) and compare what each delivers in speed, memory and target quality. On LLaMA-2-7B-32K with the same PG-19 prompts:

ContextFP16 AR (tok/s)QSpec (tok/s)BitNest (tok/s)QSpec (GiB)BitNest (GiB)
4K38.756.359.36.99.2
8K34.045.651.29.410.8
16K27.934.543.614.514.1
32K20.724.134.324.021.0

BitNest is faster at every context length, and its advantage grows from 1.05× at 4K to 1.42× at 32K (Figure 5(a)). As we mentioned in the paper (Sec. 4.4 and Figure 5(b)), QSpec uses less memory at short contexts because its target itself uses 4-bit weights. As the context grows, the KV cache dominates memory, and BitNest's nested KV cache makes it the more memory-efficient method from 16K onward.

QSpec's W4 target, however, comes at a cost in quality, which brings us to your question on other precisions. You are right that BitNest is not limited to a specific bit combination: a bWb_W-bit base plane and an rWr_W-bit refinement plane form a (bW+rW)(b_W+r_W)-bit target (Sec. 3.1). We choose W4/W8 because the degradation is negligible (W1) and both formats are widely supported by inference systems and hardware kernels. On LLaMA-3-8B, using the same tokens for all rows (accept: range over five domains; QSpec GSM8K from Table 2):

Pair (draft↔target)Target PPL / GSM8KDraft PPLAccept (%)
W4A8↔W8A8 (paper)6.226 / 49.86.47691.0–94.3
W4A16↔W8A166.216 / 48.96.47091.0–94.4
W4A4↔W8A86.230 / 48.38.66075.7–85.7
QSpec: W4A4↔W4A166.650 / 37.08.52078.8–87.9

All three BitNest pairs keep an accurate target. Their GSM8K differences (at most 1.5 points) are within one standard error of a difference between two accuracies near 50% on 1,319 questions (≈2.0 points). The W4A4↔W8A8 row is also the closest controlled comparison with QSpec: with the same W4A4 draft, BitNest reaches comparable acceptance while its target is far more accurate (48.3 vs. 37.0 GSM8K). Since a W4A4 draft reads the same weight bytes as a W4A8 draft in bandwidth-bound decoding, it only lowers acceptance, which makes W4A8↔W8A8 the most effective pair on current hardware.

AuthorsOpenReview ↗

Weakness 3 and Question 2

W3&Q2. Unclear how the kv4 operating during the inference.

Thank you for this question. We believe there is a misunderstanding: our W8A8 target uses an 8-bit KV cache, not a 4-bit one, so verification never uses the 4-bit KV view. We are sorry that we did not make this clear in the paper.

Specifically, BitNest maintains only one KV cache, and we apply the same 4/8 nesting to the KV cache as to the weights, as shown in Figure 4. Each KV entry is "stored once as nested base and refinement planes and can be accessed either as KV4 or KV8" (Appendix A.2): x^draft=q4s\hat x_{\mathrm{draft}}=q_4s and x^target=(16q4+qr) s/16\hat x_{\mathrm{target}}=(16q_4+q_r)\,s/16.

When the draft model generates tokens, it reads only the higher 4-bit plane of the KV cache, while its newly generated activations are 8-bit. That is why our draft model is W4A8 with 4-bit KV. The generated tokens are then sent to the verifier, i.e., the target model. During verification, the target model loads the full 8-bit KV cache (both planes), computes the new KV states with its W8A8 path, and writes them into the nested cache. That is why our target model is W8A8 with 8-bit KV. As stated in Sec. 3.4, "KV states generated for candidate tokens during drafting are kept temporarily; during verification, the target recomputes these states, retaining the accepted entries and discarding the rejected suffix." We therefore keep only the KV cache that has passed verification.

The target must keep KV8. As shown in Sec. 5.2 (Table 11(a)), a KV4 target lowers Qwen2.5 GSM8K accuracy from 81.6% to 78.4%, even though acceptance stays at 0.933, showing that "high draft–target agreement alone does not guarantee target fidelity". KV4 is therefore used only on the speculative path, never by the target.

As for dequantization, no separate pass or additional memory traffic is required. Dequantization is fused into our flash-decoding attention kernel: the two 4-bit planes are unpacked in registers, and the per-token scale is applied inside the attention computation. Since the two planes together occupy 8 bits per element, verification reads the same number of bytes as a standard INT8 KV cache, while a KV4 draft reads only half of them.

This saving pays off at long context. As explained in Sec. 3.4, "as the context grows, KV-cache traffic accounts for an increasing fraction of decoding cost, reducing the benefit of weight-side savings alone." We therefore compare a KV8 draft with a KV4 draft on LLaMA-2-7B-32K. Both share the same nested cache and the same KV8 verification, so memory is identical and only the draft's KV view differs:

ContextKV8 draft (tok/s)KV4 draft (tok/s)KV4 / KV8
4K60.659.30.98×
8K49.551.21.03×
16K40.643.61.07×
24K33.137.81.14×
32K28.534.31.20×

At short context, the KV cache is a small part of the bytes read per step, so a KV4 draft saves little and its slightly lower acceptance makes it marginally slower. As the context grows, KV reads dominate decoding cost, and the KV4 draft becomes increasingly faster, reaching 1.20× over the KV8 draft at 32K. This is why "at short contexts, BitNest uses KV8 for both drafting and verification," and switches to KV4 drafting at long contexts (Sec. 3.4). We will make this clearer in Sec. 3.4 and Figure 4.

We thank you again for the careful review. To summarize: (W1) BitNest targets memory-constrained deployment, where quantization is already standard practice; its W8A8 target stays nearly equivalent to BF16 (PPL within 1.4% on all models and domains), and the speculative acceleration on top of it is lossless. (W2&Q1) BitNest and QSpec differ in mechanism, so forcing them into one precision setting would remove the source of one method's speedup. At their published settings, BitNest is faster at every context length, more memory-efficient from 16K onward, and keeps a far more accurate target; with the same W4A4 draft, its target still reaches 48.3 vs. 37.0 GSM8K. BitNest is also not limited to W4/W8. Over the same W8A8 model decoded autoregressively, speculation alone adds 1.21–1.31× on top of the 1.20–1.25× from quantization, and both gains come from a single stored model. (W3&Q2) Our target always uses an 8-bit KV cache. BitNest keeps a single nested KV cache, retains only verified entries, and dequantizes on the fly with no separate pass. We will incorporate these clarifications and results into the revision. If our responses have addressed your concerns, we would be grateful if you would consider raising your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the Area Chair and Reviewers

We sincerely thank Area Chair GsKR for actively facilitating the discussion and encouraging constructive engagement, and all reviewers for their thoughtful feedback. Below, we summarize BitNest’s contributions and the additional evidence provided during rebuttal, identifying the concerns each addresses.

1. A single nested representation provides a strong draft and a higher-quality target.

BitNest first optimizes a GPTQ-W4 base, fixes it, and encodes a residual refinement to construct the W8 target. Independently accessible planes let drafting read only the base and verification read both, without an auxiliary draft-weight copy.

To address Reviewer ZHzy’s concerns about novelty and the relationship to progressive/residual quantization, we add controlled comparisons with target-first construction and MatGPTQ. Across four models, BitNest achieves lower target and draft perplexity than MatGPTQ and higher draft–target agreement. On LLaMA-3-8B, target/draft GSM8K accuracy reaches 49.8% / 41.4%, compared with 46.3% / 40.3% for MatGPTQ. These results support the importance of optimizing the low-bit base before fitting the refinement.

2. Speculative acceleration remains substantial after isolating quantization gains.

The original evaluation reports 1.48–1.61× decoding speedup over each engine’s 16-bit AR baseline across four models and six workloads. To address Reviewers X45M, ZwAF, and ZHzy’s questions about the source of this acceleration, we compare against the same nested W8A8 target running autoregressively, retaining its weight storage, kernels, and inference engine. Speculation alone contributes a further 1.21–1.31×.

We also address Reviewer ZwAF’s concern about output-length consistency through a unified 64-token comparison, reporting absolute throughput alongside within-engine speedups. BitNest achieves the highest measured throughput among the evaluated QSpec, QuantSpec, and BitNest configurations in all twelve LLaMA model–workload combinations.

3. Sustained long-context acceleration with shared weight and KV storage.

BitNest extends nesting to the KV cache: long-context drafting reads the KV4 base, while verification retains KV8. This preserves the draft’s bandwidth advantage as context grows. On LLaMA-2-7B-32K, BitNest achieves 1.66× speedup at 32K and reduces peak memory from 30.6 to 21.0 GiB relative to BitNest with BF16 KV.

To address Reviewer ZHzy’s request for comparisons with DFlash and Speculative Streaming, we add throughput, memory, and capacity measurements. On the evaluated PG-19 prefixes, BitNest maintains 1.51–1.66× speedup from 4K to 32K, while DFlash’s best evaluated vLLM configuration decreases from 1.38× to 1.01×. The memory measurements further characterize the overhead of trained drafting. Model differences and measurement protocols are explicitly identified.

These results highlight BitNest’s combined contribution: sustained long-context acceleration and efficient memory use, without auxiliary draft parameters. Real Orin NX16 measurements further demonstrate approximately 1.5× decoding speedup and extend executable context from 2K with FP16 to 10K.

4. Additional experiments clarify precision choices, KV execution, and numerical behavior.

For Reviewer X45M’s concerns about precision settings and KV usage, we add activation-precision variants and QSpec comparisons, and clarify KV reconstruction, verification-time cache updates, and fused dequantization.

For Reviewer ZwAF’s question about differing GSM8K scores, we add repeated-run measurements and fixed-configuration controls. With target execution held fixed, AR through the verification pass and speculation with either KV4 or KV8 drafting produce identical token sequences on 500/500 examples for both evaluated models.

For Reviewer ZwAF’s OpenVLA concern, we clarify its role as a transferability experiment, report draft acceptance, and explicitly retain the current prefill-related end-to-end limitation.

We appreciate the Area Chair’s coordination and the reviewers’ careful consideration. Please let us know whether any questions or concerns remain, or whether further evidence would help clarify a specific point. We would be happy to respond. All of these results are included in the revised version of our paper and highlighted in blue. Thank you again for your engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.