- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes BitNest, a self-speculative decoding framework that constructs a low-precision draft and a higher-precision target within a shared weight representation. The method stores base and refinement codes in independently accessible planes, extends this design to the KV cache, and evaluates inference quality, speed, memory usage, and edge-device performance.
Critical strengths
-
The representation and storage designs fit the stated memory-efficiency objective: drafting can access the base without loading refinement bits, while verification uses both planes.
-
The evaluation includes actual edge hardware and reports energy and memory alongside decoding speed, providing useful deployment evidence beyond a desktop-GPU benchmark.
Critical weaknesses
-
The methods use different draft–target precisions, and QuantSpec generates 64 tokens whereas the default evaluation generates 256, so the reported speedup ranking also reflects different operating conditions.
-
In the OpenVLA experiment, speculative decoding reduces decode time but increases total per-action latency from 201.5 ms to 212.7 ms, illustrating a material prefill limitation of the current implementation
Questions for authors
-
Could the authors report BitNest’s speedup over autoregressive decoding with the same nested W8A8 target across all main workloads, separating quantization gains from speculative-decoding gains?
-
Could the comparisons use a common output length and include absolute latency or throughput, in addition to speedups normalized within each inference engine?(“QuantSpec follows its official 64-token setting”)
-
The authors conclude that using KV4 only for drafting while retaining KV8 for verification better preserves target fidelity than using KV4 for both stages. However, Table 11 reports different GSM8K accuracies for W8A8+KV8 autoregressive decoding (82.4%), KV8-draft→KV8-target speculation (82.2%), and KV4-draft→KV8-target speculation (81.6%); given the stated default of greedy decoding, could the authors explain these differences?
6 comments
Question 2 and Weakness 1
Q2&W1. Mismatched output lengths and missing absolute throughput.
Thank you for this constructive suggestion. QuantSpec was evaluated at 64 generated tokens because its public code hits a device-side assert at 128 or more generated tokens in our setup, so it cannot be extended to 256. We therefore re-ran BitNest at QuantSpec's official 64-token setting, on the same 2,048-token input prompts and the same GPU. The table reports both views at this common length. The first three columns give absolute decode throughput in tok/s. The last two give the speedup normalized within each engine, i.e., over that engine's own FP16 autoregressive decoding:
| Model / domain | BitNest (tok/s) | QuantSpec +wq (tok/s) | BitNest ÷ QuantSpec | BitNest ÷ own FP16 | QuantSpec ÷ own FP16 |
|---|---|---|---|---|---|
| LLaMA-2 wiki2 | 61.7 | 54.3 | 1.14× | 1.49× | 1.16× |
| LLaMA-2 GSM8K | 63.3 | 54.9 | 1.15× | 1.57× | 1.17× |
| LLaMA-2 code | 62.8 | 57.5 | 1.09× | 1.51× | 1.23× |
| LLaMA-2 ShareGPT | 64.1 | 58.1 | 1.10× | 1.55× | 1.24× |
| LLaMA-2 longdoc | 64.3 | 55.9 | 1.15× | 1.56× | 1.19× |
| LLaMA-2 PG-19 | 60.2 | 50.9 | 1.18× | 1.46× | 1.09× |
| LLaMA-3 wiki2 | 55.6 | 44.9 | 1.24× | 1.40× | 1.08× |
| LLaMA-3 GSM8K | 57.6 | 50.2 | 1.15× | 1.46× | 1.21× |
| LLaMA-3 code | 60.6 | 48.8 | 1.24× | 1.53× | 1.17× |
| LLaMA-3 ShareGPT | 61.6 | 53.8 | 1.15× | 1.55× | 1.29× |
| LLaMA-3 longdoc | 58.2 | 46.1 | 1.26× | 1.47× | 1.11× |
| LLaMA-3 PG-19 | 56.5 | 45.2 | 1.25× | 1.42× | 1.09× |
Both views give the same ranking in every cell. In absolute terms BitNest is 1.09–1.26× faster than QuantSpec. Within each engine BitNest reaches 1.40–1.57× over FP16, against 1.08–1.29× for QuantSpec. The gap does not come from a faster inference engine on our side. QuantSpec's own FP16 baseline (gpt-fast, 41.5–46.9 tok/s) is faster than BitNest's (39.6–41.5 tok/s). BitNest thus starts from the slower 16-bit baseline and still ends with the higher speculative throughput. The difference comes from what each draft and target read per step. At batch size 1, decoding is bound by memory reads, and at a 2K context the weights dominate the bytes read. QuantSpec's target reads 16-bit weights, while BitNest's target reads 8-bit weights and its draft reads only the 4-bit base plane. Shortening the output does not change BitNest's behavior either. At 64 tokens BitNest remains 1.15–1.31× faster than its own W8A8 autoregressive decoding, matching the 1.21–1.31× measured at 256 tokens (Q1).This table will be added to the revision.
Question 1 and Weakness 2
Q1. Speedup should be measured against the same nested W8A8 autoregressive model. Thank you for this constructive suggestion. The reference below is BitNest's own W8A8 target decoded autoregressively. It uses the same stored weight planes, the same kernels and the same engine, and only the draft is removed, so the ratio isolates the speculative gain. All entries use 2,048-token input prompts and 256 generated tokens:
| Spec ÷ W8A8-AR | wiki2 | GSM8K | code | ShareGPT | longdoc | PG-19 |
|---|---|---|---|---|---|---|
| LLaMA-2-7B | 1.21× | 1.27× | 1.27× | 1.29× | 1.28× | 1.25× |
| LLaMA-3-8B | 1.25× | 1.28× | 1.31× | 1.31× | 1.29× | 1.25× |
| Qwen2-7B | 1.23× | 1.26× | 1.27× | 1.28× | 1.27× | 1.27× |
| Qwen2.5-7B | 1.26× | 1.29× | 1.29× | 1.29× | 1.29× | 1.27× |
The end-to-end gain over FP16 splits into quantization and speculation, which multiply:
| Model | Quantization (W8A8-AR ÷ FP16) | Speculation (Spec ÷ W8A8-AR) | Total (Spec ÷ FP16) |
|---|---|---|---|
| LLaMA-2-7B | 1.24–1.25× | 1.21–1.29× | 1.51–1.61× |
| LLaMA-3-8B | 1.20–1.22× | 1.25–1.31× | 1.49–1.59× |
| Qwen2-7B | 1.20–1.22× | 1.23–1.28× | 1.49–1.55× |
| Qwen2.5-7B | 1.20–1.24× | 1.26–1.29× | 1.52–1.59× |
Speculation contributes about as much as quantization, and both come from a single stored model. The measured speculative gain also matches what the acceptance rate predicts. With per-token acceptance α ≈ 0.94–0.95 and a draft/target step-cost ratio c ≈ 0.60–0.66, the analytic speedup is 1.21–1.27×, in line with the measured 1.21–1.31×.
W2. Limited end-to-end speedup on OpenVLA due to prefill overhead.
Thank you for this constructive comment. BitNest is designed for LLM generation. We include OpenVLA as a transferability test: it checks whether nested self-speculation carries over to a non-text domain, and whether it can help beyond LLM generation. Per-action breakdown on LIBERO (A6000, median over 100 frames, γ=7):
| Per action | Prefill (288 tok, ms) | Decode (6 tok, ms) | Total (ms) | Acceptance | Actions done in one round |
|---|---|---|---|---|---|
| FP16 AR | 65.9 | 135.6 | 201.5 | – | – |
| BitNest W8A8 AR | 111.7 | 105.3 | 216.9 | – | – |
| BitNest spec | 111.7 | 100.8 | 212.7 | 0.93 | 85 / 100 |
Prefill currently limits end-to-end latency. Each action needs a fresh, compute-bound 288-token prefill, and our implementation dequantizes the planes for this step. The 6 decode tokens per action also give speculation only a single round over which to amortize its cost.
The more important finding is that nesting transfers to action tokens. The W4A8 draft's acceptance is 0.93, on par with text generation. In 85 of the 100 actions, the 4-bit plane alone produces exactly the action of the 8-bit target. We believe this points to a promising direction for accelerating robot policies. In this setting, the nested planes need not be used for speculative decoding at all. They can instead serve as the foundation for dynamic bit-width selection, where the policy chooses 4 or 8 bits at each step from the same stored weights.
- Step-level precision switching. Most control steps do not need 8 bits, but some do: running the whole trajectory at 4 bits lowers LIBERO success from 86% to 80%. Easy steps can run on the 4-bit plane alone with no verification, at about 0.6 of the cost of an 8-bit step. Hard steps use the full 8-bit target. The draft–target agreement we measure is a natural signal for deciding which steps need 8 bits.
- No extra memory on the robot. Robots typically run on memory-constrained edge modules such as the Jetson Orin we evaluated. Unlike a separate draft model or a second checkpoint, BitNest changes precision without loading anything, because one stored model serves both levels.
- Decode-dominant VLAs. Policies that generate action chunks, or reason before acting, decode many tokens per prefill, so the decode gain carries through. With 140 action tokens per prefill, the W8A8 plane alone is already 1.24× faster than FP16 per chunk.
Combined with an int8 prefill GEMM that reads the planes directly, we see BitNest's nesting as a basis for dynamic-precision robot inference. In the revision, we will present OpenVLA as a transferability result with this end-to-end caveat, and add this discussion as future work.
Weakness 1
We sincerely thank the reviewer for the positive assessment, in particular for recognizing that the dual-plane design fits BitNest's memory-efficiency goal and that our edge-hardware evaluation provides deployment evidence beyond a desktop-GPU benchmark.
W1. Different draft–target precisions and output lengths across methods Thank you for this comment. We believe the differing precisions may have been read as an inconsistent setup, and we clarify this below; the output-length mismatch is answered under Q2&W1. The precisions differ because the methods themselves differ. A self-speculative method obtains its draft by lowering precision somewhere in the target, and where it lowers precision is that method's design:
--wqadds 4-bit draft MLP weights; without them it runs at 0.81–0.89× of FP16.Quality of the target being verified. Speculative decoding outputs exactly the target's tokens, so the target's precision sets the delivered quality. All rows are evaluated on the same tokens (QSpec GSM8K from Table 2):
BitNest's target stays within 0.09 PPL and 0.3 GSM8K points of FP16. QSpec's 4-bit-weight target loses 13 GSM8K points on LLaMA-3. QuantSpec's target keeps 16-bit weights, so its speedup comes at FP16 weight bandwidth.
-Absolute decode throughput. On identical prompts and hardware, BitNest exceeds QuantSpec by 1.09–1.26× in all 12 cells (Q2&W1).
-The speculative gain on its own. Speculation alone gives 1.21–1.31× over the same nested W8A8 target decoded autoregressively (Q1).