- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LeapQuant, a training-free method for quantizing recurrent states in linear-attention layers. It addresses recurrent quantization error by keeping a low-precision boundary state fixed within a window, buffering higher-precision token updates, and quantizing the reconstructed state only at the window boundary. To reduce the error of each boundary quantization, it represents a few dominant state components as high-precision Compensator Tokens and smooths the remaining residual before quantization. The authors evaluate the method on models from the Qwen, Kimi, and GLM families. They report accuracy close to FP32 with an INT8 recurrent-state representation, as well as kernel-level and end-to-end throughput improvements on several NVIDIA GPUs.
Critical strengths
- The paper clearly identifies two distinct sources of recurrent-state quantization degradation—error introduced at every update and outlier-driven quantization error—and proposes mechanisms that address each source. The ablations provide useful evidence that per-window quantization, Compensator Tokens, and smoothing contribute differently.
- The work evaluates both accuracy and systems performance across several model families and GPU types. The results suggest that state quantization can be practically useful for hybrid models with linear-attention layers, rather than only reducing a synthetic state-reconstruction metric.
Critical weaknesses
- The main accuracy and performance claims focus on 8-bit state quantization. The paper does not establish whether the method remains useful at more aggressive state precisions, which limits the strength of the broader claim that it enables low-precision recurrent-state serving.
- The reported speedups are sensitive to the baseline and workload. The paper emphasizes comparisons with FP32 state/kernel execution, while practical inference stacks may use BF16 and may have different state-management and batching costs. The end-to-end gains are more modest than the kernel-level gains, so the practical advantage should be characterized against stronger deployment-relevant baselines.
Questions for authors
- How does LeapQuant compare with a well-optimized BF16 recurrent-state implementation in end-to-end serving, using the same model, batching, and cache policy? Please separate gains from reduced state traffic from gains due to kernel or implementation differences.
- What is the accuracy–throughput trade-off at 4-bit or other lower state precisions, and does the Compensator Token plus smoothing design remain effective there?
- How sensitive are the results to window length across longer generations and different recurrence types? In particular, can the authors quantify the added buffering and boundary-reconstruction costs under realistic multi-request serving?
3 comments
Q3
- How sensitive are the results to window length across longer generations and different recurrence types? In particular, can the authors quantify the added buffering and boundary-reconstruction costs under realistic multi-request serving?
Response:
1. Sensitivity to window length.
As discussed in Section 4.4, shorter windows quantize the state more often and reduce accuracy, while longer windows bring no further gain but reduce efficiency, so we use . We verified that this conclusion holds across generation lengths and recurrence types.
Longer generations. We split the LiveCodeBench problems of Qwen3.5-9B (8 seeds) into three buckets by output length:
| Output length (mean) | 9.9K | 32.3K | 67.0K |
|---|---|---|---|
| FP32 | 97.4 | 75.3 | 28.1 |
| 96.7 | 70.7 | 26.9 | |
| 96.8 | 70.3 | 28.6 | |
| 97.1 | 74.1 | 29.2 |
Recurrence types. On Kimi-Linear-48B-A3B, / / achieve 65.0 / 66.3 / 68.3 on AIME, consistent with GDN.
2. Buffering and boundary-reconstruction costs under multi-request serving.
Our end-to-end results are measured in multi-request vLLM serving with continuous batching and up to 512 concurrent requests. Requests reach their window boundaries at different steps, so the reported throughput already includes both costs. At on Qwen3.5-9B:
- Buffering. The window buffer holds the buffered real-token updates, which takes 6.3 MB per request, compared with 52 MB for the FP32 state. Even in the worst case, where all 256 requests in the batch are at the end of their windows and read all buffered updates, reading the buffer adds less than 13% to the kernel latency, far below the speedup from quantization.
- Boundary reconstruction. At each decode step, about 1/16 of the requests reconstruct their state. As shown in Sec.3.3, the exposed kernel-level overhead is less than 7%, and the kernel remains 2.52× faster than FP32. As a comparison, in a simulated scenario where all requests reach the window boundary together, the amortized speedup goes up to 2.69x as there is no imbalance between different CTAs.
- End-to-end. With both costs included, LeapQuant improves end-to-end throughput over FP32 by 1.23–1.44× in realistic workloads including GSM8K, MMLU-Pro, and GPQA.
We thank the reviewer again for the helpful comments. We will incorporate the clarifications and additional results discussed above into the revision.
Dear Reviewer daei,
As the deadline for the discussion approaches, we wonder if you could kindly consider updating the score if our responses have addressed your concerns. If you have any remaining questions or need further clarification, we would be happy to discuss them with you.
Thank you once again for your valuable efforts in reviewing our paper.
Best regards,
The Authors
We thank the reviewer for the careful reading and the constructive feedback. We address each point below.
W1
Response:
Table 1 already evaluates LeapQuant at 6 and 4 bits. Averaged over the three models, LeapQuant reaches 72.4% at 6 bits, close to FP32's 75.5%, while the best baseline, TurboQuant, averages only 31.9%. At 4 bits, LeapQuant averages 60.4%, versus 9.7% for MXFP4 and 22.9% for TurboQuant.
We additionally include a new configuration at 4 bits. On Qwen3.5-9B:
With the new configuration, LeapQuant stays within 2.9% of FP32 on every task at 4 bits. These results show that LeapQuant remains effective under aggressive state quantization.
W2 + Q1
Response:
BF16 is not lossless. Storing the state in BF16 lowers the average accuracy in Table 1 from 75.5% to 68.7%, and Qwen3.5-9B's AIME accuracy from 87.9% to 72.1%, while LeapQuant matches FP32 (75.5%) with half as many bits. LeapQuant is therefore both more accurate and faster than a BF16 state.
As requested, we ran end-to-end serving experiments on realistic workloads, comparing LeapQuant with vLLM's native BF16 state on Qwen and GLM models. Both use the same scheduling policy (continuous batching with up to 512 sequences) and the same memory budget (0.75 GPU memory utilization), and every state format is stored at its actual size in vLLM's memory pool. Throughput (tok/s):
The gain is larger on GLM-5.3-Flash because it pairs linear attention with sparse attention instead of full attention, so linear attention takes a larger share of each decode step.
To separate the effect of reduced state traffic from implementation differences, we compare the 8-bit and BF16 GDN decode kernels under the same setup at batch size 256:
LeapQuant is 1.54× faster than the BF16 state and moves 2.34× fewer bytes. Its achieved bandwidth is slightly lower because each memory transfer in the kernel is smaller. The speedup therefore comes entirely from reduced state traffic, not from a more efficient implementation.
The end-to-end gain is smaller than the kernel-level gain because attention, MLP, and other layers are unchanged. We will add these comparisons to the revision.
Q2
Response:
We extended the ablation of Table 2 to 4 bits on Qwen3.5-9B as below.
Both Compensator Tokens and smoothing remain necessary at 4 bits. Neither recovers accuracy alone, and together they raise AIME from 0.0% to 58.1%.