acceptodds
Under review as a conference paper at ICLR 2027

LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

57 views

Abstract

Recent LLMs increasingly adopt hybrid designs that replace standard attention with linear attention, such as Gated DeltaNet (GDN) and Kimi Delta Attention (KDA). Although they compress the context into a fixed-size recurrent state and substantially reduce the cost of long-context processing, repeatedly reading and updating that state remains a major inference bottleneck. Quantization offers a natural way to reduce this cost, but can significantly degrade model quality, due to the accumulation of rounding errors and the presence of outlier rows and columns in the state. To address these challenges, we propose LeapQuant, a training-free method that achieves near-lossless performance under 8-bit recurrent-state quantization. First, to mitigate error accumulation, we propose per-window quantization, which leaps over a window of tokens and quantizes the state only once at its end. Within a window, outputs are computed from the fixed low-bit state together with high-precision buffered updates. Second, to reduce the error introduced by each quantization, LeapQuant retains the state's largest outliers as a few high-precision compensator tokens, which share the update path of real tokens. We then smooth the remaining residual before quantization to further reduce the error. Comprehensive experiments across the Qwen, Kimi, and GLM model families show that LeapQuant substantially reduces memory and compute costs during inference. With accuracy comparable to the FP32 baseline, it achieves average speedups of 2.05–3.70 at the kernel level and 1.47 for end-to-end inference on NVIDIA B200, RTX PRO 6000, and RTX 5090 GPUs.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Quantization, Linear Attention, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_daeiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LeapQuant, a training-free method for quantizing recurrent states in linear-attention layers. It addresses recurrent quantization error by keeping a low-precision boundary state fixed within a window, buffering higher-precision token updates, and quantizing the reconstructed state only at the window boundary. To reduce the error of each boundary quantization, it represents a few dominant state components as high-precision Compensator Tokens and smooths the remaining residual before quantization. The authors evaluate the method on models from the Qwen, Kimi, and GLM families. They report accuracy close to FP32 with an INT8 recurrent-state representation, as well as kernel-level and end-to-end throughput improvements on several NVIDIA GPUs.

Critical strengths
  1. The paper clearly identifies two distinct sources of recurrent-state quantization degradation—error introduced at every update and outlier-driven quantization error—and proposes mechanisms that address each source. The ablations provide useful evidence that per-window quantization, Compensator Tokens, and smoothing contribute differently.
  2. The work evaluates both accuracy and systems performance across several model families and GPU types. The results suggest that state quantization can be practically useful for hybrid models with linear-attention layers, rather than only reducing a synthetic state-reconstruction metric.
Critical weaknesses
  1. The main accuracy and performance claims focus on 8-bit state quantization. The paper does not establish whether the method remains useful at more aggressive state precisions, which limits the strength of the broader claim that it enables low-precision recurrent-state serving.
  2. The reported speedups are sensitive to the baseline and workload. The paper emphasizes comparisons with FP32 state/kernel execution, while practical inference stacks may use BF16 and may have different state-management and batching costs. The end-to-end gains are more modest than the kernel-level gains, so the practical advantage should be characterized against stronger deployment-relevant baselines.
Questions for authors
  1. How does LeapQuant compare with a well-optimized BF16 recurrent-state implementation in end-to-end serving, using the same model, batching, and cache policy? Please separate gains from reduced state traffic from gains due to kernel or implementation differences.
  2. What is the accuracy–throughput trade-off at 4-bit or other lower state precisions, and does the Compensator Token plus smoothing design remain effective there?
  3. How sensitive are the results to window length across longer generations and different recurrence types? In particular, can the authors quantify the added buffering and boundary-reconstruction costs under realistic multi-request serving?
3 comments

We thank the reviewer for the careful reading and the constructive feedback. We address each point below.

W1

  1. The main accuracy and performance claims focus on 8-bit state quantization. The paper does not establish whether the method remains useful at more aggressive state precisions, which limits the strength of the broader claim that it enables low-precision recurrent-state serving.

Response:

Table 1 already evaluates LeapQuant at 6 and 4 bits. Averaged over the three models, LeapQuant reaches 72.4% at 6 bits, close to FP32's 75.5%, while the best baseline, TurboQuant, averages only 31.9%. At 4 bits, LeapQuant averages 60.4%, versus 9.7% for MXFP4 and 22.9% for TurboQuant.

We additionally include a new configuration at 4 bits. On Qwen3.5-9B:

MethodAIMEGPQALCBMMLU
FP3287.981.364.183.3
Ours INT887.981.864.183.8
Ours MXFP685.879.859.581.7
TurboQuant 6-bit27.529.816.061.2
Ours MXFP458.165.734.080.2
Ours NVFP485.080.061.383.0
TurboQuant 4-bit3.326.813.047.4

With the new configuration, LeapQuant stays within 2.9% of FP32 on every task at 4 bits. These results show that LeapQuant remains effective under aggressive state quantization.

W2 + Q1

  1. The reported speedups are sensitive to the baseline and workload. The paper emphasizes comparisons with FP32 state/kernel execution, while practical inference stacks may use BF16 and may have different state-management and batching costs. The end-to-end gains are more modest than the kernel-level gains, so the practical advantage should be characterized against stronger deployment-relevant baselines.
  1. How does LeapQuant compare with a well-optimized BF16 recurrent-state implementation in end-to-end serving, using the same model, batching, and cache policy? Please separate gains from reduced state traffic from gains due to kernel or implementation differences.

Response:

BF16 is not lossless. Storing the state in BF16 lowers the average accuracy in Table 1 from 75.5% to 68.7%, and Qwen3.5-9B's AIME accuracy from 87.9% to 72.1%, while LeapQuant matches FP32 (75.5%) with half as many bits. LeapQuant is therefore both more accurate and faster than a BF16 state.

As requested, we ran end-to-end serving experiments on realistic workloads, comparing LeapQuant with vLLM's native BF16 state on Qwen and GLM models. Both use the same scheduling policy (continuous batching with up to 512 sequences) and the same memory budget (0.75 GPU memory utilization), and every state format is stored at its actual size in vLLM's memory pool. Throughput (tok/s):

ModelWorkloadBF16LeapQuant 8-bitvs. BF16
Qwen3.5-9BMMLU-Pro15074172441.14×
GPQA721781411.13×
Qwen3.5-35B-A3BMMLU-Pro10893124491.14×
GPQA623075041.20×
GLM-5.3-FlashMMLU-Pro506266431.31×
GPQA303341091.36×

The gain is larger on GLM-5.3-Flash because it pairs linear attention with sparse attention instead of full attention, so linear attention takes a larger share of each decode step.

To separate the effect of reduced state traffic from implementation differences, we compare the 8-bit and BF16 GDN decode kernels under the same setup at batch size 256:

Time per stepBytes moved
BF16 state0.0912 ms537 MB
LeapQuant 8-bit0.0592 ms229 MB

LeapQuant is 1.54× faster than the BF16 state and moves 2.34× fewer bytes. Its achieved bandwidth is slightly lower because each memory transfer in the kernel is smaller. The speedup therefore comes entirely from reduced state traffic, not from a more efficient implementation.

The end-to-end gain is smaller than the kernel-level gain because attention, MLP, and other layers are unchanged. We will add these comparisons to the revision.

Q2

  1. What is the accuracy–throughput trade-off at 4-bit or other lower state precisions, and does the Compensator Token plus smoothing design remain effective there?

Response:

We extended the ablation of Table 2 to 4 bits on Qwen3.5-9B as below.

MethodAIMELCBKernel
FP32 per-step87.964.11.00×
INT4 per-step0.00.04.12×
Per-window0.06.14.31×
Per-window + Comp. Tokens0.01.03.84×
Per-window + Smoothing0.02.34.27×
All58.134.03.61×

Both Compensator Tokens and smoothing remain necessary at 4 bits. Neither recovers accuracy alone, and together they raise AIME from 0.0% to 58.1%.

Q3

  1. How sensitive are the results to window length across longer generations and different recurrence types? In particular, can the authors quantify the added buffering and boundary-reconstruction costs under realistic multi-request serving?

Response:

1. Sensitivity to window length.

As discussed in Section 4.4, shorter windows quantize the state more often and reduce accuracy, while longer windows bring no further gain but reduce efficiency, so we use p=16p=16. We verified that this conclusion holds across generation lengths and recurrence types.

Longer generations. We split the LiveCodeBench problems of Qwen3.5-9B (8 seeds) into three buckets by output length:

Output length (mean)9.9K32.3K67.0K
FP3297.475.328.1
p=4p=496.770.726.9
p=8p=896.870.328.6
p=16p=1697.174.129.2

Recurrence types. On Kimi-Linear-48B-A3B, p=4p=4 / 88 / 1616 achieve 65.0 / 66.3 / 68.3 on AIME, consistent with GDN.

2. Buffering and boundary-reconstruction costs under multi-request serving.

Our end-to-end results are measured in multi-request vLLM serving with continuous batching and up to 512 concurrent requests. Requests reach their window boundaries at different steps, so the reported throughput already includes both costs. At p=16p=16 on Qwen3.5-9B:

  • Buffering. The window buffer holds the pp buffered real-token updates, which takes 6.3 MB per request, compared with 52 MB for the FP32 state. Even in the worst case, where all 256 requests in the batch are at the end of their windows and read all pp buffered updates, reading the buffer adds less than 13% to the kernel latency, far below the speedup from quantization.
  • Boundary reconstruction. At each decode step, about 1/16 of the requests reconstruct their state. As shown in Sec.3.3, the exposed kernel-level overhead is less than 7%, and the kernel remains 2.52× faster than FP32. As a comparison, in a simulated scenario where all requests reach the window boundary together, the amortized speedup goes up to 2.69x as there is no imbalance between different CTAs.
  • End-to-end. With both costs included, LeapQuant improves end-to-end throughput over FP32 by 1.23–1.44× in realistic workloads including GSM8K, MMLU-Pro, and GPQA.

We thank the reviewer again for the helpful comments. We will incorporate the clarifications and additional results discussed above into the revision.

Dear Reviewer daei,

As the deadline for the discussion approaches, we wonder if you could kindly consider updating the score if our responses have addressed your concerns. If you have any remaining questions or need further clarification, we would be happy to discuss them with you.

Thank you once again for your valuable efforts in reviewing our paper.

Best regards,
The Authors

Reviewer_s6arOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a novel recurrent state quantization approach for efficient linear attention. To mitigate error accumulation over per-step quantization, the authors introduced per-window quantization, which leaps over a window of tokens and quantizes the state only one at its end. Furthermore, to reduce the error incurred by outliers, the authors introduced high-precision compensator tokens, which greatly reduces the outliers of residuals. Finally, the authors incorporated smoothing to further reduce the outliers of residuals.

Critical strengths

The paper is very well-written, and I really enjoyed reading. The authors aim to address the problem to quantize recurrent states accurately, which is timely and important. The contributions introduced by the authors (i.e., per-window quantization, compensator tokens) are novel and seem to be effective.

Critical weaknesses
  1. For the experimental results, it seems that the authors only quantized recurrent state. However, on NPUs, weights and activations should be quantized together. Can the authors present the results with weight-activation quantization?
Questions for authors
  1. For the ablation results on r, why the accuracy drop occurs when r increases to 16?
3 comments

We thank the reviewer for the careful reading and the constructive feedback. We address each point below.

W1

  1. For the experimental results, it seems that the authors only quantized recurrent state. However, on NPUs, weights and activations should be quantized together. Can the authors present the results with weight-activation quantization?

Response:

We additionally evaluated LeapQuant on Qwen3.5-9B with W8A8 weight-activation quantization (mean ± std over 8 seeds):

BenchmarkW8A8 + FP32 stateW8A8 + LeapQuant 8-bit
AIME 202687.0 ± 1.587.9 ± 1.8
GPQA-Diamond81.8 ± 1.181.3 ± 0.8

The results show that LeapQuant remains near-lossless when combined with weight-activation quantization.

Q1

  1. For the ablation results on r, why the accuracy drop occurs when r increases to 16?

Response:

We ran the rank ablation on more models and benchmarks:

ModelTaskFP32r=4r=4r=8r=8r=16r=16
Qwen3.5-9BAIME87.987.988.187.9
GPQA81.381.881.581.8
LCB64.164.164.062.8
MMLU-Pro83.383.883.683.9
Qwen3.5-35B-A3BAIME91.591.091.291.7
GPQA84.783.984.484.3
LCB75.676.175.276.1
MMLU-Pro85.985.885.785.9

The drop at r=16r=16 only appears on LiveCodeBench for Qwen3.5-9B, which we attribute to the variance of long code generation. On all other tasks and on Qwen3.5-35B-A3B, every rank matches FP32, including r=16r=16 on LiveCodeBench for Qwen3.5-35B-A3B. In every setting, r=4r=4 stays within one point of both FP32 and the best rank, so our choice of r=4r=4 is unaffected.

We thank the reviewer again for the helpful comments. We will incorporate the clarifications and additional results discussed above into the revision.

Thanks for the authors' response.

If possible, please include W4A4 results in the revised manuscript, because most of LLM quantization works target W4 or below.

I have no further comments and maintain my positive rating.

Thanks for the suggestion. We additionally evaluated LeapQuant on Qwen3.5-9B with W4A4 weight-activation quantization (NVFP4, mean over 8 seeds):

BenchmarkW4A4 + FP32 stateW4A4 + LeapQuant 8-bit
AIME 202681.080.5
GPQA-Diamond76.677.0

The results show that LeapQuant remains near-lossless under 4-bit weight-activation quantization as well. We will include these results in the revised manuscript.

Reviewer_tEE8OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a method for training-free state-quantization for Linear Attention (LA) sequence mixers, focusing primarily on the application to Delta rule-based methods.

The proposed method, LeapQuant, is composed of three subcomponents: (1) windowed-quantization which effectively uses the chunk-wise decomposition of computing the forward pass through a LA mixer at inference time which avoids having to bear the additional quantization error that comes from repeatedly having to run the loop of load-dequantize-update-quantize at decode time which one typically has to when using the recurrent mode of LA; (2) the authors introduce compensator tokens which effectively are a set of quasi key-value pairs that captures the most energy in the state. The authors recognize that entries with large absolute value will widen the quantization scale and hence lead to overall worse quantization error. To address this the authors factor out r of these high-energy directions (approximated through power iteration) and quantize the state less those high energy directions; (3) instead of directly quantizing the residual state (the state less the compensators), they use a diagonal preconditioner that balances row mean absolute magnitudes to avoid having a wide quantization scale. The inverse of the preconditioner is then applied at dequantization time to restore the scale.

Critical strengths

The paper is a combination of techniques that work well together as shown by good empirical performance both in terms of low quantization error over long sequences and runtime speed-ups. The main contribution of the paper is exploiting the fact that one does not need to rely on running the fully recurrent kernels at decode time and can instead use the chunk-wise decomposition to reduce the quantization error incurred at decode time and combining this with the insight that one can remove high-energy components from the state prior to quantization to reduce the quantization scale, while still retaining speed-ups over unquantized baselines. I would consider the residual scaling as a lesser contribution since it applies smoothing/preconditioning methods, that are already well established in the general quantization literature [1], to the problem of state quantization.

[1] https://arxiv.org/pdf/2211.10438v7

Critical weaknesses

None

Questions for authors

I indicated that there are no critical weaknesses, however, I would like to address the following critiques here:

  • The authors write For state space models, Quamba (Chiang et al., 2025b) and MambaQuant (Yue et al., 2025) quantize the weights and activations, and Quamba2 (Chiang et al., 2025a) and Q-Mamba (Tianqi et al., 2025) further quantize the cached SSM state. These methods are specialized to Mamba’s selective SSM, whose state is updated differently from that of other linear attention architectures. This is too broad of a statement since e.g., Quamba2 is not specialized to only Mamba1 and in fact supports Mamba2, whose core recurrence is scalar-decayed LA. Could the authors clarify which components of the methods proposed in those papers transfer to GDN/KDA and which require adaptation?
  • The section that covers compensator tokens would need proper attribution: this paper is not the first to show that LA models can underutilize their state capacity [2] and that this can be exploited [3]. Without this one might be led to believe that the finding in e.g., Figure 5 is novel. The use of compensator tokens as motivated by the low-rank structure is the contribution here.
  • The speed-up that one can achieve over an unquantized state depends on the head size and the amount of memory that LeapQuant needs to allocate to its buffered updates, compensator tokens, the residual and the residual scaling factor/preconditioner. At smaller head dimensions (e.g. 64x64 or 64x128 instead of 128x128) the speed-ups may be less pronounced. Can the authors add a short experiment where it is shown how the speed-ups achieved with LeapQuant scale with state dimension?
  • Can the authors detail how many power iterations are run to extract the compensator tokens? How does one decide on that number of iterations? Does it depend on the underlying model (e.g., Qwen vs Kimi)?

[2] https://arxiv.org/pdf/2504.19561 [3] https://arxiv.org/pdf/2602.04852

4 comments

We thank the reviewer for the careful reading and the constructive feedback. We address each point below.

Q1

  • The authors write For state space models, Quamba (Chiang et al., 2025b) and MambaQuant (Yue et al., 2025) quantize the weights and activations, and Quamba2 (Chiang et al., 2025a) and Q-Mamba (Tianqi et al., 2025) further quantize the cached SSM state. These methods are specialized to Mamba's selective SSM, whose state is updated differently from that of other linear attention architectures. This is too broad of a statement since e.g., Quamba2 is not specialized to only Mamba1 and in fact supports Mamba2, whose core recurrence is scalar-decayed LA. Could the authors clarify which components of the methods proposed in those papers transfer to GDN/KDA and which require adaptation?

Response:

We agree that our statement was too broad and will incorporate a more detailed explanation as below.

First, most designs in these related works are orthogonal to recurrent-state quantization. Quamba and MambaQuant quantize weights and activations, and Quamba2's sort-and-cluster and per-state-group techniques quantize the recurrence inputs (x, B, C), which correspond to q/k/v in GDN/KDA. Q-Mamba's Efficient Selectivity Reconstruction calibrates Mamba's selective parameters and does not target the state representation either. These techniques can be combined with LeapQuant.

Meanwhile, Quamba2 and Q-Mamba quantize the recurrent state of Mamba models at every step. GDN and KDA update the state through the delta rule, whose write depends on the current state, and KDA further applies channel-wise decay, so errors from per-step quantization behave differently and accumulate over long generations. To verify this, we applied Q-Mamba's state quantization to Qwen3.5-9B:

AIMEGPQALCBMMLU-Pro
FP3287.981.364.183.3
Q-Mamba43.357.145.877.1
LeapQuant87.981.864.183.8

It loses up to 44.6 points, and the loss is largest on long-generation tasks such as AIME and LiveCodeBench, while LeapQuant matches FP32 on all four tasks. We will clarify this in the related work.

Q2

  • The section that covers compensator tokens would need proper attribution: this paper is not the first to show that LA models can underutilize their state capacity [2] and that this can be exploited [3]. Without this one might be led to believe that the finding in e.g., Figure 5 is novel. The use of compensator tokens as motivated by the low-rank structure is the contribution here.

[2] https://arxiv.org/pdf/2504.19561

[3] https://arxiv.org/pdf/2602.04852

Response:

We thank the reviewer and agree that the underutilized state capacity of linear attention models was shown in [2] and exploited in [3]. We will cite both in the Compensator Token section. Our contribution is to use this structure in a form that perfectly matches per-window quantization exactly: each Compensator Token is a rank-one update, the same form as a real token, so it is processed as one more token in the window at almost no extra cost. We will add this to the revision.

Q3

  • The speed-up that one can achieve over an unquantized state depends on the head size and the amount of memory that LeapQuant needs to allocate to its buffered updates, compensator tokens, the residual and the residual scaling factor/preconditioner. At smaller head dimensions (e.g. 64x64 or 64x128 instead of 128x128) the speed-ups may be less pronounced. Can the authors add a short experiment where it is shown how the speed-ups achieved with LeapQuant scale with state dimension?

Response:

We measured the kernel speedup of LeapQuant over the FP32 state at three head dimensions, with the same configuration (p=16p=16, r=4r=4) and batch size as in the paper:

Head dimension (dk×dvd_k \times d_v)64×6464×128128×128
Kernel speedup over FP321.88×2.37×2.52×

LeapQuant still achieves substantial speedups at smaller head dimensions. As the reviewer expected, the gain is smaller under the same configuration. The buffered updates and Compensator Tokens grow with dk+dvd_k + d_v, while the state grows with dkdvd_k d_v, so their size relative to the state increases as the head dimension shrinks. We note two points:

  1. Smaller head dimensions are rarely used in practice because they store less information per head and lower the arithmetic intensity of chunked prefill and training. Recent linear attention models in the Qwen, Kimi, and GLM families all use 128×128 heads.
  2. With smaller heads, the number of Compensator Tokens can be reduced to keep the same overhead relative to the state, which would recover a similar speedup. As few mainstream models use such heads, we haven't evaluated the accuracy of these configurations.

Q4

  • Can the authors detail how many power iterations are run to extract the compensator tokens? How does one decide on that number of iterations? Does it depend on the underlying model (e.g., Qwen vs Kimi)?

Response:

We extract all rr Compensator Tokens together with 3 rounds of block power iteration (subspace iteration) from a fixed random start. A few iterations suffice because a few singular components carry most of the state’s energy, and any part of the state that the Compensator Tokens miss is still kept in the quantized residual. We use the same setting for all models (Qwen, Kimi, and GLM) without per-model tuning.

We thank the reviewer again for the helpful comments. We will incorporate the clarifications and additional results discussed above into the revision.

I thank the authors for addressing my critiques and for the additional experiments. I will increase my score.

Dear Reviewer tEE8,

Thank you for the careful review and for raising your score. We appreciate the constructive feedback and the discussion.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.