acceptodds
Under review as a conference paper at ICLR 2027

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

55 views

Abstract

Linear attention replaces growing KV caches with fixed-size recurrent states, yet these persistent states can become a substantial memory bottleneck under concurrent serving. Directly quantizing recurrent states to low precision often leads to severe accuracy degradation, as quantization errors propagate through successive state updates. We discover that the impact of these errors depends on two complementary dimensions: temporally, errors in long-lived memory can persist across many decoding steps; spatially, errors in different key rows affect model outputs differently, while state magnitudes vary substantially along both rows and columns. Motivated by these observations, we propose STEPQuant, a spatial-temporal post-training quantization framework for Delta-rule recurrent states. STEPQuant allocates precision according to error magnitude and memory lifetime, and jointly fits key-row and value-column scales based on state distributions and key-row impact on output error. Experiments on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct across both long- and short-generation benchmarks show that STEPQuant closely matches FP32-state accuracy under a nominal 6-bit budget and outperforms uniform INT8 in its 4-bit configuration. Integrated into SGLang with optimized GPU kernels, 6-bit STEPQuant achieves over 5× recurrent-state compression and reduces total serving memory by up to 68.7%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Gated Linear Attention, Model Quantization, Efficient Inference, Recurrent State Compression

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_92zZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

STEPQuant is a mixed-precision quantization scheme targeting the recurrent states of gated Delta-rule attention architectures, as found in models such as Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct.

STEPQuant is motivated by two empirical observations: (1) Quantization error can accumulate over time as the recurrent state is repeatedly updated and requantized. (2) Quantization error has spatial dependencies: errors in certain key rows have a greater impact on attention readout, while some rows and columns of the recurrent state exhibit substantially larger magnitudes.

During offline calibration, quantization bit widths are allocated per head (Qwen) or per key row (Kimi), based on (1) memory lifetime, as estimated from the forget gates (components that forget information more quickly can be quantized more aggressively), and (2) empirical reconstruction error at different bit widths. For Qwen, reconstruction error estimates are additionally weighted by calibrated key-row readout sensitivity. A small number of particularly high-risk components, called pivots, are retained in FP16 precision.

During inference, to account for outlier magnitudes across key and value channels, separate scaling factors (r) and (c) are introduced for rows and columns, respectively. Each element is reconstructed by multiplying its low-bit representation by both scaling factors. Row scales account for both current magnitudes and offline-calibrated key-row importance, while column scales are fitted to minimize importance-weighted reconstruction error.

STEPQuant achieves near-baseline accuracy at nominal 6-bit quantization, using approximately 6.3 bits per element on average instead of FP32. The authors additionally report up to a 68.7% reduction in estimated total serving memory at a batch size of 512 under their SGLang state-pool reservation assumptions, reflecting the greater impact of recurrent-state compression when many requests are served concurrently.

Critical strengths
  1. Strong empirical results demonstrate substantial recurrent-state compression with minimal accuracy degradation. Tables 1–3 show accuracy close to the FP32 baseline at nominal 6-bit precision (approximately 6.3 bits per element on average), with consistently higher average accuracy than uniform INT8 quantization in Tables 1–2.

  2. Well-motivated treatment of temporal and spatial quantization errors. Figures 1 and 2 provide useful and compelling empirical evidence of temporal and spatial patterns, showing that recurrent-state components with longer memory lifetimes tend to accumulate greater quantization error, that certain key rows have a greater impact on attention readout error, and that substantial magnitude outliers exist along both key and value dimensions.

  3. Meaningful systems results. Using SGLang with four A800 GPUs, the authors report that at a batch size of 512, 6-bit STEPQuant reduces estimated total serving memory from 419.73 to 131.18 GiB (a 68.7% reduction) under their state-pool reservation model. They additionally demonstrate improvements in full-model decode throughput of up to 20.53% for Qwen and 11.80% for Kimi at this batch size.

Critical weaknesses
  1. The paper provides useful component ablations and an indirect comparison with DAMP in Appendix G. However, the latter compares results obtained under different evaluation settings, so a controlled comparison with a closely related recurrent-state quantization approach remains unavailable. I appreciate that DAMP’s implementation was unavailable for reproduction; nevertheless, this limits the conclusions that can be drawn about STEPQuant’s comparative accuracy and compression efficiency against concurrent approaches.

  2. The applicability of the headline memory reduction depends on the serving workload and state-pool reservation policy. Although the paper reports results across several batch sizes, the 68.7% reduction is obtained at 512 concurrent requests under a configuration reserving five state slots per supported request plus one global sentinel. Appendix F.3 explains this configuration and provides an analytical capacity model. However, the relationship between this model and the physical memory allocation in the measured serving experiments remains unclear. In particular, the reported 419.73 GiB FP32 total exceeds the aggregate memory of four A800 GPUs, even assuming the 80 GB variants. Clarifying the actual state-pool configuration and measured peak GPU memory usage in the throughput experiments, together with the sensitivity of total-memory savings to the reservation policy, would strengthen the systems evaluation.

Questions for authors

Regarding Weakness 2, could the authors clarify whether the reported serving-memory figures represent analytical reserved capacity or measured GPU memory usage? It would also be helpful to understand how the total memory savings change under different state-pool reservation policies.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 92zZ (Part I)

Thanks for your time in dealing with our work. We will answer the question and discuss point by point as follows. We hope that our response satisfactorily addresses the issues you raised. Please feel free to let us know if you have any additional concerns or questions.

W1: The paper provides useful component ablations and an indirect comparison with DAMP in Appendix G. However, the latter compares results obtained under different evaluation settings, so a controlled comparison with a closely related recurrent-state quantization approach remains unavailable. I appreciate that DAMP’s implementation was unavailable for reproduction; nevertheless, this limits the conclusions that can be drawn about STEPQuant’s comparative accuracy and compression efficiency against concurrent approaches.

  • We appreciate this question. To provide a controlled comparison, we re-implemented DAMP and evaluated it on the same models, benchmarks, and evaluation protocol as STEPQuant. We consider both DAMP's original 9.875-bit storage cost and a matched ~6.4-bit setting, where non-FP16 channels use INT4 with a Hadamard transform. At 9.875 bits, our implementation remains close to FP32, suggesting that the re-implementation captures DAMP's intended behavior.

    ModelStateBitsLCBEvalPlusAIMEMATHHMMTGPQA-DIFBenchAvg.
    Qwen3.8-27BFP323285.3184.8787.7197.674.2480.8153.6780.60
    Qwen3.8-27BDAMP9.87584.4584.586.8897.773.1185.8653.6780.88
    Qwen3.8-27BDAMP6.37513.4648.0619.7977.917.0542.9318.3333.93
    Qwen3.8-27BSTEPQuant@66.3685.4285.5487.2497.3573.380.8754.4280.59
    Kimi-Linear-48B-A3BFP323254.5276.7568.3393.2545.0869.723.061.52
    Kimi-Linear-48B-A3BDAMP9.87551.9476.9462.0893.441.8667.1722.3359.39
    Kimi-Linear-48B-A3BDAMP6.37518.0153.0431.0481.419.752.5316.038.82
    Kimi-Linear-48B-A3BSTEPQuant@66.3054.8676.5767.7193.646.2168.4322.9261.47
  • This controlled comparison shows a clear accuracy–storage advantage for STEPQuant. At a comparable ~6.4-bit storage cost, STEPQuant preserves near-FP32 performance on both models, while DAMP degrades substantially. DAMP requires about 9.9 bits to achieve comparable accuracy, indicating that STEPQuant provides more effective low-bit recurrent-state compression.

AuthorsOpenReview ↗

Response to Reviewer 92zZ (Part II)

W2 & Q1: The reported 68.7% memory reduction depends on the serving workload and state-pool reservation policy. However, it is unclear whether the reported values represent analytical reserved capacity or measured peak GPU memory. In particular, the 419.73 GiB FP32 figure exceeds the aggregate memory of four A800 GPUs. Please clarify the hardware and state-pool configurations used for the memory and throughput experiments, report measured peak GPU memory where applicable, and evaluate how the total-memory savings vary with the reservation policy.

We clarify below the distinction between reserved serving-memory capacity and measured peak GPU memory, the hardware configurations used in the two experiments, and the sensitivity of memory savings to the state-slot reservation policy.

Experimental settings.

  • Fig. 3(c) reports reserved serving-memory capacity rather than measured peak GPU memory usage. Specifically, the reported values follow the allocation-based accounting in Eq. (25), which includes model weights, the preallocated recurrent-state pool, and the provisioned attention KV cache. This metric therefore represents reserved serving capacity rather than actual peak runtime allocation.

  • We also clarify that the memory and throughput experiments use different hardware and state-pool configurations:

    ExperimentGPUsState slots
    Memory saving (Fig. 3c, W4A16)8 × A800-80GB, TP85B + 1 (SGLang default, r = 5)
    Decode throughput (App. F.2, F.5, BF16)4 × A800-80GB, TP4B + 1 (running states only, r = 1)
  • For the memory-capacity analysis, SGLang reserves five state slots per supported request plus one global sentinel (r=5r=5) to support radix caching and overlap tracking. During steady-state decoding, the four additional reserved slots per request remain idle and are not read or written by the decode step, so they do not incur additional state-update computation. Therefore, we use r=5r=5 in Fig. 3(c) to reflect this reservation policy. The reported 419.73 GiB at B=512B=512 is the corresponding reserved capacity, evaluated on eight A800-80GB GPUs.

  • For the throughput experiments, we use r=1r=1 and retain only the running states, because the additional reserved slots do not participate in per-token decoding in our fixed-batch benchmark. This substantially reduces the memory requirement and allows the throughput experiments to run on four A800-80GB GPUs.

Peak Memory Evaluation.

  • To complement the capacity analysis with physical GPU memory measurements, we additionally report aggregate peak GPU memory usage in the four-GPU throughput configuration using nvidia-smi (BF16 weights, TP4, (r=1)):

    StateMeasured Peak (GiB)
    FP32178.40
    STEPQuant@6-bit125.99 (−52.41 GiB, −29.4%)
  • Note that the 29.4% peak-memory reduction and 68.7% reserved-capacity reduction are measured under different configurations. Unlike the reserved-capacity metric in Fig. 3(c), these measurements reflect actual runtime GPU memory consumption, including additional system allocations. The results confirm that STEPQuant's state compression translates into a substantial reduction in physical GPU memory usage.

Sensitivity to the reservation policy.

  • To quantify the dependence of total-memory savings on the reservation policy, we evaluate Eq. (25) under different r, keeping the W4A16 weight, supported concurrency B=512, and KV-cache capacity consistent with Fig. 3(c).

    rFP32 (GiB)STEPQuant@6-bit (GiB)Saving
    1 (running states only)126.1168.3145.8%
    3 (radix cache, no overlap tracking)272.9299.7463.5%
    5 (SGLang default)419.73131.1868.7%
  • STEPQuant reduces reserved serving-memory capacity by 45.8% at r=1r=1 and 68.7% at r=5r=5, demonstrating consistent savings across different reservation policies. Together with the 29.4% reduction in measured peak GPU memory, these results support the practical memory benefits of recurrent-state compression.

We have revised Sec. 5.5, the caption of Fig. 3(c), and Apps. F.2/F.3/F.5 to explicitly distinguish reserved capacity from measured peak GPU memory and to document the corresponding hardware and reservation configurations.

Reviewer_7wxTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes STEPQuant, a post-training quantization framework for recurrent states in Delta-rule linear attention models. It combines lifetime-aware mixed-precision allocation with key-row-aware dual-axis scaling to address temporal error persistence and spatial variation in state magnitudes and output sensitivity. Experiments on Qwen and Kimi show near-FP32 accuracy under a nominal 6-bit state budget, supported by component ablations and an SGLang implementation demonstrating memory savings and decoding speedups.

Critical strengths
  1. STEPQuant combines mixed-precision allocation based on error magnitude and persistence with importance-aware row and column scaling, addressing complementary sources of recurrent-state quantization error.
  2. Under a nominal 6-bit state budget, the method closely matches FP32-state accuracy across both evaluated GDN and KDA models. Also, integration into SGLang yields over 5× recurrent-state compression and up to 20.53% higher full-model decoding throughput, substantiating the practical utility of the proposed approach.
Critical weaknesses
  1. Although Appendix A.2 characterizes the full readout-error quadratic form, the proposed fitting relies on row-wise importance and does not explicitly model interactions between errors in different rows. Please quantify the contribution of cross-row terms and, if substantial, explore a simple refinement of the row-wise importance measure to account for them.
  2. Lack of a controlled comparison with DAMP. Appendix G compares accuracy retention across different evaluation settings. Please evaluate both methods using the same settings, and effective storage budgets.
  3. The evaluation includes only one GDN checkpoint and one KDA checkpoint. Please include a differently sized GDN-based Qwen model and a GDN-based MoE model to assess generalization.
  4. The experiments do not isolate the contributions of the two scaling axes. Please compare row-only, column-only, and dual-axis scaling.
  5. The appendix provides prediction-drift and cross-domain lifetime-ranking analyses, but longer-horizon behavior and the sensitivity of the complete calibration configuration remain insufficiently explored. Please evaluate error accumulation over longer decoding horizons and sensitivity to calibration domain, sample size, and sequence length.
Questions for authors

Please address the concerns raised in the Critical Weaknesses, particularly through clarification and additional evidence where available.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7wxT (Part I)

W1: Although Appendix A.2 characterizes the full readout-error quadratic form, the proposed fitting relies on row-wise importance and does not explicitly model interactions between errors in different rows. Please quantify the contribution of cross-row terms and, if substantial, explore a simple refinement of the row-wise importance measure to account for them.

  • We quantified the contribution of cross-row terms using the full readout-error quadratic form in Appendix A.2. Specifically, we measure the ratio of the expected cross-row terms to the expected full quadratic form over (H) future decode steps: CrossRowShare(H)=E[CrossRowTerms(H)]E[FullQuadraticForm(H)]\mathrm{CrossRowShare}(H) = \frac{\mathbb E[\mathrm{CrossRowTerms}(H)]} {\mathbb E[\mathrm{FullQuadraticForm}(H)]}. The results are summarized below:

    ModelBitsH = 16H = 128H = 1024
    Qwen3.8-27B6-bit-0.075-0.033-0.003
    Qwen3.8-27B4-bit0.0530.0360.028
    Kimi-Linear-48B-A3B6-bit0.007-0.004-0.014
    Kimi-Linear-48B-A3B4-bit0.0680.040.03
  • The cross-row terms make only a limited net contribution to the expected readout error. At (H=1024), the signed contribution remains within 3% across both models and bit widths. Therefore, explicitly modeling cross-row interactions would add complexity for limited gain, and we retain the row-wise importance measure as a practical approximation.

W2: Lack of a controlled comparison with DAMP. Appendix G compares accuracy retention across different evaluation settings. Please evaluate both methods using the same settings, and effective storage budgets.

  • We agree that a direct comparison under identical settings is necessary. We therefore re-implemented DAMP and evaluated both methods using the same models, benchmarks, and evaluation protocol. We report DAMP at its original 9.875-bit storage cost and at a matched ~6.4-bit budget for a fair comparison with STEPQuant.

    ModelStateBitsLCBEvalPlusAIMEMATHHMMTGPQA-DIFBenchAvg.
    Qwen3.8-27BFP323285.3184.8787.7197.674.2480.8153.6780.60
    Qwen3.8-27BDAMP9.87584.4584.586.8897.773.1185.8653.6780.88
    Qwen3.8-27BDAMP6.37513.4648.0619.7977.917.0542.9318.3333.93
    Qwen3.8-27BSTEPQuant@66.3685.4285.5487.2497.3573.380.8754.4280.59
    Kimi-Linear-48B-A3BFP323254.5276.7568.3393.2545.0869.723.061.52
    Kimi-Linear-48B-A3BDAMP9.87551.9476.9462.0893.441.8667.1722.3359.39
    Kimi-Linear-48B-A3BDAMP6.37518.0153.0431.0481.419.752.5316.038.82
    Kimi-Linear-48B-A3BSTEPQuant@66.3054.8676.5767.7193.646.2168.4322.9261.47
  • At matched storage costs, STEPQuant consistently preserves near-FP32 accuracy, while DAMP degrades substantially. DAMP requires a much higher storage cost (~9.9 bits) to achieve comparable performance, showing that STEPQuant provides a better accuracy–storage trade-off.

AuthorsOpenReview ↗

Response to Reviewer 7wxT (Part II)

W3: The evaluation includes only one GDN checkpoint and one KDA checkpoint. Please include a differently sized GDN-based Qwen model and a GDN-based MoE model to assess generalization.

  • We thank the reviewer for the suggestion. To evaluate the generalization of STEPQuant across different model sizes and architectures, we conducted additional experiments on Qwen3.6-35B-A3B (GDN-based MoE), Qwen3.5-9B (dense GDN), and Qwen3.6-27B (dense GDN). The results are as follows:

    Qwen3.6-35B-A3B (GDN MoE)

    StateAIMEHMMTGPQA-DLCBMATHEvalPlusIFBenchAvg.
    Qwen3.6-35B-A3B FP3277.0852.8476.7768.0698.8081.5536.0070.16
    INT85.423.6037.8830.7175.4069.7432.6736.49
    INT60.000.1913.132.5612.2031.0024.3311.92
    INT40.000.0015.660.283.8013.118.007.26
    STEPQuant@6-bit78.5457.9576.7768.9197.8081.9235.3371.03
    STEPQuant@4-bit33.3322.9267.6850.2491.0078.7834.6754.09

    Qwen3.5-9B (dense GDN)

    StateAIMEHMMTGPQA-DLCBMATHEvalPlusIFBenchAvg.
    FP3272.9251.8978.2858.7795.869.3737.6766.39
    INT838.5422.3575.2540.7693.0066.9734.6753.08
    INT60.000.0015.6613.3645.6042.6229.3320.94
    INT40.000.000.000.005.807.2023.675.24
    STEPQuant@6-bit74.1752.6581.3157.6396.670.1134.0066.64
    STEPQuant@4-bit59.3837.578.2853.8495.869.9333.3361.15

    Qwen3.6-27B (dense GDN)

    StateAIMEHMMTGPQA-DLCBMATHEvalPlusIFBenchAvg.
    Qwen3.6-27B FP3279.7956.4477.2769.7698.4081.7338.6771.72
    INT870.6252.8473.7463.4199.0078.6041.6768.55
    INT630.0024.2464.1436.5990.0064.5837.0049.51
    INT40.000.0025.2513.5537.0049.4534.3322.80
    STEPQuant@6-bit80.4259.4777.2771.1898.2082.4740.0072.72
    STEPQuant@4-bit79.5857.0176.5269.5797.2081.5540.0071.63
  • STEPQuant@6-bit matches or exceeds FP32 accuracy on average across all three models, demonstrating its generalization across different model sizes and both dense and MoE GDN architectures. We also find that Qwen3.6-35B-A3B is particularly sensitive to state quantization, yet STEPQuant@4-bit still substantially outperforms uniform INT8.

W4: The experiments do not isolate the contributions of the two scaling axes. Please compare row-only, column-only, and dual-axis scaling.

  • We conducted an ablation study comparing (i) row-only scaling, (ii) column-only scaling, (iii) dual-axis scaling (without row-impact weighting, ωi=1\omega_i=1), and (iv) dual-axis scaling (with row-impact weighting). All experiments use uniform 6-bit quantization without precision allocation or FP16 pivots. The results are as follows:

    ModelMethodAIMEGPQA-DLCBAvg.
    Qwen3.8-27BRow only33.9651.5230.4338.63
    Qwen3.8-27BColumn only45.2157.3234.0345.52
    Qwen3.8-27BDual-axis (w/o ωi\omega_i)77.0879.0479.1578.42
    Qwen3.8-27BDual-axis (w ωi\omega_i)79.7980.6880.0980.19
    Kimi-Linear-48B-A3BRow only22.7157.8342.3740.97
    Kimi-Linear-48B-A3BColumn only28.9655.5643.2242.58
    Kimi-Linear-48B-A3BDual-axis (w/o ωi\omega_i)52.561.1148.9154.17
    Kimi-Linear-48B-A3BDual-axis (w ωi\omega_i)63.5466.6752.8061.00
  • Dual-axis scaling substantially outperforms both row-only and column-only scaling on both models, demonstrating the benefit of modeling both axes. Incorporating the row-impact score ωi\omega_i provides further improvements. These results validate the design of our Key-Row-Aware Dual-axis Fitting.

AuthorsOpenReview ↗

Response to Reviewer 7wxT (Part III)

W5: The appendix provides prediction-drift and cross-domain lifetime-ranking analyses, but longer-horizon behavior and the sensitivity of the complete calibration configuration remain insufficiently explored. Please evaluate error accumulation over longer decoding horizons and sensitivity to calibration domain, sample size, and sequence length.

  • We conducted additional experiments to examine long-horizon error accumulation and sensitivity to calibration configurations from the following perspectives:

(a) Long-horizon prediction drift. We extended the forced-decode evaluation to approximately 64K tokens on Qwen3.8-27B and measured excess NLL relative to FP32 across decoding intervals.

StateMean0–8K8–16K16–24K24–32K32–40K40–48K48–56K56–63KSlope / 1K
STEPQuant@60.0043-0.00070.00140.00140.00360.00520.00520.01050.00850.00018
INT62.2760.7481.2451.5792.0102.5622.8783.7183.8710.058
  • As shown in the table, STEPQuant@6 maintains low excess NLL over long decoding horizons, with a slope of only 0.00018 per 1K tokens, compared to 0.058 for uniform INT6. These results indicate that STEPQuant substantially mitigates long-horizon error accumulation.

(b) Single-error propagation. We further measured the propagation of a single injected quantization error over subsequent decoding steps. Values are normalized to the readout error at step 1 and averaged within each interval.

ModelStateStep 1Steps 2–16Steps 17–128Steps 129–1024
Qwen3.8-27BSTEPQuant@610.5010.1520.042
Qwen3.8-27BSTEPQuant@410.3650.0550.0066
Kimi-Linear-48B-A3BSTEPQuant@610.2730.0360.0083
Kimi-Linear-48B-A3BSTEPQuant@410.2760.0310.0026
  • We observe that isolated quantization errors decay substantially, with average readout error below 5% of its initial value over steps 129–1024.

(c) Calibration sensitivity. We evaluated different calibration domains, sample counts, and sequence lengths at 6 bits. We measured excess NLL and PPL against FP32 over 8K decode steps on four PG-19 books.

CalibrationQwen excess NLLQwen PPLKimi excess NLLKimi PPL
WikiText-2, 32 × 2048 (reference)0.10729.6120.024810.068
Github, 32 × 20480.10449.5850.01319.951
WikiText-2, 8 × 20480.10899.6280.026110.081
WikiText-2, 32 × 5120.10549.5940.032610.146
  • Across these settings, excess NLL varies by at most 0.012 nats/token, indicating that STEPQuant is robust to the different calibration configurations.

(d) Calibration plan stability. We further compared the precision assignments and FP16 pivot selections under different calibration settings against the reference configuration (WikiText-2, 32 × 2048). Here, Agree denotes the fraction of units assigned the same bit width, while Pivot overlap measures the fraction of shared FP16 pivots.

CalibrationQwen agreeQwen pivot overlapKimi agreeKimi pivot overlap
WikiText-2, 8 × 20480.9911.0000.9820.926
WikiText-2, 16 × 20480.9921.0000.9840.941
WikiText-2, 64 × 20480.9931.0000.9870.943
WikiText-2, 32 × 5120.9610.8750.9370.607
WikiText-2, 32 × 40960.9851.0000.9740.855
Pile-CC, 32 × 20480.9560.8750.9450.711
Github, 32 × 20480.9240.9380.9130.568
ArXiv, 32 × 20480.9460.9380.9290.736
  • Bit-width assignments remain highly consistent across calibration configurations, with at least 91.3% agreement on both models. This further supports the robustness of STEPQuant's precision allocation to calibration setup.

Overall, these experiments show that STEPQuant remains stable over long decoding horizons and is robust to substantial changes in calibration domain, sample count, and sequence length.

Reviewer_KuKxOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Paper proposes StepQuant which quantizes the recurrent states of GatedDeltaNet (GDN) and KimiDeltaAttention (KDA) models. It combines lifetime weighted mixed precision allocation, sparse FP16 pivots and dual axis scaling informed by key row readout sensitivity. The method closely matches FP32-state accuracy at 6-bit budget on Qwen3.8-27B and Kimi-Linear-48B-A3B with substantial state-memory savings and inference latency improvements.

Critical strengths
  1. The paper discusses a practical problem with convincing results. Fixed size recurrent states can still consume significant memory at high concurrency.
  2. Evaluation covers GDN and KDA along with BF16 and 4bit weights across seven reasoning benchmarks and six short generation tasks.
  3. Actual hardware implementation in SGLang gives real memory improvements and inference speedup.
  4. Analysis provides useful evidence about recurrent-state sensitivity.
Critical weaknesses
  1. Temporal axis quantization It is not clear if lifetime awareness provides any benefits. Can the authors compare lifetime-weighted allocation against distortion only mixed precision quantization. Specifically, setting Lu=1L_u = 1 while keeping the quantizer and everything else identical. Without this comparison, the improvement can largely be due to ordinary distortion based mixed precision allocation and FP16 pivots.

  2. Spatial axis quantization For the spatial component, it is not apparent if the row impact score provides any meaningful improvements. Specifically, the authors should compare the same dual-axis fitting procedure with ωi=1\omega_i = 1.

  3. Comparison to related works can be enhanced. Please compare StepQuant to Q-Mamba [1] under identical quantizer and pivot settings. Additionally, DAMP [2] also uses reconstruction error and decay-based persistence to select high precision recurrent state channels and need to be compared with.

  4. Additonal storage required for FP16 Pivot should be reported in the main tables

  5. Increase long context evaluation coverage It will be beneficial to have controlled RULER and LongBench evaluations are different sequence lengths to understand the impact of recurrent state quantization.

  6. Beyond GDN The authors compare GDN and KDA recurrent states which are fairly similar in compute. Can the authors also attempt to validate StepQuant on other linear attention variants like Mamba2?

[1] Q-Mamba: Towards more efficient Mamba models via post-training quantization

[2] DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization

Questions for authors

Please see weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer KuKx (Part I)

We sincerely thank the reviewer for providing valuable feedback. We detail our response below point by point. Please kindly let us know whether you have any further concerns.

W1: Temporal axis quantization. It is not clear if lifetime awareness provides any benefits. Can the authors compare lifetime-weighted allocation against distortion only mixed precision quantization. Specifically, setting Lu=1L_u=1 while keeping the quantizer and everything else identical. Without this comparison, the improvement can largely be due to ordinary distortion based mixed precision allocation and FP16 pivots.

  • Thanks for the insightful suggestion. We conducted the requested controlled ablation. We kept the quantizer, bit budget, FP16 pivots, and allocation algorithm unchanged, and compared the results with and without lifetime weighting (Lu=1L_u=1) under the 6-bit setting using only the temporal component. The results are as follows:

  • ModelAIMEGPQA-DLCBAvg.
    Qwen3.8-27B (Lu=1L_u=1)37.7154.5541.9444.73
    Qwen3.8-27B (with LuL_u)74.5879.6772.6475.63
    Kimi-Linear-48B-A3B (Lu=1L_u=1)51.8861.8743.9352.56
    Kimi-Linear-48B-A3B (with LuL_u)61.6766.1648.9158.91

    Removing lifetime weighting (Lu=1L_u=1) consistently degrades performance across all evaluated benchmarks and both models. This demonstrates the benefit of lifetime-aware allocation for recurrent-state quantization.

W2: Spatial axis quantization. For the spatial component, it is not apparent if the row impact score provides any meaningful improvements. Specifically, the authors should compare the same dual-axis fitting procedure with ωi=1\omega_i=1.

  • Thank you for the suggestion. We conducted the requested ablation study by setting ωi=1\omega_i=1 while keeping all other configurations unchanged. We evaluated the spatial-only component under both 4-bit and 6-bit budgets. The results are as follows:

    ModelBitsAIMEGPQA-DLCBAvg.
    Qwen3.8-27B (ω=1\omega = 1)677.0879.0479.1578.42
    Qwen3.8-27B (with ω\omega)679.7980.6880.0980.19
    Qwen3.8-27B (ω=1\omega = 1)467.9262.1267.9666.00
    Qwen3.8-27B (with ω\omega)475.2170.7175.9273.95
    Kimi-Linear-48B-A3B (ω=1\omega = 1)652.5061.1148.9154.17
    Kimi-Linear-48B-A3B (with ω\omega)663.5466.6752.8061.00
    Kimi-Linear-48B-A3B (ω=1\omega = 1)421.0439.934.9831.97
    Kimi-Linear-48B-A3B (with ω\omega)426.6750.5139.5138.89
  • As shown in the table, incorporating row-impact weighting ωi\omega_i consistently improves performance across all evaluated datasets, both models, and both bit budgets. These results demonstrate that the row-impact score provides a clear benefit beyond dual-axis fitting alone.

AuthorsOpenReview ↗

Response to Reviewer KuKx (Part II)

W3: Comparison to related works can be enhanced. Please compare StepQuant to Q-Mamba [1] under identical quantizer and pivot settings. Additionally, DAMP [2] also uses reconstruction error and decay-based persistence to select high precision recurrent state channels and need to be compared with.

  • We thank the reviewer for the useful suggestion. We conducted additional comparisons with both (a) Q-Mamba and (b) DAMP.

  • (a) Comparison with Q-Mamba. We incorporated STEPQuant's bit allocation and FP16 pivots into Q-Mamba while retaining its original quantization method. We then compared it with STEPQuant under the same 4-bit and 6-bit budgets. The results are as follows:

    ModelMethodBitsAIMEGPQA-DLCBAvg.
    Qwen3.8-27BQ-Mamba673.1376.7777.9775.95
    Qwen3.8-27BQ-Mamba + allocation/pivots685.080.8181.0482.28
    Qwen3.8-27BSTEPQuant687.2480.8785.4284.51
    Qwen3.8-27BQ-Mamba40.006.0616.877.64
    Qwen3.8-27BQ-Mamba + allocation/pivots40.000.000.000.00
    Qwen3.8-27BSTEPQuant486.2581.5786.3584.72
    Kimi-Linear-48B-A3BQ-Mamba638.5457.0744.5546.72
    Kimi-Linear-48B-A3BQ-Mamba + allocation/pivots661.8867.9352.860.87
    Kimi-Linear-48B-A3BSTEPQuant667.7168.4354.8663.67
    Kimi-Linear-48B-A3BQ-Mamba40.0010.1028.9113.00
    Kimi-Linear-48B-A3BQ-Mamba + allocation/pivots48.1341.9241.0030.35
    Kimi-Linear-48B-A3BSTEPQuant459.1764.1453.2958.87
  • The results show that STEPQuant consistently outperforms Q-Mamba even when using the same bit allocation and FP16 pivots. The performance gap becomes particularly large under the 4-bit budget. These results show that the gains of STEPQuant cannot be attributed solely to bit allocation or FP16 pivots, and prove the effectiveness of our spatial quantization design.

  • (b) Comparison with DAMP. We previously compared relative FP32 accuracy with DAMP in App. G. Since DAMP's code is unavailable, we re-implemented it and evaluated it using the same models and evaluation protocol. We tested DAMP at its original 9.875-bit storage cost and at a matched ~6.4-bit budget, using INT4 with a Hadamard transform for non-FP16 channels. Our re-implementation achieves near-FP32 accuracy at 9.875 bits, indicating that the implementation reasonably reproduces DAMP's behavior.

    ModelStateBitsLCBEvalPlusAIMEMATHHMMTGPQA-DIFBenchAvg.
    Qwen3.8-27BFP323285.3184.8787.7197.674.2480.8153.6780.60
    Qwen3.8-27BDAMP9.87584.4584.586.8897.773.1185.8653.6780.88
    Qwen3.8-27BDAMP6.37513.4648.0619.7977.917.0542.9318.3333.93
    Qwen3.8-27BSTEPQuant@66.3685.4285.5487.2497.3573.380.8754.4280.59
    Kimi-Linear-48B-A3BFP323254.5276.7568.3393.2545.0869.723.061.52
    Kimi-Linear-48B-A3BDAMP9.87551.9476.9462.0893.441.8667.1722.3359.39
    Kimi-Linear-48B-A3BDAMP6.37518.0153.0431.0481.419.752.5316.038.82
    Kimi-Linear-48B-A3BSTEPQuant@66.3054.8676.5767.7193.646.2168.4322.9261.47
  • From table above, under matched storage budgets (~6.4 bits), STEPQuant substantially outperforms DAMP across both models. DAMP requires a substantially higher precision (~9.9 bits) to approach the accuracy achieved by STEPQuant at ~6.3 bits. These results demonstrate the advantage of STEPQuant for low-bit recurrent-state quantization.

AuthorsOpenReview ↗

Response to Reviewer KuKx (Part III)

W4: Additional storage required for FP16 Pivot should be reported in the main tables.

  • We agree with the reviewer’s suggestion. We have added an effective bit-width column to the main tables (Tables 1–2), accounting for integer codes, FP16 scales, and FP16 pivots. Detailed storage calculations are provided in Appendix D. The effective storage costs are summarized below:
ModelSTEPQuant@6-bitSTEPQuant@4-bit
Qwen3.8-27B6.361 bits/value4.621 bits/value
Kimi-Linear-48B-A3B6.300 bits/value4.300 bits/value

W5: Increase long-context evaluation coverage. It would be beneficial to include controlled RULER and LongBench evaluations at different sequence lengths to understand the impact of recurrent-state quantization.

  • We added evaluations on RULER NIAH and LongBench-E (including 13 tasks) to assess STEPQuant under long-context settings at multiple sequence lengths.

    ModelMethodRULER-128K NIAHRULER-32K NIAHLB-E 0-4KLB-E 4-8KLB-E 8K+LB-E avg
    Qwen3.8-27BFP32100.00100.0059.5157.3654.2657.22
    Qwen3.8-27BSTEPQuant@6-bit100.00100.0059.6457.4754.3857.30
    Qwen3.8-27BSTEPQuant@4-bit100.00100.0059.2957.0754.1757.01
    Kimi-Linear-48B-A3BFP3299.5399.5054.9350.2348.9851.63
    Kimi-Linear-48B-A3BSTEPQuant@6-bit99.4199.5654.7049.8549.2551.39
    Kimi-Linear-48B-A3BSTEPQuant@4-bit99.5099.6253.9350.048.6751.07
  • The results show that STEPQuant@6-bit closely matches FP32 performance across all context lengths, while STEPQuant@4-bit remains within 1.0 percentage point of FP32 in every length bucket. Overall, STEPQuant preserves long-context performance without noticeable degradation as sequence length increases.

W6: Beyond GDN. The authors compare GDN and KDA recurrent states, which are fairly similar in compute. Can the authors also attempt to validate STEPQuant on other linear attention variants like Mamba2?

  • We thank the reviewer for the valuable suggestion.Mamba2 differs from GDN/KDA in its recurrent update, using only the α\alpha decay term and a transposed state formulation. We directly quantize the recurrent state of Mamba2-2.7B and compare STEPQuant with Q-Mamba and uniform quantization using WikiText-2 perplexity:

    State (Mamba2-2.7B)Bits/value (exact)WikiText-2 PPL ↓vs. FP32
    FP32329.062–
    INT8 (sym, g128)8.1259.158+1.06%
    INT6 (sym, g128)6.12515.22+68.0%
    INT4 (sym, g128)4.125272.7collapse
    Q-Mamba@88.3759.147+0.95%
    Q-Mamba@66.3759.526+5.13%
    Q-Mamba@44.37514.82+63.6%
    STEPQuant@66.429.072+0.11%
    STEPQuant@44.609.266+2.25%
  • STEPQuant remains effective on Mamba2, closely matching FP32 performance at 6 bits and maintaining strong performance even at 4 bits, while Q-Mamba shows clear degradation at lower bit widths.

Comments on the paper

2 comments

Dear Reviewer,

I hope this message finds you well. I am writing to kindly remind you that the authors’ rebuttal has been posted and is now available for your review in the system.

We would greatly appreciate it if you could review the rebuttal and provide your assessment at your earliest convenience, as your input is important for us to move forward with a timely decision.

Thank you very much for your time and continued support.

ACs

As only 24 hours remain in the author rebuttal period, I sincerely invite the reviewers to review the author's response and promptly raise any remaining questions or indicate whether you believe the author has addressed the issues you have raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.