- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
STEPQuant is a mixed-precision quantization scheme targeting the recurrent states of gated Delta-rule attention architectures, as found in models such as Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct.
STEPQuant is motivated by two empirical observations: (1) Quantization error can accumulate over time as the recurrent state is repeatedly updated and requantized. (2) Quantization error has spatial dependencies: errors in certain key rows have a greater impact on attention readout, while some rows and columns of the recurrent state exhibit substantially larger magnitudes.
During offline calibration, quantization bit widths are allocated per head (Qwen) or per key row (Kimi), based on (1) memory lifetime, as estimated from the forget gates (components that forget information more quickly can be quantized more aggressively), and (2) empirical reconstruction error at different bit widths. For Qwen, reconstruction error estimates are additionally weighted by calibrated key-row readout sensitivity. A small number of particularly high-risk components, called pivots, are retained in FP16 precision.
During inference, to account for outlier magnitudes across key and value channels, separate scaling factors (r) and (c) are introduced for rows and columns, respectively. Each element is reconstructed by multiplying its low-bit representation by both scaling factors. Row scales account for both current magnitudes and offline-calibrated key-row importance, while column scales are fitted to minimize importance-weighted reconstruction error.
STEPQuant achieves near-baseline accuracy at nominal 6-bit quantization, using approximately 6.3 bits per element on average instead of FP32. The authors additionally report up to a 68.7% reduction in estimated total serving memory at a batch size of 512 under their SGLang state-pool reservation assumptions, reflecting the greater impact of recurrent-state compression when many requests are served concurrently.
Critical strengths
-
Strong empirical results demonstrate substantial recurrent-state compression with minimal accuracy degradation. Tables 1–3 show accuracy close to the FP32 baseline at nominal 6-bit precision (approximately 6.3 bits per element on average), with consistently higher average accuracy than uniform INT8 quantization in Tables 1–2.
-
Well-motivated treatment of temporal and spatial quantization errors. Figures 1 and 2 provide useful and compelling empirical evidence of temporal and spatial patterns, showing that recurrent-state components with longer memory lifetimes tend to accumulate greater quantization error, that certain key rows have a greater impact on attention readout error, and that substantial magnitude outliers exist along both key and value dimensions.
-
Meaningful systems results. Using SGLang with four A800 GPUs, the authors report that at a batch size of 512, 6-bit STEPQuant reduces estimated total serving memory from 419.73 to 131.18 GiB (a 68.7% reduction) under their state-pool reservation model. They additionally demonstrate improvements in full-model decode throughput of up to 20.53% for Qwen and 11.80% for Kimi at this batch size.
Critical weaknesses
-
The paper provides useful component ablations and an indirect comparison with DAMP in Appendix G. However, the latter compares results obtained under different evaluation settings, so a controlled comparison with a closely related recurrent-state quantization approach remains unavailable. I appreciate that DAMP’s implementation was unavailable for reproduction; nevertheless, this limits the conclusions that can be drawn about STEPQuant’s comparative accuracy and compression efficiency against concurrent approaches.
-
The applicability of the headline memory reduction depends on the serving workload and state-pool reservation policy. Although the paper reports results across several batch sizes, the 68.7% reduction is obtained at 512 concurrent requests under a configuration reserving five state slots per supported request plus one global sentinel. Appendix F.3 explains this configuration and provides an analytical capacity model. However, the relationship between this model and the physical memory allocation in the measured serving experiments remains unclear. In particular, the reported 419.73 GiB FP32 total exceeds the aggregate memory of four A800 GPUs, even assuming the 80 GB variants. Clarifying the actual state-pool configuration and measured peak GPU memory usage in the throughput experiments, together with the sensitivity of total-memory savings to the reservation policy, would strengthen the systems evaluation.
Questions for authors
Regarding Weakness 2, could the authors clarify whether the reported serving-memory figures represent analytical reserved capacity or measured GPU memory usage? It would also be helpful to understand how the total memory savings change under different state-pool reservation policies.
2 comments
Response to Reviewer 92zZ (Part II)
W2 & Q1: The reported 68.7% memory reduction depends on the serving workload and state-pool reservation policy. However, it is unclear whether the reported values represent analytical reserved capacity or measured peak GPU memory. In particular, the 419.73 GiB FP32 figure exceeds the aggregate memory of four A800 GPUs. Please clarify the hardware and state-pool configurations used for the memory and throughput experiments, report measured peak GPU memory where applicable, and evaluate how the total-memory savings vary with the reservation policy.
We clarify below the distinction between reserved serving-memory capacity and measured peak GPU memory, the hardware configurations used in the two experiments, and the sensitivity of memory savings to the state-slot reservation policy.
Experimental settings.
Fig. 3(c) reports reserved serving-memory capacity rather than measured peak GPU memory usage. Specifically, the reported values follow the allocation-based accounting in Eq. (25), which includes model weights, the preallocated recurrent-state pool, and the provisioned attention KV cache. This metric therefore represents reserved serving capacity rather than actual peak runtime allocation.
We also clarify that the memory and throughput experiments use different hardware and state-pool configurations:
Experiment GPUs State slots Memory saving (Fig. 3c, W4A16) 8 × A800-80GB, TP8 5B + 1 (SGLang default, r = 5) Decode throughput (App. F.2, F.5, BF16) 4 × A800-80GB, TP4 B + 1 (running states only, r = 1) For the memory-capacity analysis, SGLang reserves five state slots per supported request plus one global sentinel () to support radix caching and overlap tracking. During steady-state decoding, the four additional reserved slots per request remain idle and are not read or written by the decode step, so they do not incur additional state-update computation. Therefore, we use in Fig. 3(c) to reflect this reservation policy. The reported 419.73 GiB at is the corresponding reserved capacity, evaluated on eight A800-80GB GPUs.
For the throughput experiments, we use and retain only the running states, because the additional reserved slots do not participate in per-token decoding in our fixed-batch benchmark. This substantially reduces the memory requirement and allows the throughput experiments to run on four A800-80GB GPUs.
Peak Memory Evaluation.
To complement the capacity analysis with physical GPU memory measurements, we additionally report aggregate peak GPU memory usage in the four-GPU throughput configuration using nvidia-smi (BF16 weights, TP4, (r=1)):
State Measured Peak (GiB) FP32 178.40 STEPQuant@6-bit 125.99 (−52.41 GiB, −29.4%) Note that the 29.4% peak-memory reduction and 68.7% reserved-capacity reduction are measured under different configurations. Unlike the reserved-capacity metric in Fig. 3(c), these measurements reflect actual runtime GPU memory consumption, including additional system allocations. The results confirm that STEPQuant's state compression translates into a substantial reduction in physical GPU memory usage.
Sensitivity to the reservation policy.
To quantify the dependence of total-memory savings on the reservation policy, we evaluate Eq. (25) under different r, keeping the W4A16 weight, supported concurrency B=512, and KV-cache capacity consistent with Fig. 3(c).
r FP32 (GiB) STEPQuant@6-bit (GiB) Saving 1 (running states only) 126.11 68.31 45.8% 3 (radix cache, no overlap tracking) 272.92 99.74 63.5% 5 (SGLang default) 419.73 131.18 68.7% STEPQuant reduces reserved serving-memory capacity by 45.8% at and 68.7% at , demonstrating consistent savings across different reservation policies. Together with the 29.4% reduction in measured peak GPU memory, these results support the practical memory benefits of recurrent-state compression.
We have revised Sec. 5.5, the caption of Fig. 3(c), and Apps. F.2/F.3/F.5 to explicitly distinguish reserved capacity from measured peak GPU memory and to document the corresponding hardware and reservation configurations.
Response to Reviewer 92zZ (Part I)
Thanks for your time in dealing with our work. We will answer the question and discuss point by point as follows. We hope that our response satisfactorily addresses the issues you raised. Please feel free to let us know if you have any additional concerns or questions.
W1: The paper provides useful component ablations and an indirect comparison with DAMP in Appendix G. However, the latter compares results obtained under different evaluation settings, so a controlled comparison with a closely related recurrent-state quantization approach remains unavailable. I appreciate that DAMP’s implementation was unavailable for reproduction; nevertheless, this limits the conclusions that can be drawn about STEPQuant’s comparative accuracy and compression efficiency against concurrent approaches.