- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DAMP, a post-training mixed-precision quantization method for recurrent states in hybrid language models employing Gated DeltaNet (GDN) and Kimi Delta Attention (KDA). The authors observe that quantization errors are unevenly distributed across key channels and that learned decay affects the persistence of these errors during recurrent updates.
Motivated by these observations, DAMP introduces a channel-selection criterion that combines local quantization error with decay-based error retention. Channels identified as high-risk are preserved in FP16, while the remaining channels are quantized to INT8. The method further incorporates a fused CUDA kernel to reduce recurrent-state update overhead.
Experiments on Qwen3.6-35B, Kimi-Linear-48B, and Kimi-K3 demonstrate accuracy close to FP32 across several reasoning and code generation benchmarks. The authors report a 69.1% reduction in recurrent-state storage and up to a 19.0% reduction in decoding time per output token (TPOT), both relative to FP32 recurrent-state storage.
Critical strengths
1. Effective accuracy preservation under recurrent-state quantization.
The proposed method demonstrates promising results in preserving model accuracy while reducing recurrent-state storage precision. The experiments show that uniform low-bit quantization can substantially degrade reasoning performance, whereas selectively retaining certain channels in higher precision recovers much of the lost accuracy. The consistent improvements across multiple GDN/KDA-based models support the effectiveness of the proposed approach.
2. Consideration of temporal error retention in precision allocation.
Incorporating learned decay into channel selection is a reasonable and potentially useful extension of conventional error-aware mixed-precision quantization. Unlike independently quantized activations or cached tensors, recurrent states undergo repeated updates, allowing quantization errors to propagate across decoding steps. The proposed criterion attempts to account for this temporal characteristic, and the ablation results provide empirical evidence of its benefit over error-only channel selection.
Critical weaknesses
1. Insufficient evidence of practical significance, particularly for long-context deployment.
My primary concern is whether recurrent-state compression constitutes a sufficiently important optimization target in the long-context, large-scale serving scenarios emphasized by the paper.
Unlike attention KV caches, recurrent states have a fixed size independent of context length. As the context grows, attention KV caches may account for an increasingly large proportion of the inference memory footprint, potentially diminishing the relative benefit of recurrent-state compression. However, the paper does not adequately quantify the contribution of recurrent-state storage to total GPU memory consumption under representative long-context workloads.
This concern is further amplified by the choice of baseline. The headline 69.1% storage reduction is measured against FP32 recurrent-state storage. While FP32 is the reference implementation considered by the authors, FP16/BF16 are important practical alternatives. Relative to FP16, DAMP reduces recurrent-state storage by approximately 38.1%, with a potentially much smaller reduction in overall GPU memory consumption.
More importantly, the efficiency evaluation is not well aligned with the stated motivation. Section 6.3 evaluates decoding performance using only 256 input tokens and 64 output tokens, despite motivating the method through long-context reasoning and agentic workloads. These sequence lengths are considerably shorter than those typically associated with the targeted applications.
This discrepancy is particularly consequential because recurrent-state update costs are approximately independent of context length, whereas attention-related computation and KV-cache traffic can increase with sequence length. Consequently, short-context evaluations may overestimate the relative contribution of recurrent-state updates to end-to-end decoding latency, thereby presenting an overly favorable picture of the achievable acceleration.
Although the paper includes long-context accuracy evaluations, these do not establish the corresponding efficiency benefits. Furthermore, the headline TPOT improvements are reported relative to FP32 rather than directly emphasizing the incremental acceleration over optimized FP16/BF16 implementations.
Taken together, these limitations make it difficult to assess whether DAMP provides a practically meaningful improvement for the long-context deployment scenarios that motivate the work.
2. Limited methodological novelty beyond established mixed-precision quantization strategies.
The main methodological approach follows a well-established principle in quantization research: identify channels that are particularly sensitive to quantization, preserve them in higher precision, and apply more aggressive quantization to the remaining channels. Related strategies have been extensively explored for model weights, activations, attention mechanisms, KV caches, and recurrent architectures.
The primary distinction of DAMP appears to be the incorporation of learned decay into the channel-selection criterion. While this is a reasonable adaptation to recurrent-state dynamics, the underlying intuition that different channels have unequal sensitivity to quantization is not new.
Moreover, the incremental improvements attributable to decay-awareness appear relatively modest. For example, Table 4 reports improvements over error-only selection of 1.54, 0.71, and 0.68 percentage points on AIME, GPQA-Diamond, and LiveCodeBench, respectively. These results support the usefulness of incorporating decay information but do not, by themselves, establish a substantial methodological advance.
Overall, while I appreciate the effectiveness of the proposed method, its core design appears closely related to existing channel-wise mixed-precision quantization approaches. The paper would benefit from a clearer articulation and stronger empirical justification of the distinctive contribution of decay-aware selection.
Questions for authors
Q1. Practical significance of recurrent-state compression.
Could the authors provide a breakdown of total GPU memory consumption, including model weights, attention KV caches, active recurrent states, and cached recurrent-state checkpoints, under representative long-context serving workloads (e.g., 16K, 32K, 64K, and 128K tokens)?
In particular, what fraction of the total GPU memory footprint is attributable to recurrent states, and how much does DAMP reduce overall memory consumption relative to FP16/BF16 storage? Does this translate into a meaningful improvement in maximum serving concurrency or throughput under a fixed memory budget?
Q2. Representativeness of the efficiency evaluation.
What motivated the choice of 256 input tokens and 64 output tokens for the main decoding-efficiency experiments, given the paper's emphasis on long-context reasoning and agentic applications?
Could the authors provide end-to-end efficiency results for substantially longer contexts and generations, such as 16K–128K input tokens and 1K–4K output tokens, across representative batch sizes?
Of particular interest is how the proportion of decoding time spent on recurrent-state updates, and the resulting acceleration from DAMP, changes as context length increases.
Q3. End-to-end performance relative to FP16/BF16.
Could the authors explicitly quantify DAMP's end-to-end wall-clock acceleration relative to optimized FP16/BF16 recurrent-state storage, rather than FP32?
Although 16-bit baselines are included in the efficiency evaluation, the headline TPOT improvements are reported relative to FP32. Direct comparisons in terms of TPOT, total request latency, throughput, and peak GPU memory consumption would help establish the incremental practical benefit of DAMP over simpler 16-bit alternatives.
Q4. Distinctive contribution of decay-aware channel selection.
Could the authors further clarify the methodological distinction between DAMP and existing sensitivity-based or error-aware mixed-precision quantization approaches?
In particular, given the relatively modest improvements over error-only selection in Table 4, what evidence demonstrates that incorporating decay-based error retention provides a substantial advantage beyond conventional channel-wise precision allocation? A more detailed characterization of the benefit specifically attributable to decay-awareness would help assess the novelty and significance of the proposed method.
Thanks for the response and new experiments. They address most of my points. The group-16 result (65.60 on AIME) shows keeping a few channels in FP16 beats adding more scales, and the FP16-relative speedups answer my efficiency question. In the revision, please include the Qwen3.6 Error-only row, CIs, GPU details, and FP16 numbers next to the FP32 ones in the abstract (19% TPOT reads very differently from 1.8% to 7.4%).
Two remaining points:
First, the new numbers sharpen my framing concern. On Qwen3.6 AIME, Random to Error-only adds 15.5 points, while decay weighting adds just 1.7. Stochastic rounding alone also boosts INT8+SR from 63.98 to 77.89 at 9 bits with no extra storage. Please frame this as error-ranked channel protection with a decay refinement, rather than making decay-awareness the core idea. I'd also like to see DAMP + SR (which may allow a smaller K_hi) and the LCB-v6 numbers for the SR and group-16 rows, as LCB was where INT8+SR dropped most.
Second, on the CIs: resampling generations within each problem misses problem-level variance, which matters with only 30 AIME problems. Since all selectors run on the same problems, a paired bootstrap over problems for DAMP vs. Error-only would be cleaner. Right now, the Kimi-Linear GPQA-D and LCB intervals overlap.
Overall, I remain positive and will keep my score.