acceptodds
Under review as a conference paper at ICLR 2027

DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization

48 views

Abstract

Complex reasoning and agentic applications increasingly rely on long-context inference, where growing KV caches increase both memory usage and decoding overhead. Hybrid models reduce these costs by combining Softmax Attention with Gated DeltaNet (GDN) or Kimi Delta Attention (KDA), which maintain fixed-size recurrent states. These states are commonly stored in FP32 and consume substantial GPU memory, while their updates are limited by memory bandwidth. Quantization can reduce both storage footprint and memory traffic, but we find that uniform INT8 and FP8 degrade complex reasoning accuracy, while INT4 and NVFP4 collapse it to near zero. To our knowledge, this is the first study of post-training recurrent-state quantization for GDN and KDA. Our analysis reveals that outliers in GDN and KDA states are concentrated in particular key channels and value dimensions. Learned decay influences how much quantization error is retained. We find that largely the same GDN heads and KDA key channels exhibit slow decay across tasks. Based on these insights, we propose DAMP, which jointly considers quantization error and decay-based error retention to select high-risk key channels offline. Under a fixed storage budget, it retains these channels in FP16 and stores the remainder in INT8. We evaluate DAMP on Qwen3.6-35B, Kimi-Linear-48B and Kimi-K3 across six reasoning and code generation benchmarks. At 9.9 bits per state value, DAMP maintains average accuracy close to FP32. In SGLang, DAMP reduces recurrent-state storage by 69.1%, accelerates the recurrent-state update kernel by up to , and lowers full-model time per output token by up to 19.0%.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Efficient LLM Inference, Hybrid Attention, Recurrent-state Mixed-precision Quantization

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_gLN6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DAMP, a post-training mixed-precision quantization method for recurrent states in hybrid language models employing Gated DeltaNet (GDN) and Kimi Delta Attention (KDA). The authors observe that quantization errors are unevenly distributed across key channels and that learned decay affects the persistence of these errors during recurrent updates.

Motivated by these observations, DAMP introduces a channel-selection criterion that combines local quantization error with decay-based error retention. Channels identified as high-risk are preserved in FP16, while the remaining channels are quantized to INT8. The method further incorporates a fused CUDA kernel to reduce recurrent-state update overhead.

Experiments on Qwen3.6-35B, Kimi-Linear-48B, and Kimi-K3 demonstrate accuracy close to FP32 across several reasoning and code generation benchmarks. The authors report a 69.1% reduction in recurrent-state storage and up to a 19.0% reduction in decoding time per output token (TPOT), both relative to FP32 recurrent-state storage.

Critical strengths

1. Effective accuracy preservation under recurrent-state quantization.

The proposed method demonstrates promising results in preserving model accuracy while reducing recurrent-state storage precision. The experiments show that uniform low-bit quantization can substantially degrade reasoning performance, whereas selectively retaining certain channels in higher precision recovers much of the lost accuracy. The consistent improvements across multiple GDN/KDA-based models support the effectiveness of the proposed approach.

2. Consideration of temporal error retention in precision allocation.

Incorporating learned decay into channel selection is a reasonable and potentially useful extension of conventional error-aware mixed-precision quantization. Unlike independently quantized activations or cached tensors, recurrent states undergo repeated updates, allowing quantization errors to propagate across decoding steps. The proposed criterion attempts to account for this temporal characteristic, and the ablation results provide empirical evidence of its benefit over error-only channel selection.

Critical weaknesses

1. Insufficient evidence of practical significance, particularly for long-context deployment.

My primary concern is whether recurrent-state compression constitutes a sufficiently important optimization target in the long-context, large-scale serving scenarios emphasized by the paper.

Unlike attention KV caches, recurrent states have a fixed size independent of context length. As the context grows, attention KV caches may account for an increasingly large proportion of the inference memory footprint, potentially diminishing the relative benefit of recurrent-state compression. However, the paper does not adequately quantify the contribution of recurrent-state storage to total GPU memory consumption under representative long-context workloads.

This concern is further amplified by the choice of baseline. The headline 69.1% storage reduction is measured against FP32 recurrent-state storage. While FP32 is the reference implementation considered by the authors, FP16/BF16 are important practical alternatives. Relative to FP16, DAMP reduces recurrent-state storage by approximately 38.1%, with a potentially much smaller reduction in overall GPU memory consumption.

More importantly, the efficiency evaluation is not well aligned with the stated motivation. Section 6.3 evaluates decoding performance using only 256 input tokens and 64 output tokens, despite motivating the method through long-context reasoning and agentic workloads. These sequence lengths are considerably shorter than those typically associated with the targeted applications.

This discrepancy is particularly consequential because recurrent-state update costs are approximately independent of context length, whereas attention-related computation and KV-cache traffic can increase with sequence length. Consequently, short-context evaluations may overestimate the relative contribution of recurrent-state updates to end-to-end decoding latency, thereby presenting an overly favorable picture of the achievable acceleration.

Although the paper includes long-context accuracy evaluations, these do not establish the corresponding efficiency benefits. Furthermore, the headline TPOT improvements are reported relative to FP32 rather than directly emphasizing the incremental acceleration over optimized FP16/BF16 implementations.

Taken together, these limitations make it difficult to assess whether DAMP provides a practically meaningful improvement for the long-context deployment scenarios that motivate the work.

2. Limited methodological novelty beyond established mixed-precision quantization strategies.

The main methodological approach follows a well-established principle in quantization research: identify channels that are particularly sensitive to quantization, preserve them in higher precision, and apply more aggressive quantization to the remaining channels. Related strategies have been extensively explored for model weights, activations, attention mechanisms, KV caches, and recurrent architectures.

The primary distinction of DAMP appears to be the incorporation of learned decay into the channel-selection criterion. While this is a reasonable adaptation to recurrent-state dynamics, the underlying intuition that different channels have unequal sensitivity to quantization is not new.

Moreover, the incremental improvements attributable to decay-awareness appear relatively modest. For example, Table 4 reports improvements over error-only selection of 1.54, 0.71, and 0.68 percentage points on AIME, GPQA-Diamond, and LiveCodeBench, respectively. These results support the usefulness of incorporating decay information but do not, by themselves, establish a substantial methodological advance.

Overall, while I appreciate the effectiveness of the proposed method, its core design appears closely related to existing channel-wise mixed-precision quantization approaches. The paper would benefit from a clearer articulation and stronger empirical justification of the distinctive contribution of decay-aware selection.

Questions for authors

Q1. Practical significance of recurrent-state compression.

Could the authors provide a breakdown of total GPU memory consumption, including model weights, attention KV caches, active recurrent states, and cached recurrent-state checkpoints, under representative long-context serving workloads (e.g., 16K, 32K, 64K, and 128K tokens)?

In particular, what fraction of the total GPU memory footprint is attributable to recurrent states, and how much does DAMP reduce overall memory consumption relative to FP16/BF16 storage? Does this translate into a meaningful improvement in maximum serving concurrency or throughput under a fixed memory budget?

Q2. Representativeness of the efficiency evaluation.

What motivated the choice of 256 input tokens and 64 output tokens for the main decoding-efficiency experiments, given the paper's emphasis on long-context reasoning and agentic applications?

Could the authors provide end-to-end efficiency results for substantially longer contexts and generations, such as 16K–128K input tokens and 1K–4K output tokens, across representative batch sizes?

Of particular interest is how the proportion of decoding time spent on recurrent-state updates, and the resulting acceleration from DAMP, changes as context length increases.

Q3. End-to-end performance relative to FP16/BF16.

Could the authors explicitly quantify DAMP's end-to-end wall-clock acceleration relative to optimized FP16/BF16 recurrent-state storage, rather than FP32?

Although 16-bit baselines are included in the efficiency evaluation, the headline TPOT improvements are reported relative to FP32. Direct comparisons in terms of TPOT, total request latency, throughput, and peak GPU memory consumption would help establish the incremental practical benefit of DAMP over simpler 16-bit alternatives.

Q4. Distinctive contribution of decay-aware channel selection.

Could the authors further clarify the methodological distinction between DAMP and existing sensitivity-based or error-aware mixed-precision quantization approaches?

In particular, given the relatively modest improvements over error-only selection in Table 4, what evidence demonstrates that incorporating decay-based error retention provides a substantial advantage beyond conventional channel-wise precision allocation? A more detailed characterization of the benefit specifically attributable to decay-awareness would help assess the novelty and significance of the proposed method.

Reviewer_Zs9zOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper looks at quantizing the recurrent state in GDN/KDA hybrid LLMs, which is normally kept in FP32. The authors first show that uniform 8-bit state hurts long-generation reasoning a lot (Qwen3.6-35B on AIME 2026 goes from 85.5% to 18.5% with INT8) while RULER and MMLU-Pro barely move. They then observe that after value-side scaling and reordering (SR), most of the remaining INT8 error sits in a few key channels, and that effective decay is stable across tasks. DAMP scores each key channel offline by its quantization error weighted by cumulative squared decay over the next 256 steps (Eq. 15), keeps the top 16 channels per head in FP16 and the rest in INT8+SR, for 9.9 bits per value. Scales and permutations are folded into the projections and a fused kernel handles dequant/update/requant. Experiments cover Qwen3.6-35B, Kimi-Linear-48B and Kimi-K3 on reasoning, code and long-context benchmarks, plus kernel latency, TPOT in SGLang, and prefix-cache hit rate.

Critical strengths
  1. It works on models people actually serve. At 9.9 bits DAMP matches FP32/FP16 on all three models, where INT8+SR loses 20+ points on AIME and LCB for Qwen3.6 (Table 1). The fused kernel turns this into real gains (2.59x on the update kernel, 19% TPOT at batch 256, better prefix-cache hit rate in Table 3), and it runs on a 2.8T model in SGLang. That is the main reason I lean accept.

  2. The diagnostic result is worth having on its own. Tables 1 and 6 together show that RULER and MMLU-Pro completely miss this failure mode. Anyone evaluating state compression for linear-attention hybrids should know that.

Critical weaknesses
  1. I'm not convinced "decay-aware" is what makes it work. KhiK_{hi} is fixed per head, so head-level decay never affects which channels get picked. For GDN, where decay is shared within a head, Eq. 15 is basically error-only ranking with a mild reweighting. Appendix D more or less says this, but Table 8 doesn't include the Error-only row for Qwen3.6, which is the comparison that matters. For KDA (Table 4), DAMP beats Error-only by 1.5 / 0.7 / 0.7 points on AIME / GPQA-D / LCB with no error bars, and AIME has 30 problems. My guess is most of the gain comes from plain error-based mixed-precision channel protection, which is a known idea (e.g. MixKVQ). The score also ignores the delta-rule erasure term (I−βtktkt⊤)(I-\beta_t k_t k_t^\top) and how state error reaches the output through qtq_t, and there's no check that CuC_u actually predicts accumulated error.

  2. Missing matched-budget baselines, and the reference point flatters the results. DAMP spends 0.9 extra bits, but the only other 9.9-bit method compared is a different channel selector. INT8+SR with group size 16 (~10 bits) or stochastic rounding are obvious alternatives, and stochastic rounding targets the actual failure (bias accumulating over thousands of requantizations). Also, 69% storage saving, 2.59x and 19% TPOT are all vs FP32, but FP16 state is essentially lossless in Table 1. Against FP16 the storage saving is about 38%, and the speed gains over FP16 only appear in Fig. 4 plots. Still useful, but less impressive than the abstract makes it sound.

Questions for authors
  1. [Clarification] What does Error-only give for GDN on Qwen3.6 (the missing Table 8 row)? Can you add standard errors or multiple seeds for Tables 4 and 8?
  2. [Clarification] How do INT8+SR with group size 16, and INT8+SR with stochastic rounding, do on AIME 2026 and LCB-v6 for Qwen3.6? If either gets close to DAMP the case for the mixed-precision layout gets much weaker.
  3. [Clarification] What are the latency and TPOT gains vs FP16 state at each batch size, and which GPU was used for Fig. 1a, Fig. 4 and Table 3?
  4. [Clarification] Does CuC_u predict accumulated per-channel error ∥ΔSt,u∥2\|\Delta S_{t,u}\|^2 on held-out trajectories better than error-only ranking does?
  5. [Future work] Since the damage comes from long decoding, have you tried periodic higher-precision refresh or less frequent requantization instead of static channel protection?
1 comment

Thanks for the response and new experiments. They address most of my points. The group-16 result (65.60 on AIME) shows keeping a few channels in FP16 beats adding more scales, and the FP16-relative speedups answer my efficiency question. In the revision, please include the Qwen3.6 Error-only row, CIs, GPU details, and FP16 numbers next to the FP32 ones in the abstract (19% TPOT reads very differently from 1.8% to 7.4%).

Two remaining points:

First, the new numbers sharpen my framing concern. On Qwen3.6 AIME, Random to Error-only adds 15.5 points, while decay weighting adds just 1.7. Stochastic rounding alone also boosts INT8+SR from 63.98 to 77.89 at 9 bits with no extra storage. Please frame this as error-ranked channel protection with a decay refinement, rather than making decay-awareness the core idea. I'd also like to see DAMP + SR (which may allow a smaller K_hi) and the LCB-v6 numbers for the SR and group-16 rows, as LCB was where INT8+SR dropped most.

Second, on the CIs: resampling generations within each problem misses problem-level variance, which matters with only 30 AIME problems. Since all selectors run on the same problems, a paired bootstrap over problems for DAMP vs. Error-only would be cleaner. Right now, the Kimi-Linear GPQA-D and LCB intervals overlap.

Overall, I remain positive and will keep my score.

Reviewer_NLn3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DAMP, a post-training quantization method for the recurrent states of GDN- and KDA-based hybrid attention transformer models. This paper observe that quantization error is concentrated in a subset of key channels and that learned decay affects how long this error persists across recurrent updates. DAMP applies scaling and reordering along the value dimension to reduce magnitude imbalance within quantization groups, then combines local quantization error with decay-based error retention to select key channels for FP16 storage while the remaining channels are stored in INT8. Across three models, it reports accuracy close to FP32 at an average of 9.9 bits per state value.

Critical strengths
  • As the adoption of hybrid attention architectures increases, the paper addresses the timely issue.
  • The paper formulates error propagation mathematically, providing an intuitive explanation of how learned decay affects the accumulation of quantization errors.
  • The proposed method is implemented in a serving framework, and its performance is evaluated in terms of not only operator latency but also end-to-end TPOT, demonstrating how the performance improvements translate into overall inference performance.
  • The paper also evaluates multi-turn serving scenarios by comparing TTFT and prefix-cache hit rates, going beyond single decoding requests.
Critical weaknesses
  • The novelty is limited. The existence of outliers in specific channels is a phenomenon that has been widely observed in KV cache quantization and LLM activation quantization works, although not necessarily in recurrent states. Accordingly, techniques such as channel-wise quantization and scaling/reordering have already been extensively developed.
  • As shown in Table 4 (and Table 8), the performance differences resulting from different high-precision key channel selection criteria do not seem particularly significant. This may suggest that decay-aware scoring is not very effective. Without the decay-aware consideration, the novelty specific to recurrent state quantization could be substantially weakened.
  • In general, KV cache is expected to be a more significant bottleneck in long-context scenarios, whereas the paper argues that GDN states are the main bottleneck. It would be helpful to report the proportion of overall system latency attributable to GDN across different batch sizes and with or without prefix caching.
  • The paper assumes extremely large batch sizes, but such large batches may not be practically achievable in real serving systems [1, 2]. A more realistic setting, such as a batch size of around 8, would be expected. However, at smaller batch sizes, the GDN operator speedup appears to have little effect (Figure 4).
  • Compared to FP16, it seems that the speedup is limited. Since FP16 is an easily applicable alternative, it should be considered the practical baseline.
  • Calibration robustness needs to be evaluated. How much does performance differ when the calibration and inference domains/sequence lengths match versus when they do not?

[1] Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve, OSDI’24

[2] ReSET: Accurate Latency-Critical NVFP4 Reasoning via Step-Aware Temperature Scaling, NeurIPS’26

Questions for authors

Please see Weaknesses

Reviewer_wEq5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes DAMP, a post-training method for quantizing recurrent states in GDN- and KDA-based language models. It combines local quantization error with decay-based error retention to select a small subset of key channels for FP16 storage, while storing the remaining channels in INT8. Empirically, DAMP generally maintains accuracy close to FP32 while reducing recurrent-state storage and decoding latency across the evaluated models.

Critical strengths
  1. DAMP uses a mechanistically motivated criterion for precision allocation, combining local quantization error with decay-based retention to prioritize channels with large, persistent errors.
  2. Offline channel selection, permutations absorbed into model parameters, and a fused update kernel keep runtime overhead low and translate storage savings into inference speedups. Stable channel selection across calibration subsets further supports this static design.
Critical weaknesses
  1. The benefit of decay-awareness is not fully isolated. For GDN, decay only temporally reweights local errors under the fixed per-head precision budget, yet Table 8 omits the Error-only baseline needed to assess this benefit. For KDA, gains over Error-only are relatively modest.
  2. The risk score in Eq. (15) approximates error retention using decay alone, omitting the key-dependent delta-rule correction in Eq. (3). While downstream results support its usefulness, the paper does not directly validate whether this approximation preserves channel rankings by accumulated-error risk. Comparing the predicted rankings with the propagated effects of channel-wise quantization perturbations under the full recurrence would strengthen the mechanistic justification.
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.