acceptodds
Under review as a conference paper at ICLR 2027

DASC: Decay-Aware State Compression for Hybrid Linear-Attention Serving

Abstract

Hybrid linear-attention architectures have recently scaled to large open-weight models, offering quality competitive with full attention while substantially reducing key/value (KV) cache growth. However, their in-place recurrent-state updates complicate cache management: prefix reuse requires state checkpoints alongside full-attention KV, while storing state checkpoints in full increases memory pressure, leading to more evictions and repeated prefill. By analyzing the decay structure of Gated DeltaNet (GDN) and Kimi Delta Attention (KDA), we find that different heads and channels retain prefix information over markedly different timescales, which we term retention horizons. This variation suggests substantial compression potential in persistent state checkpoints. Building on this observation, we introduce Decay-Aware State Compression (DASC), which derives retention horizons from model weights, packs selected long-horizon units into ragged state checkpoints, and optionally balances their storage across tensor-parallel ranks. On reuse, DASC either zero-fills omitted units or refreshes them from a bounded suffix with additional compute cost. We evaluate DASC on Kimi-Linear, Qwen3-Next, and Kimi-K3, covering model quality and serving performance. Across the evaluated retrieval and reasoning tasks, conservative configurations maintain quality close to full state checkpoint caching, while suffix refresh recovers accuracy at more aggressive compression. Implemented in SGLang, DASC provides – state checkpoint compression at representative operating points, reducing mean Time to First Token (TTFT) by 16.1–42.0% and mean end-to-end (E2E) latency by 4.9–10.2%.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
hybrid linear attention, prefix caching, recurrent state compression, efficient LLM inference, tensor parallelism

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sXGvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Decay-Aware State Compression (DASC) for recurrent-state checkpoints in hybrid linear-attention prefix caches. DASC estimates static retention horizons from learned decay parameters and a fixed representative gate input, retaining long-horizon GDN heads or KDA channels. It combines this selection with ragged storage, optional tensor-parallel placement balancing, and a two-tier cache implementation in SGLang. On reuse, omitted units are either zero-filled or approximately reconstructed through bounded suffix replay; full-attention KV remains unchanged. The evaluation covers Kimi-Linear, Qwen3-Next, and Kimi-K3, using retrieval, reasoning, conversational-memory, and WildChat serving workloads. At selected conservative configurations, the paper reports 1.32–2.22× state-checkpoint compression, 16.1–42.0% lower mean TTFT, and 4.9–10.2% lower mean end-to-end latency, alongside quality estimates close to the dense baseline on separately evaluated tasks.

Critical strengths

1. Practical systems contribution. DASC addresses checkpoint memory pressure with an integrated SGLang implementation. Matched-memory experiments include temporary buffers and report latency across all requests, providing meaningful evidence of serving benefits.

2. Validation across architectures and tasks. The evaluation covers GDN and KDA across three models, extends beyond simple retrieval, and includes count-matched Random selection. Disabling dense spill in quality experiments prevents compression from being bypassed.

Critical weaknesses

1. Retention duration does not directly establish predictive importance. Short-horizon units may contain recent information essential to the answer. Correlations with observed decay and state magnitude do not establish the error caused by omitting those units. Although the authors acknowledge this distinction, Random comparisons provide only partial validation. Direct omission-impact analysis and stronger selection baselines are needed to justify horizon as the central selection criterion.

2. The practical quality–latency advantage remains incompletely characterized. Quality and latency are measured on separate workloads and cache protocols, leaving their joint behavior under successive approximate reuse unclear. Serving evaluation uses four-turn traces capped at 16k tokens, limiting evidence across reuse and memory-pressure conditions despite the additional Kimi-K3 queueing tests. Furthermore, the paper evaluates DASC+INT8 but lacks an INT8-only quality and serving comparison, leaving DASC’s incremental benefit over quantization unresolved.

Questions for authors
  1. Table 1(b) validates the relationship between static and observed decay, but does not directly measure the consequences of omission. Does removing short-horizon units cause less prediction loss or task degradation than removing matched groups of long-horizon units? A comparison against a fixed, calibration-based state-magnitude selector at matched per-layer storage budgets would also help establish the specific value of the horizon criterion.

  2. Does the warmup–replay quality evaluation include checkpoints produced after an earlier approximate cache load and subsequently reused again? Repeated reuse may propagate approximation errors, while quality and serving are currently evaluated under different workloads and cache policies. Evidence measuring both quality and latency along such reuse chains would clarify whether the reported trade-off holds in multi-turn serving.

  3. How strongly do the reported gains depend on prefix reuse and state-cache pressure? A breakdown of compressed hits, dense hits, misses, avoided recomputation, and checkpoint-management overhead in the existing runs would help explain when increased cache capacity outweighs packing, communication, or refresh costs. This would make the applicability of the results clearer without requiring evaluation of every deployment setting.

  4. Appendix E.1 reports 3.88× checkpoint compression from INT8 alone, while Table 3 evaluates DASC+INT8 without an INT8-only quality arm. How do these alternatives compare in quality, TTFT, and end-to-end latency under the same state-memory budget and cache-management policy? This comparison would establish DASC’s incremental practical benefit beyond quantization, rather than only its benefit over dense storage.

4 comments

Thank you for the questions. We add direct selection and reuse evidence, clarify the original serving results, and report INT8 quality and capacity.

Q1. Does horizon selection predict omission impact?

We repeated the offline calibration and conducted a W64 omission ablation on Kimi-Linear. We compare the normal DASC mask with its strict complement, which retains the channels DASC discards and discards those it retains. All arms use the same questions and the paper's sampling, generation, and scoring settings, excluding warmup (AIME: 30×31 scored replays; HMMT: 33×31; GPQA: 198×15).

MethodTemporal channels retainedAIME2026 (%)HMMT 2026 Feb (%)GPQA-Diamond (%)
Dense100%62.2643.7963.60
DASC W6455.86%62.6944.1863.84
W64 strict complement44.14%43.3325.7156.09

The complement is 19.36/18.47/7.75 points lower on AIME2026/HMMT/GPQA-Diamond, using the displayed scores. This directly measures the task impact of changing which channels are omitted. The comparison is not budget-matched (44.14% versus 55.86% retained), so it does not isolate ranking at equal storage; a matched-budget state-magnitude selector remains untested.

Q2. Successive approximate reuse

The new tests reuse checkpoints produced after earlier approximate loads, extending the original warmup–replay protocol.

Successive recovery. Kimi-Linear DASC-NR W16 stays within 0.28 points of same-depth Dense on 16K RULER through 16 restores (13 subtasks × 100 independent instances):

Actual checkpoint restoresDenseDASC-NR W16DASC − Dense (pp)
0: cold full prefill93.02——
493.9494.21+0.27
894.1094.05−0.06
1693.7493.81+0.07

Each chain appends segments of the same final input, repeatedly saves and restores checkpoints, then scores only the final answer. Intermediate one-token outputs are excluded from history. Same-depth Dense controls for segmentation; direct-predecessor checkpoint hits are 100% on all eight TP ranks. No growing additional macro-average loss is observed within this range.

Long-context multi-turn quality. SCBench KV/VT/English-QA use 100/90/23 conversations, 500/450/118 answers, and mean initial lengths of 122,850/124,554/96,826 tokens. Each method's actual answers enter subsequent history:

SubtaskDense overallDASC overallΔ (pp)Dense later turnsDASC later turnsΔ (pp)
KV100.00100.000.00100.00100.000.00
VT72.7672.04−0.7182.6181.72−0.89
QA33.9033.05−0.8534.7433.68−1.05

Q3. Prefix reuse, cache pressure, and serving benefit

Our primary serving endpoints are TTFT and E2E latency, including the applicable cache-management, communication, replay, and miss costs; E2E additionally includes decoding (§5.3; Appendix C.3). Intermediate hit/eviction metrics help explain these endpoints. We therefore provide the original Kimi-K3 breakdown below.

All arms use the same allocated state-memory budget (native states, compressed checkpoints, and temporary buffers), unchanged KV allocation, and the paper's common trace and global-LRU policy. The trace has 256 four-turn conversations; headline latency uses all requests, not only hits.

ArmHit: all turns (%)EvictionsTTFT mean / P95 (ms)Δ mean TTFTE2E mean / P95 (s)Δ mean E2E
dense_native54.921,732257.80 / 551.86—11.330 / 26.409—
nr_off_batch_w6468.921,634216.39 / 459.29−16.06%10.647 / 24.350−6.03%
nr_tp_batch_w6469.271,630218.49 / 442.09−15.25%10.745 / 24.608−5.17%
wr_off_batch_w6468.891,634304.23 / 591.47+18.01%11.994 / 27.851+5.86%
wr_tp_batch_w6469.241,630306.35 / 593.39+18.83%12.036 / 27.885+6.23%
nr_off_batch_w12870.991,583215.19 / 452.90−16.53%10.607 / 24.116−6.39%
nr_tp_batch_w12871.271,576215.37 / 437.45−16.46%10.714 / 24.370−5.44%
wr_off_batch_w12871.171,583299.18 / 555.07+16.05%12.009 / 27.727+5.99%
wr_tp_batch_w12871.271,575295.85 / 563.25+14.76%11.953 / 27.492+5.50%
nr_off_batch_w51273.461,401204.15 / 422.77−20.81%10.451 / 23.439−7.76%
nr_tp_batch_w51273.461,374201.36 / 416.42−21.89%10.477 / 23.737−7.53%
wr_off_batch_w51273.461,401292.34 / 524.17+13.40%11.907 / 27.593+5.09%
wr_tp_batch_w51273.461,374299.70 / 576.04+16.25%11.913 / 27.487+5.14%

Hit averages all four turns; latency changes are relative to Dense. NR-Off-W64 raises Hit by 14.00 points and reduces mean TTFT/E2E by 16.06%/6.03%. At W512, NR-Off/TP reach 73.46% Hit and lower TTFT by 20.81%/21.89%; WR has similar hits but higher total latency, consistent with additional recovery overhead. Thus extra capacity helps when it avoids recomputation, but higher hit rate alone does not guarantee lower latency.

Q4. INT8-only versus DASC+INT8

Quantization is complementary: it reduces bits per value, while DASC reduces retained units. Only cached temporal state is quantized; running state stays FP32, convolution state BF16, and weights/FA-KV are unchanged. The supplementary INT8-only aggregates use the same questions and generation/scoring settings as Table 3; identical warmup/replay accounting is not assumed.

BenchmarkDenseDASC-NR W16INT8-onlyDASC-NR W16 + INT8
AIME202660.6560.3260.6562.15
HMMT 2026 Feb37.5438.6137.8337.15
IMOAnswerBench27.8326.7528.3327.75
GPQA-Diamond61.8561.3161.9561.28
MMLU-Pro67.5067.3268.6467.48

Fixed-budget capacity. With a 5.196 GiB/GPU state-plus-metadata ceiling, unchanged 15.099 GiB/GPU KV, and the same native-state reserve for compressed arms, the TP2 byte/slot accounting gives:

MethodNative slotsSeparate compressed slots
Dense256, shared active/cache—
DASC-NR128329
INT8-only128445
DASC-NR+INT81281,019

DASC+INT8 supports 1,019 versus 445 compressed slots (2.29×) at this budget. This is a capacity benefit, not a predicted latency speedup. We limit this comparison to benchmark quality and budget/slot accounting; no further performance runs are planned.

Thank you for the detailed response and the additional experiments. The new results on repeated checkpoint reuse and long-context tasks helped address several of my concerns. I also appreciate the clarifications in the revised paper. I will keep my rating at 3.

Thank you very much for your positive feedback and for taking the time to review our revised paper and rebuttal. Thank you again for your time and consideration.

Reviewer_4SmqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DASC, a decay-aware compression method for recurrent-state checkpoints in hybrid linear-attention models. It observes that Gated DeltaNet and Kimi Delta Attention heads or channels have heterogeneous retention horizons, and derives input-independent horizons from model decay weights to select long-lived state units. DASC stores these units in ragged checkpoints, optionally balances them across tensor-parallel ranks, and supports either zero-filling omitted units or refreshing them from a bounded suffix at reuse time. The method is implemented in SGLang and evaluated on Qwen3-Next, Kimi-Linear, and Kimi-K3 using RULER retrieval tasks, mathematical and general reasoning benchmarks, conversational memory, and WildChat-1M serving traces. Results show that conservative decay-aware configurations remain close to dense checkpoint quality, while more aggressive settings trade accuracy for substantially higher capacity. Representative operating points achieve 1.32–2.22× checkpoint compression, reduce mean TTFT by 16.1–42.0% and end-to-end latency by 4.9–10.2%, and suffix refresh recovers additional accuracy at increased compute cost.

Critical strengths

DASC identifies a practically important and previously underexplored source of redundancy in hybrid linear-attention prefix caches: different recurrent-state heads or channels have different retention horizons. Converting model decay parameters into an input-independent selection rule is simple, architecture-aware, and compatible with existing serving systems, while ragged storage and tensor-parallel balancing make the idea implementable.

The evaluation connects compression quality to real serving outcomes. The method is tested across three hybrid models, retrieval, reasoning, and conversational-memory tasks, as well as WildChat serving traces with matched HBM budgets. The reported near-dense quality at conservative settings, 1.32–2.22× checkpoint compression, and 16.1–42.0% TTFT reductions provide meaningful evidence of practical utility.

Critical weaknesses

DASC compresses only the recurrent-state portion of the cache; full-attention KV remains unchanged and is excluded from the reported compression ratios. Consequently, the 1.32–2.22× state-checkpoint capacity gains may translate into much smaller reductions in total hybrid-cache memory, especially when full-attention KV is a substantial fraction of the footprint. The paper does not provide a clear breakdown of total memory savings across context lengths and workloads.

The central weight-only horizon heuristic is evaluated on only three model families and uses fixed representative gate inputs and thresholds. Although static horizons correlate with observed decay, this does not establish that they identify task-relevant information, and quality degrades at more aggressive cutoffs. The method therefore has limited evidence of robustness to input-dependent decay, other hybrid architectures, or broader workloads; suffix refresh also remains an approximate recovery mechanism rather than an exact reconstruction.

Questions for authors
  1. Since DASC leaves full-attention KV unchanged, what are the resulting savings in total HBM usage, including full-attention KV, recurrent states, allocator overhead, and tensor-parallel communication? Please report these values across context lengths rather than only state-checkpoint compression ratios.

  2. How sensitive are the results to the fixed representative gate inputs, (ϵ\epsilon), and (Wmax⁡W_{\max})? Were these choices selected on a validation set, and how would an input-dependent horizon or a learned selection baseline compare under the same memory budget?

  3. The quality results often use N=30N=30 instances per subtask–length setting, and many DASC values are reported as point estimates. Can the authors provide confidence intervals or multiple-seed results for the main quality comparisons, particularly at aggressive compression ratios?

  4. WildChat serving enables the two-tier cache with dense spill, whereas the quality experiments disable dense spill. How much of the reported TTFT and E2E improvement comes from DASC itself versus differences in cache admission, spill, hit rate, and eviction behavior? Can all systems be compared under identical cache-hit rates and total HBM budgets?

  5. DASC-WR refreshes omitted units from a zero-initialized suffix and is explicitly approximate. What are the replay lengths and additional FLOPs for each reported operating point, and how do quality and latency change when the suffix budget is varied independently of (Wmax⁡W_{\max})?

15 comments

Thank you for the detailed questions and careful reading and helpful feedback. We first clarify the evidence already in the paper, then provide the additional results.

Q1. Total memory savings

Hybrid attention reduces long-context KV growth, but fixed-size recurrent checkpoints can remain a substantial storage burden for short prefixes. DASC targets this overhead. It is orthogonal to full-attention KV compression and can be combined with KV quantization; that separate compression target is outside this paper's scope.

At equal checkpoint count n and unchanged KV footprint K, compare n·S(Dense) + K with n·S(DASC) + K, rather than equal-sized preallocated pools. For Kimi-Linear W16 NR, TP2, no balancing, and 32 independent prefixes, complete payload per checkpoint falls from 20.703 to 7.923 MiB on the most-loaded rank:

ContextDense state + KV (MiB/rank)DASC state + KV (MiB/rank)Saving
1K914.500505.54744.72%
4K1670.5001261.54724.48%
8K2678.5002269.54715.27%
16K4694.5004285.5478.71%
32K8726.5008317.5474.69%
64K16790.50016381.5472.44%

These are cache-payload savings, not whole-device HBM reductions. Weights, active states, allocator reservations, and workspaces enter total HBM separately (Appendix C.1). Off adds no placement communication; TP balancing communicates at checkpoint store/load boundaries (§4.3).

Q2. Calibration and robustness

The original paper specifies both the gate references and their evaluation (§§3.3–3.4, Appendix A.2). Qwen3-Next/Kimi-Linear use a_s = -0.3; their 32 ShareGPT prefixes validate horizons and do not construct masks. Kimi-K3 uses 16 ShareGPT prefixes to calibrate a_s = 0.8 and 16 held-out prefixes for validation. No downstream test set is used for calibration. Table 1(b) already reports median static/empirical-horizon correlations of 0.801/0.856/0.895 across these models.

The calibration can be updated offline while DASC's horizon-based selection rule remains unchanged and the resulting mask stays fixed during serving; strong agreement with observed decay and near-baseline quality across five benchmarks support the effectiveness of this approach.

The paper fixes ε = 10⁻³ and sweeps W_max (Tables 2, 5–6); count-matched Random uses five mask seeds. No validation-based threshold-selection rule is reported. To further test offline calibration flexibility:

For Kimi-Linear, we extend the offline scalar reference to a fixed a_mask ∈ R^(20×32×128) table, fitted on 16 calibration prefixes and checked on 16 held-out prefixes, without downstream test-set calibration. The table is frozen during serving; DASC's selection rule and the model's input-dependent gates are unchanged. At W64:

PhaseSpearman ↑Raw-horizon Pearson ↑Keep/drop agreement ↑False drops ↓
Prefill0.99440.983794.67%3.51%
Decode0.98580.945390.22%7.45%

Correlations are medians over 320 prefix–layer pairs; Pearson uses raw horizons. Agreement compares H > 64 decisions; false drops use all channel–prefix pairs as denominator. These are decay diagnostics, not direct importance measurements.

Downstream quality with the newly calibrated table. At W64, DASC retains 55.86% of temporal channels and achieves 1.744× full-checkpoint compression (42.65% saving) on Kimi-Linear:

BenchmarkBaselineDASC-W64Difference (pp)
AIME202662.2662.69+0.43
HMMT43.7944.18+0.39
GPQA Diamond63.6063.84+0.24
LiveCodeBench58.1258.31+0.19
LongBench44.9644.23−0.73

These point estimates stay within 0.73 points of the baseline across all five tasks. They support flexibility in offline calibration while DASC's selection remains request-independent; this is not a scalar/epsilon sensitivity sweep or an online/learned-selector comparison at matched memory.

Q3. Sample size and confidence intervals

We chose N=30 for RULER to balance available compute across the overall three-model, multi-benchmark evaluation. The original RULER sweep covers Qwen3-Next and Kimi-Linear, with 13 subtasks and multiple lengths/cutoffs (§5.1, Appendix C.2). To strengthen the uncertainty analysis, we completed N=100 per subtask–length setting and added 128K on Kimi-Linear TP8:

Method4K8K16K128KFour-length mean [95% CI]
Dense95.7394.5793.3087.6392.81 [92.31, 93.30]
DASC-NR W1695.4894.1493.6387.5592.70 [92.20, 93.19]
DASC-NR W6495.2193.6893.3287.3392.39 [91.88, 92.89]
DASC-NR W12894.9093.3593.1287.1292.12 [91.61, 92.63]
DASC-NR W51293.1092.4291.2485.6390.60 [90.05, 91.14]
DASC-NR W102492.7991.8491.3084.9290.21 [89.65, 90.77]

Each instance has one unscored warmup and three greedy replays. CIs use 50,000 stratified paired bootstrap replicates, keeping three replays together and equally weighting subtasks/lengths. The independent sample size is 100, not 300. Paired differences quantify losses at aggressive cutoffs:

MethodMean difference vs Dense (percentage points)95% paired CI
DASC-NR W16−0.11[−0.27, +0.05]
DASC-NR W64−0.42[−0.70, −0.15]
DASC-NR W128−0.69[−0.98, −0.40]
DASC-NR W512−2.21[−2.60, −1.83]
DASC-NR W1024−2.60[−3.02, −2.17]

These CIs describe input-sampling uncertainty for the fixed run. Table 2 already includes multiple generation seeds (31/31/3/15/7 measured samples for AIME/HMMT/IMO/GPQA/MMLU-Pro); this differs from the new instance-level CIs and the five-Random-mask SD.

Q4. Cache policies, hit rates, and memory fairness

DASC targets inference performance, so our primary endpoints are TTFT and E2E latency. They include the applicable cache-management, communication, recovery, and miss costs; E2E also includes decoding (§5.3; Appendix C.3). Hit rates and evictions are intermediate diagnostics. To address your attribution question, we provide their breakdown alongside latency from the original Kimi-K3 experiments below.

The paper matches allocated state memory, including both tiers and temporary buffers (Appendix C.1/C.3):

B(Dense, native) + B(Dense, tmp) = B(DASC, native) + B(DASC, compressed) + B(DASC, tmp).

Model/KV allocations, traces, and run seeds are shared. A global LRU governs both cached representations. Quality experiments disable dense spill so loads cannot bypass compression; serving enables it to use spare native slots. TTFT/E2E are measured over all requests, including cache misses.

ArmHit: all turns (%)EvictionsTTFT mean / P95 (ms)Δ mean TTFTE2E mean / P95 (s)Δ mean E2E
dense_native54.921,732257.80 / 551.86—11.330 / 26.409—
nr_off_batch_w6468.921,634216.39 / 459.29−16.06%10.647 / 24.350−6.03%
nr_tp_batch_w6469.271,630218.49 / 442.09−15.25%10.745 / 24.608−5.17%
wr_off_batch_w6468.891,634304.23 / 591.47+18.01%11.994 / 27.851+5.86%
wr_tp_batch_w6469.241,630306.35 / 593.39+18.83%12.036 / 27.885+6.23%
nr_off_batch_w12870.991,583215.19 / 452.90−16.53%10.607 / 24.116−6.39%
nr_tp_batch_w12871.271,576215.37 / 437.45−16.46%10.714 / 24.370−5.44%
wr_off_batch_w12871.171,583299.18 / 555.07+16.05%12.009 / 27.727+5.99%
wr_tp_batch_w12871.271,575295.85 / 563.25+14.76%11.953 / 27.492+5.50%
nr_off_batch_w51273.461,401204.15 / 422.77−20.81%10.451 / 23.439−7.76%
nr_tp_batch_w51273.461,374201.36 / 416.42−21.89%10.477 / 23.737−7.53%
wr_off_batch_w51273.461,401292.34 / 524.17+13.40%11.907 / 27.593+5.09%
wr_tp_batch_w51273.461,374299.70 / 576.04+16.25%11.913 / 27.487+5.14%

Hit averages all four turns; latency changes are relative to Dense. NR-Off-W64 increases Hit from 54.92% to 68.92%, with 16.06% lower TTFT and 6.03% lower E2E. WR has similar hits but higher latency, consistent with additional recovery overhead. These are integrated-system gains at matched memory;

Q5. Suffix refresh: quality and cost

Fig. 4(d,h,l) compares NR and WR quality and latency (§5.4). At W128, WR improves all five task estimates by 0.8–7.5 points over NR; the MMLU-Pro sweep reaches 66.59–68.00% versus Dense's 67.50%. Fig. 4(l) pairs MMLU-Pro accuracy with WildChat TTFT from separate workloads; Q4's K3 rows likewise show higher total latency with WR.

WR is explicitly approximate (§4.2): zero-initialized suffix replay cannot reconstruct omitted pre-window history. Eq. (7) bounds replay by W_max and prefix length (upper bounds of 16/64/128/512 tokens, not measured replay lengths); NR has no suffix-replay cost. The reported sweep couples cutoff and replay budget; it does not provide an independent suffix-budget sweep or per-point FLOP counts.

Reviewer_fyx5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DASC to reduce the memory footprint of recurrent-state checkpoints used for prefix caching in hybrid attention models. DASC estimates static retention horizons from the model’s decay parameters and selectively retains long-horizon state units at the head level for GDN and the channel level for KDA. It combines a compact storage layout with optional tensor-parallel storage balancing to increase cache capacity. On a cache hit, omitted state units are either zero-filled or approximately reconstructed through short-suffix replay.

Critical strengths
  1. Clear presentation. The paper is well organized and clearly explains the motivation, method, and system design.
  2. Well-defined practical problem. The paper addresses the memory overhead of recurrent-state checkpoints in hybrid-model prefix caching, a concrete problem with practical relevance to LLM serving.
Critical weaknesses
  1. Lack of competitive baselines. The experiments mainly compare DASC against Dense and Random. These comparisons demonstrate compression benefits and the value of decay-aware selection over random selection, but do not establish its advantages over alternative checkpoint-management, state-compression, or replay approaches. Consequently, DASC’s comparative quality–memory–latency benefits remain unclear.

  2. Limited characterization of the quality–efficiency trade-off and applicability boundaries. RULER evaluation is restricted to contexts of at most 16k tokens, with some simple retrieval tasks already saturated, limiting evidence about quality under longer and more demanding contexts. Furthermore, the trade-off plot combines quality and latency measured on different workloads, rather than directly demonstrating their joint trade-off on the same workload. Quality under repeated compression and cache reuse is also insufficiently characterized. These gaps make it difficult to determine the conditions under which the reported efficiency gains remain compatible with acceptable task quality.

Questions for authors
  1. Could you compare DASC with at least one recent representative method addressing the same problem?
  2. Could you extend the comparison to RULER at 128k contexts or beyond, and provide a quality–efficiency trade-off plot for DASC and the baseline to make their relative strengths clearer?
6 comments

Thank you for the questions and careful reading and helpful feedback.We address the baseline and quality–efficiency concerns with the paper's evidence and completed additional experiments.

Q1. Comparison with a recent alternative

To our knowledge, DASC is the first study of recurrent-prefix-checkpoint compression for hybrid LLM serving. At submission, we had not identified a published, directly comparable checkpoint-compression method; Dense and count-matched Random therefore tested compression benefits and the value of horizon selection. During rebuttal, we examined recent open-source work and adapted SketchSSM to this checkpoint problem.

Original SketchSSM retains full update states and approximates decoding reads. Our adaptation instead stores FP32 low-rank factors at checkpoint boundaries and reconstructs the state on reuse. The g16/g32 settings allocate ranks adaptively across heads. Compression includes recurrent and convolution state, excluding unchanged FA-KV. SketchSSM

MethodCheckpoint compression ↑AIME 2026 ↑GPQA Diamond ↑LiveCodeBench ↑LongBench ↑
Dense1.00×60.6561.8558.3144.96
DASC W162.22×60.3261.3158.7744.34
Sketch g163.62×57.3161.1457.9444.18
Sketch g321.90×59.7861.5258.1244.23

DASC compresses more than g32 and scores higher on three of four tasks; GPQA is 0.21 points lower. Against g16, it sacrifices compression for better quality, including +3.01 points on AIME. These are point estimates at different compression ratios.

Serving: TP8, client/server concurrency 256/32, 1,024 requests, and the same per-GPU state budget of 128 Dense slots; mean input length is approximately 3,566 tokens.

MethodMean TTFT ↓ (ms)P95 TTFT ↓ (ms)TPOT ↓ (ms)Output throughput ↑ (tok/s)
Dense120.88310.5510.71851.11
DASC90.11164.429.77852.89
Sketch g1690.93190.409.88851.82
Sketch g3298.58194.6010.36851.40

DASC reduces mean/P95 TTFT by 25.5%/47.1% versus Dense, with nearly unchanged output throughput. Quality and serving use separate protocols.

Q2. 128K evaluation and the quality–efficiency trade-off

We chose N=30 for RULER to balance available compute across the overall three-model, multi-benchmark evaluation. The original RULER sweep covers Qwen3-Next and Kimi-Linear, with 13 subtasks and multiple lengths/cutoffs (§5.1, Appendix C.2). To strengthen the uncertainty analysis, we completed N=100 per subtask–length setting and added 128K on Kimi-Linear TP8 covers 13 subtasks at 4K/8K/16K/128K; each independent instance has one unscored warmup and three greedy replays. Full running state is retained; only prefix checkpoints are compressed.

MethodReplay score [95% CI]Difference vs Dense (pp) [95% paired CI]
Dense87.63 [86.51, 88.75]—
DASC-NR W1687.55 [86.43, 88.66]−0.08 [−0.44, +0.26]
DASC-NR W6487.33 [86.24, 88.42]−0.30 [−0.85, +0.25]
DASC-NR W12887.12 [86.06, 88.17]−0.52 [−1.16, +0.11]
DASC-NR W51285.63 [84.54, 86.69]−2.01 [−2.79, −1.25]
DASC-NR W102484.92 [83.75, 86.06]−2.71 [−3.68, −1.79]

The 95% CIs use 50,000 paired bootstrap replicates, resampling instances within each subtask–length setting and keeping all three replays together. At 128K, W64 differs from Dense by −0.30 points [−0.85, +0.25]; the interval does not establish equivalence. We have not evaluated Sketch at 128K or contexts beyond 128K.

Quality–efficiency comparison. Please see Fig. 4(l), §5.4: it already shows NR/WR operating points by pairing Kimi-Linear MMLU-Pro accuracy with WildChat TTFT. The evaluation spans Kimi-Linear-48B, Qwen3-Next-80B, and Kimi-K3, with challenging mathematical reasoning, GPQA, MMLU-Pro, retrieval, and conversational-memory benchmarks (§5.1). Given this scope and available compute, we did not extend the trade-off evaluation to further deployment scenarios. Fig. 4(l) is explicitly a cross-workload comparison, not a jointly measured same-request curve.

Successive recovery. Kimi-Linear DASC-NR W16 stays within 0.28 points of same-depth Dense on 16K RULER through 16 restores (13 subtasks × 100 independent instances):

Actual checkpoint restoresDenseDASC-NR W16DASC − Dense (pp)
0: cold full prefill93.02——
493.9494.21+0.27
894.1094.05−0.06
1693.7493.81+0.07

Each chain appends segments of the same final input, repeatedly saves and restores checkpoints, then scores only the final answer. Intermediate one-token outputs are excluded from history. Same-depth Dense controls for segmentation; direct-predecessor checkpoint hits are 100% on all eight TP ranks. No growing additional macro-average loss is observed within this range.

Long-context multi-turn quality. SCBench KV/VT/English-QA use 100/90/23 conversations, 500/450/118 answers, and mean initial lengths of 122,850/124,554/96,826 tokens. Each method's actual answers enter subsequent history:

SubtaskDense overallDASC overallΔ (pp)Dense later turnsDASC later turnsΔ (pp)
KV100.00100.000.00100.00100.000.00
VT72.7672.04−0.7182.6181.72−0.89
QA33.9033.05−0.8534.7433.68−1.05

Thank you for your response. Regarding the first concern, I do not think my concern has been fully addressed. SketchSSM was not originally designed to optimize the same problem as DASC. I believe methods discussed in the Related Work, such as Q-Mamba, are more closely related and would provide more meaningful comparisons. Regarding the second concern, the authors still have not provided the quality-efficiency trade-off plot that I was most interested in. I would like to see a direct comparison between DASC and other relevant methods under the same evaluation setting to better demonstrate the advantages of DASC. Therefore, I will maintain my original rating.

Comments on the paper

1 comment

Summary of Rebuttal Findings

Dear Area Chair,

Thank you for considering our submission. We briefly summarize DASC’s core idea and main results below.

  • Core idea. DASC uses learned decay parameters to identify, before serving, which recurrent heads or channels retain information over long spans. It stores those units in compressed prefix checkpoints while leaving the full-attention KV cache unchanged, so more checkpoints fit in the same memory budget. On reuse, omitted units can be zero-filled or refreshed by replaying a short suffix. To our knowledge, DASC is the first study of pruning-based compression of recurrent state checkpoints for prefix reuse in hybrid LLM serving.
  • Main result. At the same state-memory budget, configurations that keep accuracy close to the dense baseline reduce mean time to first token by 42.0% on Kimi-Linear, 25.6% on Qwen3-Next, and 16.1% on Kimi-K3. Mean end-to-end latency falls by 10.2%, 4.9%, and 6.0%, respectively.

We thank the reviewers for their feedback. Our rebuttal addressed four questions raised across the reviews.

  • Does decay-based selection preserve task-relevant information? Reviewers 4Smq and sXGv questioned whether static retention horizons identify channels that matter for prediction. We validated the offline calibration on held-out prefixes and tested the effect of reversing the W64 selection mask. The standard mask outperformed its complement on all three reasoning benchmarks.
  • Does quality hold up under repeated reuse and at longer contexts? Reviewers fyx5 and sXGv raised this concern, and 4Smq asked for stronger uncertainty estimates. On 16K RULER, DASC-NR W16 stayed within 0.3 points of the uncompressed baseline at 4, 8, and 16 successive restores. At 128K, W64 scored 87.33 versus 87.63 for the baseline, a difference of −0.30 points (paired 95% CI [−0.85, +0.25]). We also tested multi-turn reuse on SCBench. In the follow-up, sXGv noted that the repeated-reuse and long-context experiments helped address several of their concerns.
  • When do memory savings translate into lower latency? Reviewers 4Smq and sXGv asked how cache misses and recovery costs affect serving performance. We reported cache memory including full-attention KV across context lengths and broke down hits, evictions, and latency at a fixed memory budget. On Kimi-K3, W64 without suffix refresh raised the cache hit rate from 54.9% to 68.9% and reduced mean time to first token by 16.1% relative to the baseline. Enabling suffix refresh at the same cutoff yielded a similar hit rate, but replay overhead pushed latency above the baseline.
  • How does DASC compare with other storage methods? For fyx5, we added a SketchSSM-inspired checkpoint baseline and reported its quality, compression, and serving latency alongside DASC. For sXGv, we compared INT8 alone with DASC+INT8 on quality and cache capacity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.