- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Decay-Aware State Compression (DASC) for recurrent-state checkpoints in hybrid linear-attention prefix caches. DASC estimates static retention horizons from learned decay parameters and a fixed representative gate input, retaining long-horizon GDN heads or KDA channels. It combines this selection with ragged storage, optional tensor-parallel placement balancing, and a two-tier cache implementation in SGLang. On reuse, omitted units are either zero-filled or approximately reconstructed through bounded suffix replay; full-attention KV remains unchanged. The evaluation covers Kimi-Linear, Qwen3-Next, and Kimi-K3, using retrieval, reasoning, conversational-memory, and WildChat serving workloads. At selected conservative configurations, the paper reports 1.32–2.22× state-checkpoint compression, 16.1–42.0% lower mean TTFT, and 4.9–10.2% lower mean end-to-end latency, alongside quality estimates close to the dense baseline on separately evaluated tasks.
Critical strengths
1. Practical systems contribution. DASC addresses checkpoint memory pressure with an integrated SGLang implementation. Matched-memory experiments include temporary buffers and report latency across all requests, providing meaningful evidence of serving benefits.
2. Validation across architectures and tasks. The evaluation covers GDN and KDA across three models, extends beyond simple retrieval, and includes count-matched Random selection. Disabling dense spill in quality experiments prevents compression from being bypassed.
Critical weaknesses
1. Retention duration does not directly establish predictive importance. Short-horizon units may contain recent information essential to the answer. Correlations with observed decay and state magnitude do not establish the error caused by omitting those units. Although the authors acknowledge this distinction, Random comparisons provide only partial validation. Direct omission-impact analysis and stronger selection baselines are needed to justify horizon as the central selection criterion.
2. The practical quality–latency advantage remains incompletely characterized. Quality and latency are measured on separate workloads and cache protocols, leaving their joint behavior under successive approximate reuse unclear. Serving evaluation uses four-turn traces capped at 16k tokens, limiting evidence across reuse and memory-pressure conditions despite the additional Kimi-K3 queueing tests. Furthermore, the paper evaluates DASC+INT8 but lacks an INT8-only quality and serving comparison, leaving DASC’s incremental benefit over quantization unresolved.
Questions for authors
-
Table 1(b) validates the relationship between static and observed decay, but does not directly measure the consequences of omission. Does removing short-horizon units cause less prediction loss or task degradation than removing matched groups of long-horizon units? A comparison against a fixed, calibration-based state-magnitude selector at matched per-layer storage budgets would also help establish the specific value of the horizon criterion.
-
Does the warmup–replay quality evaluation include checkpoints produced after an earlier approximate cache load and subsequently reused again? Repeated reuse may propagate approximation errors, while quality and serving are currently evaluated under different workloads and cache policies. Evidence measuring both quality and latency along such reuse chains would clarify whether the reported trade-off holds in multi-turn serving.
-
How strongly do the reported gains depend on prefix reuse and state-cache pressure? A breakdown of compressed hits, dense hits, misses, avoided recomputation, and checkpoint-management overhead in the existing runs would help explain when increased cache capacity outweighs packing, communication, or refresh costs. This would make the applicability of the results clearer without requiring evaluation of every deployment setting.
-
Appendix E.1 reports 3.88× checkpoint compression from INT8 alone, while Table 3 evaluates DASC+INT8 without an INT8-only quality arm. How do these alternatives compare in quality, TTFT, and end-to-end latency under the same state-memory budget and cache-management policy? This comparison would establish DASC’s incremental practical benefit beyond quantization, rather than only its benefit over dense storage.
4 comments
Q3. Prefix reuse, cache pressure, and serving benefit
Our primary serving endpoints are TTFT and E2E latency, including the applicable cache-management, communication, replay, and miss costs; E2E additionally includes decoding (§5.3; Appendix C.3). Intermediate hit/eviction metrics help explain these endpoints. We therefore provide the original Kimi-K3 breakdown below.
All arms use the same allocated state-memory budget (native states, compressed checkpoints, and temporary buffers), unchanged KV allocation, and the paper's common trace and global-LRU policy. The trace has 256 four-turn conversations; headline latency uses all requests, not only hits.
| Arm | Hit: all turns (%) | Evictions | TTFT mean / P95 (ms) | Δ mean TTFT | E2E mean / P95 (s) | Δ mean E2E |
|---|---|---|---|---|---|---|
| dense_native | 54.92 | 1,732 | 257.80 / 551.86 | — | 11.330 / 26.409 | — |
| nr_off_batch_w64 | 68.92 | 1,634 | 216.39 / 459.29 | −16.06% | 10.647 / 24.350 | −6.03% |
| nr_tp_batch_w64 | 69.27 | 1,630 | 218.49 / 442.09 | −15.25% | 10.745 / 24.608 | −5.17% |
| wr_off_batch_w64 | 68.89 | 1,634 | 304.23 / 591.47 | +18.01% | 11.994 / 27.851 | +5.86% |
| wr_tp_batch_w64 | 69.24 | 1,630 | 306.35 / 593.39 | +18.83% | 12.036 / 27.885 | +6.23% |
| nr_off_batch_w128 | 70.99 | 1,583 | 215.19 / 452.90 | −16.53% | 10.607 / 24.116 | −6.39% |
| nr_tp_batch_w128 | 71.27 | 1,576 | 215.37 / 437.45 | −16.46% | 10.714 / 24.370 | −5.44% |
| wr_off_batch_w128 | 71.17 | 1,583 | 299.18 / 555.07 | +16.05% | 12.009 / 27.727 | +5.99% |
| wr_tp_batch_w128 | 71.27 | 1,575 | 295.85 / 563.25 | +14.76% | 11.953 / 27.492 | +5.50% |
| nr_off_batch_w512 | 73.46 | 1,401 | 204.15 / 422.77 | −20.81% | 10.451 / 23.439 | −7.76% |
| nr_tp_batch_w512 | 73.46 | 1,374 | 201.36 / 416.42 | −21.89% | 10.477 / 23.737 | −7.53% |
| wr_off_batch_w512 | 73.46 | 1,401 | 292.34 / 524.17 | +13.40% | 11.907 / 27.593 | +5.09% |
| wr_tp_batch_w512 | 73.46 | 1,374 | 299.70 / 576.04 | +16.25% | 11.913 / 27.487 | +5.14% |
Hit averages all four turns; latency changes are relative to Dense. NR-Off-W64 raises Hit by 14.00 points and reduces mean TTFT/E2E by 16.06%/6.03%. At W512, NR-Off/TP reach 73.46% Hit and lower TTFT by 20.81%/21.89%; WR has similar hits but higher total latency, consistent with additional recovery overhead. Thus extra capacity helps when it avoids recomputation, but higher hit rate alone does not guarantee lower latency.
Q4. INT8-only versus DASC+INT8
Quantization is complementary: it reduces bits per value, while DASC reduces retained units. Only cached temporal state is quantized; running state stays FP32, convolution state BF16, and weights/FA-KV are unchanged. The supplementary INT8-only aggregates use the same questions and generation/scoring settings as Table 3; identical warmup/replay accounting is not assumed.
| Benchmark | Dense | DASC-NR W16 | INT8-only | DASC-NR W16 + INT8 |
|---|---|---|---|---|
| AIME2026 | 60.65 | 60.32 | 60.65 | 62.15 |
| HMMT 2026 Feb | 37.54 | 38.61 | 37.83 | 37.15 |
| IMOAnswerBench | 27.83 | 26.75 | 28.33 | 27.75 |
| GPQA-Diamond | 61.85 | 61.31 | 61.95 | 61.28 |
| MMLU-Pro | 67.50 | 67.32 | 68.64 | 67.48 |
Fixed-budget capacity. With a 5.196 GiB/GPU state-plus-metadata ceiling, unchanged 15.099 GiB/GPU KV, and the same native-state reserve for compressed arms, the TP2 byte/slot accounting gives:
| Method | Native slots | Separate compressed slots |
|---|---|---|
| Dense | 256, shared active/cache | — |
| DASC-NR | 128 | 329 |
| INT8-only | 128 | 445 |
| DASC-NR+INT8 | 128 | 1,019 |
DASC+INT8 supports 1,019 versus 445 compressed slots (2.29×) at this budget. This is a capacity benefit, not a predicted latency speedup. We limit this comparison to benchmark quality and budget/slot accounting; no further performance runs are planned.
Thank you for the detailed response and the additional experiments. The new results on repeated checkpoint reuse and long-context tasks helped address several of my concerns. I also appreciate the clarifications in the revised paper. I will keep my rating at 3.
Thank you very much for your positive feedback and for taking the time to review our revised paper and rebuttal. Thank you again for your time and consideration.
Thank you for the questions. We add direct selection and reuse evidence, clarify the original serving results, and report INT8 quality and capacity.
Q1. Does horizon selection predict omission impact?
We repeated the offline calibration and conducted a W64 omission ablation on Kimi-Linear. We compare the normal DASC mask with its strict complement, which retains the channels DASC discards and discards those it retains. All arms use the same questions and the paper's sampling, generation, and scoring settings, excluding warmup (AIME: 30×31 scored replays; HMMT: 33×31; GPQA: 198×15).
The complement is 19.36/18.47/7.75 points lower on AIME2026/HMMT/GPQA-Diamond, using the displayed scores. This directly measures the task impact of changing which channels are omitted. The comparison is not budget-matched (44.14% versus 55.86% retained), so it does not isolate ranking at equal storage; a matched-budget state-magnitude selector remains untested.
Q2. Successive approximate reuse
The new tests reuse checkpoints produced after earlier approximate loads, extending the original warmup–replay protocol.
Successive recovery. Kimi-Linear DASC-NR W16 stays within 0.28 points of same-depth Dense on 16K RULER through 16 restores (13 subtasks × 100 independent instances):
Each chain appends segments of the same final input, repeatedly saves and restores checkpoints, then scores only the final answer. Intermediate one-token outputs are excluded from history. Same-depth Dense controls for segmentation; direct-predecessor checkpoint hits are 100% on all eight TP ranks. No growing additional macro-average loss is observed within this range.
Long-context multi-turn quality. SCBench KV/VT/English-QA use 100/90/23 conversations, 500/450/118 answers, and mean initial lengths of 122,850/124,554/96,826 tokens. Each method's actual answers enter subsequent history: