acceptodds
Under review as a conference paper at ICLR 2027

LARK: Low-Rank Archived Retention of KV Cache via Linear Expected-Attention Scoring

Abstract

Long chain-of-thought (CoT) reasoning generates tens of thousands of tokens, making KV cache the primary memory and bandwidth bottleneck. Token eviction irreversibly discards history; its accuracy hinges on ranking keys by attention from queries that do not yet exist, which content predicts only in their frame, reached by an inverse rotation per candidate. We propose LARK, a KV cache compression system built to archive rather than discard. LARK comprises: (1)LEAP, a linear scoring operator strictly equivalent to pre-RoPE expected-query phase scoring yet executing as a single projection on cached post-RoPE keys, with cost independent of offset and query-sample counts; (2)LACE, a low-rank archival pipeline whose evicted tokens are compressed into factors that leave the model cache yet keep participating in attention, re-scored at segment level by the same form; (3)a systems co-design enabling CUDA-graph-compatible variable-length continuous batching. Across four models and four benchmarks at a k generation limit, LARK retains accuracy far above eviction-based baselines as the budget tightens. At it reaches near-lossless accuracy with 5.0% of the dense device KV memory, with 2.0-3.3x iso-batch throughput.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache compression, low-rank, long-CoT

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_dtWqOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In long chain-of-thought, eviction fails because deleted tokens get needed again and the model loops until it runs out of budget. The evicted history is low-rank, so we can archive it instead of deleting it. The authors propose LEAP to score keys by expected future attention on cached post-RoPE keys and stores each evicted segment as rank-64/32 fp8 factors. Across four models and four benchmarks LARK reaches near-lossless accuracy with 5.0% of the dense device KV memory, with 2.0-3.3x iso-batch throughput.

Critical strengths
  1. Clear idea with large accuracy gains at tight budgets across 4 models × 4 benchmarks.
  2. The failure analysis is convincing that eviction baselines fail by never finishing, not by reasoning badly.
  3. The authors perform ablations on each component, rank, and k.
Critical weaknesses

NA

Questions for authors

Authors should report Confidence intervals for AIME and GPQA

Reviewer_icavOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Unlike conventional KV eviction methods that permanently discard evicted entries, this paper proposes compressing them into low-rank representations, storing them on the CPU, and bringing relevant archives back to the GPU for reuse in subsequent attention computations. It achieves higher accuracy than eviction-based methods, particularly under tight cache budgets.

Critical strengths

S1. Practical value: Unlike conventional eviction methods, this work compresses and archives evicted KV entries for reuse during subsequent reasoning. It achieves higher accuracy than eviction-based baselines, particularly under tight cache budgets, making it useful for long reasoning in memory-constrained settings.

S2. Implementation and evaluation: The authors implement the method in a serving engine and evaluate memory usage and throughput alongside accuracy. It is encouraging that the method provides practical speed gains despite the additional compression cost.

Critical weaknesses

W1. Limited novelty: As I understand it, keeping important KV entries on the GPU and storing the rest on the CPU for later reuse is similar to existing KV cache offloading methods [1], and low-rank KV compression has also been studied [2]. LARK identifies online archival and direct attention over compressed factors as key differences from prior work. However, without direct comparisons against relevant low-rank and offloading methods, it remains unclear how much these differences improve the accuracy–memory–latency trade-off.

[1] Sun, Hanshi, et al. "Shadowkv: Kv cache in shadows for high-throughput long-context llm inference." ICML, 2025.

[2] Chang, Chi-Chih, et al. "xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction." ICML, 2026.

W2. Memory efficiency under matched precision: As I understand it, the main experiments compare LARK and the baselines under the same token budget, but LARK stores its active KV cache in FP8 while the baselines use BF16. Using FP8 would allow the baselines to retain more KV entries within the same GPU memory budget. It would therefore be useful to verify whether LARK’s accuracy gains hold under matched precision and total GPU KV memory budgets.

Questions for authors

See weakness

3 comments
AuthorsOpenReview ↗

Follow-up on Matched-Precision Results: Two Days Remaining

Dear Reviewer icav,

Thank you for engaging with our rebuttal and for your follow-up question on matched precision.

With only two days remaining in the discussion period, we would appreciate your feedback on our November 16 clarification. The FP8 rows compare LARK, BeaconKV, and SnapKV with all three using FP8 KV caches at the same device-memory budget; we corrected the table header to make this explicit. LARK achieves higher accuracy in all four reported FP8 benchmark–capacity comparisons.

Together with the added ShadowKV and xKV evaluations, these results support the practical value of retaining compressed history during long decoding beyond the memory savings from quantization alone.

We would be grateful to know whether this clarification addresses your remaining concern. We welcome further discussion so that the additional evidence and LARK’s contribution can be fully reflected in the final evaluation.

Wish you a pleasant day ahead.

Authors of LARK

Thanks for the clarification and the additional experiments. They address my main concern, so I’m raising my rating to 3. I still have some reservations about the novelty over prior work, so I consider this a borderline accept (marginally above the threshold in the old ICLR scale).

Dear Reviewer icav,

Thank you for your updated evaluation and for raising the score. We appreciate your recognition of our work. Your constructive feedback has helped us strengthen LARK’s the evaluation of its effectiveness and compatibility with existing strategies.


We briefly state LARK's novelty in two points:

  • LARK is the first efficient low-rank KV compression framework for long decode: we introduce an innovative low-rank archival design, and an end-to-end implementation in sglang from kernels to the framework that works with CUDA graphs, prefix caching, and continuous batching, delivering the first low-rank algorithm with near-lossless throughput.
  • LARK firstly proposes a linear scorer strictly equivalent to scoring with the expected pre-RoPE query, which thereby natively supports per-frequency rotation-rate rescaling schemes such as YaRN, it is also the prerequisite that makes efficient low-rank scoring possible.

We thank the reviewer again for these questions: we have added related ablations and analysis to manuscript. We believe they substantially strengthen LARK overall quality.

Reviewer_rwtfOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper tackles the problem of kv cache growth during decoding. The authors propose to “archive” unimportant tokens instead of evicting them, i.e., completely discarding them. Each evicted segment becomes a low-rank factorization that still takes part in exact attention. A linear expected-query score ranks both cached tokens and archived segments. Experiments on three datasets and four models show LARK outperformed four baselines regarding the budget-accuracy tradeoff.

Critical strengths

S1. The performance of LARK looks strong compared to four popular baselines.

Critical weaknesses

W1. The presentation of this paper requires a significant amount of polishing of the logics. The motivations are not connected to any observed issues or how the approach or its subcomponents are designed. Many terms that are not conventional in the literature are also used directly without establishing its meaning, like “archive”, “tiering”. Some variables are also used without explaining, e.g., B represents tokens or pages (despite it can be inferred from Figure 3.a that it is tokens)? There is also no overview of the proposed approach. This is a clear issue of over-relying on LLM writing or polishing the text.

W2. Similar ideas seem to be exploited earlier [1]. Please explain the differences, and include it in the experiments.

W3. When tiering is introduced into the problem design, an alternative approach is also sparse kv cache retrieval. Please discuss over the competitive edges of tiered “archival” vs. sparse kv cache retrieval approaches [1][2], and compare them experimentally.

[1] Devoto, A., Jeblick, M. and Jégou, S., 2025. Expected attention: Kv cache compression by estimating attention from future queries distribution. arXiv preprint arXiv:2510.00636.

[2] Mao, Y., Wang, Q., Ester, M. and Li, K., 2026. IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs. arXiv preprint arXiv:2604.10539.

[3] Chen, R., Wang, Z., Cao, B., Wu, T., Zheng, S., Li, X., Wei, X., Yan, S., Li, M. and Liang, Y., 2024. Arkvale: Efficient generative llm inference with recallable key-value eviction. Advances in Neural Information Processing Systems, 37, pp.113134-113155.

Questions for authors

Q1. How does RoPE frequencies impact the performance of LARK? For examples, Qwen3 needs YaRN to rescales the RoPE frequencies when the length of the sequences go beyond 32k.

Q2. Is the code and experimental scripts going to be open-sourced?

3 comments
AuthorsOpenReview ↗

Follow-up: Two Days Remaining in Discussion

Dear Reviewer rwtf,

Thank you again for your detailed review and recognition of LARK’s strong empirical performance.

With only two days remaining in the discussion period, we are following up on our November 9 rebuttal and November 14 reminder. We have revised the algorithm overview, terminology, and budget notation; provided analysis and evaluations concerning Expected Attention, ShadowKV, and IceCache; and added YaRN compatibility experiments.

LARK’s central contribution is making evicted history remain useful during long reasoning through compact archival and a shared linear scoring mechanism. The supplementary comparisons help clarify how this design differs from existing scoring and retrieval approaches.

We have devoted considerable effort to addressing your concerns and would sincerely value the opportunity to discuss the additional evidence before the deadline. Please let us know whether our responses resolve the issues you raised or which concerns remain, so that we can address them while discussion is still open.

Thank you for your time and consideration.

Wish you a pleasant day ahead.

Authors of LARK

Thank you for the rebuttal. I appreciate the clarification and the additional experiments, However, it is still very hard for me to understand how the claimed insights (only insight 2 is relative concrete; the other two are also very high-level) lead to which parts of the designs, and why.

Overall, I increase my rating to 2 and maintain other scores.

Thank you for your updated evaluation and for raising the score!

For the question "hard to understand":

  • we believe this likely stems from the fact that LARK builds on an in-depth study of the pre-RoPE scoring mechanism combined with a multi-level low-rank design and pre-RoPE scoring is a relatively new scoring paradigm proposed in this year [1]. Fully understanding the paper therefore requires some familiarity with both directions (which the initial submission, due to space constraints, did not introduce in full).
  • We fully understand the feel you have; we have accordingly further refined the manuscript, elaborating the insight behind the scoring mechanism, to making it easier to read.

Below, we briefly state LARK's insight state in 3 points:

Insight 1 gives the input to the scorer. Content similarity to the oracle while a random-query control sits at zero, so the unpredictable side is the query; distance alone recovers 0.730.73 only once realigned to future positions (Figure 2b).

Insight 2 gives how that input is computed. The pre-RoPE frame is the only one with a stable query centre (Figure 2c), but scoring in it costs a lot; the composition rules move the rotation to the statistics side, so the identical quantity becomes cheap.

Insight 3 gives how to compression. Better ranking cannot buy capacity, so C2 archives evicted segments as factors that still join attention.

Taken together, this is where our novelty sits. LARK is the first efficient low-rank KV compression framework built for long decode: an archival design that keeps evicted history re-scorable rather than discarding it, a linear scorer strictly equivalent to scoring with the expected pre-RoPE query, hence native to YaRN, and makes low-rank scoring affordable. Moreover, we make an end-to-end implementation in sglang, from kernels up to the framework, compatible with CUDA graphs, prefix caching and continuous batching. Together they deliver the first low-rank method with near-lossless throughput.


Thank you again for pressing on LARK, and we are grateful that you took a second pass at it! We wish our response could solve your question and raise your evaluation of LARK.

[1] TriAttention: Efficient Long Reasoning with Trigonometric KV Compression

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.