acceptodds
Under review as a conference paper at ICLR 2027

LAB-Sparse: Local Accumulative Boost for Adaptive Semantic-Block Retention in KV Cache

Abstract

The expanding context windows of large language models (LLMs) have brought substantial gains in reasoning and comprehension, yet this expansion comes at the cost of a linearly growing Key-Value (KV) cache that strains GPU memory and increases decoding latency.Existing KV sparsification methods, whether heuristic token-level eviction or fixed-size block retention, share a common limitation: they treat the KV cache as a collection of isolated units. This overlooks a fundamental property of natural language—semantic information is typically organized as continuous spans rather than isolated tokens. Fixed-size blocks are particularly problematic: a coherent semantic unit may be split across multiple blocks, and averaging scores within a block can dilute or filter out critical tokens. Consequently, these methods often suffer from fragmented critical information and degraded continuity in multi-hop reasoning and long-context retrieval. To address this, we propose LAB-Sparse, a dynamic continuous block retention method powered by Local Accumulative Boost (LAB). Our core insight is to elevate the basic unit of sparsification from individual tokens to dynamic-length contiguous blocks, where block lengths are determined by contiguous runs above a quantile-based threshold, which implicitly reflect local importance density. The LAB mechanism applies a neighborhood cumulative gain to smooth token-level importance scores, thereby boosting tokens surrounded by high-importance neighbors and suppressing isolated noise. On this enhanced distribution, a quantile-based interval delineation strategy identifies high-value contiguous regions, followed by a greedy block-wise retention policy that prioritizes complete contiguous blocks under a fixed KV budget. Experiments on Llama-3.1-8B, Qwen2.5-7B, and Mistral-7B across LongBench and RULER show that LAB-Sparse achieves competitive or superior average accuracy compared with strong baselines such as H2O, SnapKV, StreamingLLM, ChunkKV, and PyramidKV, while remaining close to full attention in the 4K–16K range. Under a fixed KV budget of 512, it matches the decoding throughput and memory savings of block-based baselines at 32K ( and 12.6%), while improving average accuracy over token-level methods.Anonymous code will be submitted as supplementary material.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, KV Cache Sparsification, Efficient Inference, Dynamic Block Retention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NHh6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed LAB, which uses Local Accumulative Boost to define the blocks for KV cache, make these blocks contain semantic information.

Critical strengths
  • I think the idea of using LAB is kind of like a 1-D convolution, is interesting.
Critical weaknesses
  • The paper lacks a sufficiently thorough ablation of the key LAB hyperparameters, especially the neighborhood radius rr and scaling factor α\alpha. Although Appendix A includes a small sensitivity study, it is limited to PG19 at 16K on Mistral-7B with only 8 samples. A more systematic analysis across tasks and models would be useful to show that the reported gains are robust and not sensitive to these manually chosen values.
  • I am unclear about the design of the interval delineation step. The threshold τ\tau is computed from the enhanced score distribution s~i\tilde{s}_i, but candidate tokens are selected using the raw normalized scores si≥τs_i \geq \tau. Therefore, the local smoothing does not directly determine whether a token belongs to a block. This seems somewhat inconsistent with the motivation that LAB helps form semantically coherent blocks. I would like the authors to explain why this cross-distribution comparison is preferable to directly thresholding the enhanced scores s~i≥τ\tilde{s}_i \geq \tau.
Questions for authors
  • What is the latency (TTFT) overhead of this method, especially for very long context like 128K? It's not reported in the paper.
  • I am interested in the use of the term "semantic block." The selected blocks are defined as contiguous runs of high-attention-score tokens, rather than by sentence, paragraph, entity, or other semantic boundaries. The diagnostic analysis only shows that high-importance tokens tend to be spatially clustered in a few examples, which does not necessarily imply that each selected block corresponds to a coherent semantic unit. Can the author quantitatively evaluate whether the retained blocks actually align with meaningful semantic spans?
Reviewer_ujioOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LAB-Sparse, a training-free KV cache sparsification method that replaces token-level eviction or fixed-size block retention with dynamic-length contiguous block selection. The method consists of Local Accumulative Boost, quantile-based interval delineation and a greedy block-wise retention policy under a fixed KV budget. Experiments on LongBench and RULER show improvements over traditional token-level baselines.

Critical strengths

Shifting the pruning unit from isolated tokens to dynamically sized contiguous blocks is a well-motivated conceptual advance.The method requires no extra fine-tuning and adds negligible prefill overhead while yielding consistent accuracy improvements over baseline approaches.

Critical weaknesses
  1. One main claim of this paper is KV sparsification basic unit moves from single tokens or fixed chunks to dynamic length continuous semantic blocks. But SABlock(Chen et al.,arXiv:2510.22556) already uses almost same idea and workflow. It does semantic segmentation, segment guided token scoring, sets variable block size automatically, keeps blocks based on budget limit. But authors fully miss citation, discussion and experiment comparison with SABlock.

  2. The method implicitly assumes spatial clustering in attention maps matches semantic continuity. But paper does not add checks to confirm these smooth continuous blocks really match coherent semantic units. Neighbour smoothing works on raw signal level. It can not reliably tell simple syntax continuity apart from real semantic meaning.

  3. Tests run only under one compression budget. Readers hard to know how performance changes for different compression ratios.

Questions for authors

Have you evaluated the retention rate of task-critical isolated tokens under LAB smoothing, and do you have empirical evidence showing that smoothing does not degrade performance on tasks requiring the retrieval of sparse, discontinuous key facts?

Reviewer_s6dROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

LAB-Sparse studies KV cache compression by retaining variable-length contiguous token blocks instead of selecting isolated tokens or fixed-size chunks. Its Local Accumulative Boost mechanism adds nearby attention scores to each token’s importance score, with distance-dependent weights. The method computes a quantile threshold from the enhanced scores, applies it to the original scores to form candidate blocks, and selects blocks or subsegments within a budget that also reserves recent tokens. Compression decisions are made during prefill, shared between neighboring layer pairs, and reused during decoding. The experiments evaluate three model families on LongBench and RULER, with component ablations, parameter sensitivity checks, and inference measurements. The reported results show competitive LongBench averages and efficiency comparable to block-based baselines, while retrieval performance varies substantially across tasks and context lengths.

Critical strengths
  1. The method is simple and training-free. Keeping variable-length blocks is a reasonable approach to reducing the fragmentation caused by fixed block boundaries.

  2. The experiments cover three model families and several task types. The Qasper ablation provides a useful example where local score enhancement helps.

  3. The three-stage presentation, pseudocode, and detailed efficiency table make the procedure and its measured trade-offs easy to inspect.

Critical weaknesses
  1. The Qwen results in Table 1 contain a systematic anomaly. DUR and QAS are identical in five rows, while ChunkKV and StreamingLLM have crossed values (29.01/37.60 and 37.60/29.01). This suggests a reporting error, but the table alone cannot identify the correct values. The task-level claims and the 34.19 versus 34.02 average comparison need verification against the original outputs.

  2. The selection rule does not establish the claimed benefit of LAB. At fixed raw scores and quantile, thresholding raw scores with the enhanced-score quantile can only keep or shrink the candidate set.

  3. A reliable advantage over ChunkKV has not been established. The reported LongBench differences are only +0.15, +0.17, and −0.20 points, and the Qwen numbers require the check above.

Questions for authors
  1. The scoring definition is ambiguous. Eq. (2) uses the last query step, while §3.2.4 refers to the last query block. Please state how queries are aggregated, how head averaging maps to GQA groups, and whether all heads in a layer share the retained positions.

  2. How does LAB compare with ChunkKV and SnapKV at budgets 128, 256, 512, 1024, and 2048?

  3. How does LAB differ from SentenceKV and DynSplit-KV in its definition of semantic units, boundary selection, and retained information?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.