- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents AbridgeKV, a learned KV-cache compression method for long chain-of-thought reasoning. It compresses completed blocks of generated tokens into the KV states of learned pause tokens and removes the original cache. Three key design choices are in-block positions, layer-wise KV budget allocation, and large compression blocks. On Qwen3-4B-Base, AbridgeKV achieves 43.0 accuracy compared with 43.8 for full KV, while reducing average KV memory by 3.54× and improving problem completion speed by 3.2× under saturated H100 serving.
Critical strengths
-
The learned compression approach substantially reduces KV-cache memory while maintaining reasoning accuracy close to full KV. The generation length also remains similar to the full-KV baseline.
-
The three design choices are clearly motivated and supported by ablations. In-block positions improve accuracy by 6.2 points, and layer-wise allocation provides additional improvements under the same memory budget.
-
The paper demonstrates practical serving benefits in vLLM, including 3.54× lower KV memory and 3.2× faster problem completion. Comparisons with R-KV and LightThinker further support the results.
Critical weaknesses
-
The evaluation only uses Qwen3-4B-Base and mathematical reasoning benchmarks. It is unclear whether the method generalizes to other model sizes, architectures, or reasoning tasks.
-
The method uses a fixed compression budget for every reasoning block within each layer. Different blocks may contain different amounts of important information, so uniform block-level allocation may not be optimal.
Questions for authors
-
Have the authors tested AbridgeKV on larger models or other long-generation tasks, such as code generation and agentic reasoning?
-
How should the compression block size be chosen in practice? Could adaptive block sizes further improve the accuracy–memory trade-off?
Author Response
Thank you for the encouraging review. You asked whether the method generalizes to another domain and to a larger model. We trained code-generation models and Qwen3-8B models with the same recipe. The method generalizes: the per-layer model stays within one point of full KV in both settings.
W1 and Q1: other tasks and larger models
The method carries over to code generation. We trained full-KV, uniform and per-layer models on long chain-of-thought competitive-programming solutions with the same recipe as for math, and evaluated the three models on the 336 hard and medium LiveCodeBench problems released between August 2024 and April 2025 (16 samples per problem, graded on the hidden tests). Full KV solves 25.4% and the per-layer model solves 25.1%. The gap is 0.3 points
−1.5,+0.9at the same output length. Uniform allocation is 1.9 points below full KV, and per-layer allocation recovers most of the loss (+1.6 over uniform
+0.3,+2.8), so per-layer allocation matters in code as in math.
−2.4,+0.5The per-layer schedule measured on math transfers to code. We re-measured the per-layer schedule on the code model, at a cost of about one GPU-hour. The code schedule agrees with the math schedule on 32 of 36 layers. Which layers need more rows appears to be a property of the model more than of the domain.
The method carries over to a model twice the size. We trained the same three models from Qwen3-8B-Base with the same recipe. On the 446-problem suite full KV scores 48.9 and the per-layer model scores 48.0. The gap is 0.9 points
at the same output length, and matches the gap at 4B under the same protocol. There is no sign that compression costs more as the model grows: with uniform allocation the gap is 1.2 points at 8B against 2.3 at 4B. The 8B per-layer model used the 4B schedule unchanged. With one training seed per model at 8B, the 0.3-point lead of per-layer over uniform allocation is not resolved.
Agentic reasoning. We have not yet evaluated agentic tasks, and the paper will list agentic evaluation as a limitation. The mechanism applies to agentic tasks directly, since an agent's trajectory is long generated context, and an agentic evaluation is our next planned experiment.
W2: a fixed budget per block
A fixed budget is what makes the method cheap to train and serve. With a fixed budget the attention mask depends only on token positions, so training runs on stock kernels at 76% of ordinary fine-tuning throughput. Each request's memory is also known in advance, so a server can pack requests and realize the 3.5x saving. A budget that follows the content of each block is a natural extension, as you note, and would need a learned decision about how many rows to write and a mask built per sample.
Per-head allocation does not improve accuracy and costs serving speed or memory. Per-head allocation within a layer did not improve accuracy at equal rows, and with current cache layouts per-head allocation is 3.5x slower or needs 1.5x the memory. Per-layer allocation is the granularity that leaves both the training and the serving kernels unchanged. The paper now lists per-block and per-head allocation as future work.
Q2: choosing the block size
Larger blocks help because more recent text stays exact, so the exact window is the setting to choose. The new control is a C=512 model that also keeps the previous 511 tokens exact. The control recovers the whole gain of C=1024 (+1.6 points over C=512, in two training seeds) while compressing as often as C=512. The window is paid for in memory, since each request holds twice as much exact text in every layer. At our output lengths the memory cost is small (3.36x less KV than full KV, against 3.49x without the window), and the cost grows as generations get shorter.
Per-layer allocation at C=512 is the choice when memory is the constraint. The per-layer model is 0.5 points below the doubled-window control, within the noise of the evaluation, and adds no memory. A larger window is the simpler choice when memory is plentiful. The window and per-layer allocation are alternatives: at C=1024, per-layer allocation adds nothing further.
Adaptive block sizes would give up the position-only attention mask that keeps training and serving cheap. Block boundaries chosen from content need a learned decision about where each block ends and a mask built per sample, as a content-dependent budget does (W2). LightThinker already uses small blocks at content-chosen boundaries, and in our setting LightThinker is about 10 points below full KV even at our row budget. The paper lists adaptive block sizes as future work.
We hope the code-generation and 8B results answer your questions, and we are glad to provide anything further.