acceptodds
Under review as a conference paper at ICLR 2027

AbridgeKV: Compressing Long Reasoning into the KV Cache of Learned Pause Tokens

Abstract

Long chain-of-thought decoding is memory-bound: the KV cache of the generated tokens grows linearly with the test-time compute being spent. Evicting cache entries does not fix this, because a model never trained on a cache with entries removed generates longer, often up to the length limit. We instead train the model to compress its own output. writes each completed block of generated text into the KV entries of a few learned pause tokens and frees the block's cache, trained end to end with the next-token loss. Three novel choices make this work: the summary rows take positions inside the block they replace, the row budget is spread across layers by each layer's measured value, and the blocks are large (512 tokens). On 446 competition problems with 32K-token generations, keeping a quarter of the generated cache matches an identically trained full-KV model in accuracy and output length; the strongest published evictor at a matched budget writes far longer, and the closest trained compressor, retrained on the same data, loses 10 points. Served in vLLM, the cache is smaller per sequence and a saturated H100 finishes a problem faster than with full KV.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Key-Value Compression, Large Language Model, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ALjjOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AbridgeKV, a learned KV-cache compression method for long chain-of-thought reasoning. It compresses completed blocks of generated tokens into the KV states of learned pause tokens and removes the original cache. Three key design choices are in-block positions, layer-wise KV budget allocation, and large compression blocks. On Qwen3-4B-Base, AbridgeKV achieves 43.0 accuracy compared with 43.8 for full KV, while reducing average KV memory by 3.54× and improving problem completion speed by 3.2× under saturated H100 serving.

Critical strengths
  1. The learned compression approach substantially reduces KV-cache memory while maintaining reasoning accuracy close to full KV. The generation length also remains similar to the full-KV baseline.

  2. The three design choices are clearly motivated and supported by ablations. In-block positions improve accuracy by 6.2 points, and layer-wise allocation provides additional improvements under the same memory budget.

  3. The paper demonstrates practical serving benefits in vLLM, including 3.54× lower KV memory and 3.2× faster problem completion. Comparisons with R-KV and LightThinker further support the results.

Critical weaknesses
  1. The evaluation only uses Qwen3-4B-Base and mathematical reasoning benchmarks. It is unclear whether the method generalizes to other model sizes, architectures, or reasoning tasks.

  2. The method uses a fixed compression budget for every reasoning block within each layer. Different blocks may contain different amounts of important information, so uniform block-level allocation may not be optimal.

Questions for authors
  1. Have the authors tested AbridgeKV on larger models or other long-generation tasks, such as code generation and agentic reasoning?

  2. How should the compression block size be chosen in practice? Could adaptive block sizes further improve the accuracy–memory trade-off?

1 comment
AuthorsOpenReview ↗

Author Response

Thank you for the encouraging review. You asked whether the method generalizes to another domain and to a larger model. We trained code-generation models and Qwen3-8B models with the same recipe. The method generalizes: the per-layer model stays within one point of full KV in both settings.

W1 and Q1: other tasks and larger models

The method carries over to code generation. We trained full-KV, uniform and per-layer models on long chain-of-thought competitive-programming solutions with the same recipe as for math, and evaluated the three models on the 336 hard and medium LiveCodeBench problems released between August 2024 and April 2025 (16 samples per problem, graded on the hidden tests). Full KV solves 25.4% and the per-layer model solves 25.1%. The gap is 0.3 points

−1.5,+0.9−1.5, +0.9

at the same output length. Uniform allocation is 1.9 points below full KV, and per-layer allocation recovers most of the loss (+1.6 over uniform

+0.3,+2.8+0.3, +2.8

), so per-layer allocation matters in code as in math.
The per-layer schedule measured on math transfers to code. We re-measured the per-layer schedule on the code model, at a cost of about one GPU-hour. The code schedule agrees with the math schedule on 32 of 36 layers. Which layers need more rows appears to be a property of the model more than of the domain.
The method carries over to a model twice the size. We trained the same three models from Qwen3-8B-Base with the same recipe. On the 446-problem suite full KV scores 48.9 and the per-layer model scores 48.0. The gap is 0.9 points

−2.4,+0.5−2.4, +0.5

at the same output length, and matches the gap at 4B under the same protocol. There is no sign that compression costs more as the model grows: with uniform allocation the gap is 1.2 points at 8B against 2.3 at 4B. The 8B per-layer model used the 4B schedule unchanged. With one training seed per model at 8B, the 0.3-point lead of per-layer over uniform allocation is not resolved.
Agentic reasoning. We have not yet evaluated agentic tasks, and the paper will list agentic evaluation as a limitation. The mechanism applies to agentic tasks directly, since an agent's trajectory is long generated context, and an agentic evaluation is our next planned experiment.

W2: a fixed budget per block

A fixed budget is what makes the method cheap to train and serve. With a fixed budget the attention mask depends only on token positions, so training runs on stock kernels at 76% of ordinary fine-tuning throughput. Each request's memory is also known in advance, so a server can pack requests and realize the 3.5x saving. A budget that follows the content of each block is a natural extension, as you note, and would need a learned decision about how many rows to write and a mask built per sample.
Per-head allocation does not improve accuracy and costs serving speed or memory. Per-head allocation within a layer did not improve accuracy at equal rows, and with current cache layouts per-head allocation is 3.5x slower or needs 1.5x the memory. Per-layer allocation is the granularity that leaves both the training and the serving kernels unchanged. The paper now lists per-block and per-head allocation as future work.

Q2: choosing the block size

Larger blocks help because more recent text stays exact, so the exact window is the setting to choose. The new control is a C=512 model that also keeps the previous 511 tokens exact. The control recovers the whole gain of C=1024 (+1.6 points over C=512, in two training seeds) while compressing as often as C=512. The window is paid for in memory, since each request holds twice as much exact text in every layer. At our output lengths the memory cost is small (3.36x less KV than full KV, against 3.49x without the window), and the cost grows as generations get shorter.
Per-layer allocation at C=512 is the choice when memory is the constraint. The per-layer model is 0.5 points below the doubled-window control, within the noise of the evaluation, and adds no memory. A larger window is the simpler choice when memory is plentiful. The window and per-layer allocation are alternatives: at C=1024, per-layer allocation adds nothing further.
Adaptive block sizes would give up the position-only attention mask that keeps training and serving cheap. Block boundaries chosen from content need a learned decision about where each block ends and a mask built per sample, as a content-dependent budget does (W2). LightThinker already uses small blocks at content-chosen boundaries, and in our setting LightThinker is about 10 points below full KV even at our row budget. The paper lists adaptive block sizes as future work.
We hope the code-generation and 8B results answer your questions, and we are glad to provide anything further.

Reviewer_jhV3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

ABRIDGEKV trains a reasoning model to replace each completed 512-token response block's KV with learned pause-token KV. Codes use positions inside the original block and fixed per-layer quotas selected from measured sensitivity. The prompt remains uncompressed, and earlier blocks' codes accumulate. On 446 math problems under the paper's 32K cap, with approximately 13.3K mean output length, the 4:1 model approaches a matched full-KV model's accuracy and output lengths. A vLLM implementation reduces physical cache use and improves saturated serving throughput.

Critical strengths
  1. Positioning has a strong controlled result. In-block positions improve over appended positions by 6.2 points at matched training and budget; the gap persists after excluding position-extrapolation cases. Per-layer allocation adds approximately 1.6 points over uniform allocation, and code removal/replacement tests show that the codes carry usable information.
  2. Compression translates into useful physical savings. With matched 2B-token training, the 4:1 model scores 43.0 avg@8 versus 43.8 for full KV, with paired gap -0.8 [-2.3, +0.8], and similar output lengths. Resident KV is approximately 3.5 times smaller. Under saturated serving, reported time per problem is 0.31 times full KV's; the separate 16K decode benchmark gives 1.05-fold throughput at batch one.
Critical weaknesses
  1. The methodological advance is incremental. Learned context and reasoning compression have close precedents in Activation Beacon and LightThinker; the additions are mainly position placement, layer budgets, and block size. The position result is useful, but the block-size sweep changes the exact window, compression frequency, and code capacity together, leaving their relative contributions unresolved.
  2. Evidence for long-dependency reasoning is narrow. Evaluation uses one 4B backbone on math, with approximately 13.3K mean output under the paper's 32K cap; the planted-fact probe tests 1-8 blocks with C=512. Without accuracy broken down by reference output length, it remains unclear whether the aggregate near-full-KV performance holds for the longest generations.
Questions for authors
  • Can the block-size results distinguish the benefit of fewer compression events from that of a larger exact window?
  • For problems with mean full-KV output length above 24K, what is the paired accuracy gap and its confidence interval?
3 comments
AuthorsOpenReview ↗

Author Response Part 1/2

Thank you for the review, and especially for two precise questions that an experiment could settle. We ran a new control for the block-size question and a breakdown by output length for the long-problem question. The gain of larger blocks comes from the exact window, and the gap to full KV does not grow on the longest problems. New results on code generation and on Qwen3-8B extend the evidence to a second domain and a larger model.

W1: novelty

Prior learned compression does not carry over to long generated reasoning. LightThinker was evaluated on outputs of about 3.4K tokens, and Activation Beacon compresses the input, never the model's output. In our setting a typical solution passes through 26 successive compressions, and every later step reasons from the compressed state. Retrained in our stack on our data, LightThinker loses about 10 points in this setting, with or without our row budget, where our method loses 1.8 points. The step from short outputs to tens of thousands of generated tokens is therefore more than incremental.
In-block placement and per-layer allocation account for most of our advantage over LightThinker. Placing summary rows inside the replaced block is worth 6.2 points with data, budget and training matched. With appended positions, the standard choice, the same model is 8.5 points below full KV, close to LightThinker's loss of about 10 points. The reason is geometric: each appended summary pushes all later text further from the earlier text, while in-block positions keep the original distances. Per-layer allocation adds +0.9 points

+0.1,+1.7+0.1, +1.7

, and two training seeds now confirm the gain. No prior work on compressing generated reasoning makes or ablates in-block placement or per-layer allocation.
We do not claim block size as a contribution. The block-size sweep reports how accuracy changes with block size at a fixed 4:1 compression ratio. The original sweep changed compression frequency, code capacity and the exact window together. The control under Q1 separates the three factors and shows that larger blocks help because more recent text stays exact.

Q1: fewer compressions or a larger exact window?

The gain comes from the exact window, not from fewer compressions. We trained the control your question calls for: a C=512 model that also keeps the previous 511 tokens uncompressed. The control compresses exactly as often as C=512 and into as many rows per compression, but has the same average exact window as C=1024. The control recovers the whole gain of the larger block (+1.6 points over C=512

+0.5,+2.6+0.5, +2.6

across two training seeds) and matches C=1024. C=1024 also halves the number of compressions, and the halving adds nothing.
A larger exact window raises accuracy at the cost of memory. Compression frequency and code capacity are held fixed in the control, and only the exact window changes, so the three factors of the sweep are now separated. Per-layer allocation at C=512 is an alternative without the extra memory: the per-layer model is 0.5 points below the control, within the noise of the evaluation. We will add the control and state the block-size result as an effect of the exact window.

AuthorsOpenReview ↗

Author Response Part 2/2

Q2: accuracy on the longest problems

The gap does not grow with length. On the 9 problems where full KV averages more than 24K tokens, the paired gap is +3.5 points in our favour

−1.4,+9.7−1.4, +9.7

. Nine problems cannot resolve a small gap, so we also report the neighbouring bin and the trend. On the 156 problems between 16K and 24K the gap is −0.6

−1.9,+0.8−1.9, +0.8

, which is smaller than the overall gap, and across all 446 problems the gap changes by +0.1 points per 8K tokens of output

−0.8,+1.0−0.8, +1.0

. Compression error shows no sign of accumulating with length.
The trend holds when problem length is measured on independent samples. The trend is unchanged when each problem's length is taken from an independent full-KV training seed, so problems are not selected on the samples being scored. The full breakdown by length (−1.3, −3.6, −0.6 and +3.5 points for under 8K, 8K to 16K, 16K to 24K and above 24K) goes in the appendix.

W2: breadth of the evidence

Exact recall declines gradually over 24 compressions, and approximate recall stays high. We extended the planted-value probe from 8 to 24 compressions (about 12K tokens). Full KV recalls the planted value in 98% of cases. Our model recalls the exact value in 73% of cases after one compression and in 57% after 24, so most of the loss occurs at the first compression. Approximate recall holds up better: at least three of the four digits are right in every case after one compression and in 89% of cases after 24 (details in our answer to CzDQ Q4).
The result carries over to code generation. We trained full-KV, uniform and per-layer models on long chain-of-thought code generation. On 336 hard and medium LiveCodeBench problems the per-layer model is 0.3 points below full KV

−1.5,+0.9−1.5, +0.9

at the same output length.
The cost of compression does not grow with model size. With Qwen3-8B-Base and the same recipe, the per-layer model is 0.9 points below full KV

−2.4,+0.5−2.4, +0.5

on the 446-problem suite, again at the same output length.
We hope the window control, the length breakdown and the code-generation and Qwen3-8B results address your concerns, and we are glad to answer any further questions.

Dear Reviewer jhV3,

Thank you again for your review. As the discussion period closes soon, we wanted to briefly follow up on our response above, which runs the two experiments your questions asked for:

  • Block size: a new control separating the two effects shows that the gain from larger blocks comes from the larger exact window, not from fewer compression events.
  • Longest problems: broken down by output length, the gap to full KV does not grow on the longest problems, and the trend holds when length is measured on independent samples.

We also added results on code generation and on Qwen3-8B, extending the evidence beyond one domain and one model, and clarified how our setting differs from LightThinker and Activation Beacon. We would be grateful to know whether these address your concerns, and we are happy to clarify anything that remains.

The authors

Reviewer_CzDQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

AbridgeKV compresses the KV cache of long reasoning generations. After each block of 512 generated tokens, learned pause tokens encode the completed block into their KV entries, and the original block's entries are discarded. The main design choices are positions inside the replaced block, layer-specific cache quotas informed by measured utility, and relatively large compression blocks. The paper compares variants using the same base model and an SFT budget of 2B original response tokens, and implements the method in vLLM.

Critical strengths
  1. The principal ablations are comparatively well controlled, and the evaluation extends beyond a single small AIME set. With the same base model, training-data order, and response-token budget, layer-specific allocation improves suite accuracy by approximately 1.5 percentage points over uniform allocation. Positions inside the replaced block also provide a substantial benefit. The appendix addresses the alternative explanation that the positional result arises only from extrapolation beyond the training range. A paired, problem-level bootstrap across 446 problems is more convincing than an isolated result on 30 questions.
  2. The paper connects the compression algorithm to an actual serving implementation. It implements layer-specific cache quotas instead of estimating memory savings solely from nominal ratios, and reports throughput at batch sizes 1, 8, and 32, cache occupancy, generated length, and truncation rates. Interventions on the code KV entries provide causal evidence that the compressed state supports subsequent recall. These results make a credible case for practical value on the studied mathematics model.
Critical weaknesses
  1. The comparison does not fully isolate the advance over the closest prior methods. LightThinker, Activation Beacon, DMC, MEMENTO, and HybridThinker already study learned summaries, online KV compression, or heterogeneous cache budgets; the paper cites these works. The contribution should therefore be framed around the specific positional and allocation choices. In the LightThinker reproduction, the average persistent budget is 97 rows per 512 tokens, versus 128 for AbridgeKV, while segmentation and positions also differ. The 9.8-point accuracy gap cannot be attributed entirely to compression quality or positions without a comparison at matched resident bytes and a clear tuning protocol. R-KV 4096 achieves 42.6%, compared with 43.0% for AbridgeKV, with a paired difference interval crossing zero. Its budget is also somewhat larger, so neither significant accuracy superiority nor an exactly matched budget is established by this comparison.
  2. The accuracy and efficiency claims need tighter boundaries. The interval against full KV permits a decrease of approximately 2.3% points; failure to detect a difference does not establish non-inferiority. Independent training seeds for the compressed model are missing. Serving comparisons also mix configurations: LightThinker uses synchronous scheduling, real prompts, and sampling, whereas other measurements largely use asynchronous scheduling and generation-length replay. The R-KV port compacts every 64 steps rather than the reference's 128. The disclosures are welcome, but implementation and scheduling can affect the timing ranking. Finally, training costs are only partially characterized: total GPU-hours, pilot training, and quota-search costs are needed to assess the amortization of deployment gains.
Questions for authors

Q1 Experiment affecting the main comparison. Can LightThinker be tuned to the same average resident bytes as AbridgeKV, with accuracy, generated length, and training cost reported together? A segmentation-by-position control would help separate the proposed design choices. The existing appended-position and block-size ablations are useful but do not fully replace a matched comparison with the closest method. Q2 Experiment affecting efficiency claims. Can you report accuracy-throughput curves, individual-request latency, and p95 latency with the same real prompts, sampling, scheduling, memory cap, and timing boundaries? Please include both fixed concurrency and each method's best concurrency, and test R-KV compaction every 64 and 128 steps to distinguish algorithmic effects from port overhead. Q3 Resource clarification. Please report end-to-end SFT GPU-hours, peak memory, the computation introduced by code tokens, and the extra costs of the uniform pilot model and quota calibration. Please also identify reproducible checkpoint, environment, and serving-patch versions. Q4 Targeted analysis . The four-digit recall probe covers approximately 1-8 blocks. Can you analyze recall and constraint retention over longer distances and provide failure cases?

4 comments
AuthorsOpenReview ↗

Author Response Part 1/3

Thank you for a careful and constructive review. In response we retrained LightThinker at our row budget, trained a second seed of the per-layer and uniform models, and timed all methods under one serving protocol. Under each control our method remains more accurate than LightThinker and faster than R-KV and full KV. The accuracy cost against full KV is now measured at 1.8 points.

W1 and Q1: the advance over LightThinker and R-KV

No prior compressor of generated reasoning makes or ablates our two design choices. The first choice is where the summary rows are placed (inside the replaced block), and the second is how the row budget is divided across layers. The revised paper frames the contribution around the two choices, as you suggest. LightThinker and HybridThinker give every summary the same size; the uneven budgets in the literature belong to training-free eviction, or arise in DMC as a by-product of merging. MEMENTO's summaries are text the model writes, and Activation Beacon compresses the prompt, never the output.
At a matched budget LightThinker is still about 10 points less accurate. As you asked, we retrained LightThinker at our row budget. With 12 cache tokens per thought, LightThinker holds 126 rows per 512 tokens against our 128. The data, the data order and the training budget are the same, and every other setting follows the published LightThinker configuration for Qwen. The accuracy does not move: 33.2, against 33.3 at the original budget. The row budget therefore does not explain the gap. LightThinker's outputs are also 10% longer than full-KV outputs, so the lower accuracy comes with more decode steps.
The gap comes from two choices that our ablations isolate: segmentation and position. LightThinker compresses at every paragraph break. On our outputs a paragraph break comes about every 53 tokens, so LightThinker compresses ten times as often as our method. At a fixed ratio our own accuracy already falls when the block shrinks from 512 to 256 tokens. LightThinker's summaries take appended positions, and appended positions cost 6.2 points against in-block positions in our stack with everything else equal. LightThinker's boundaries also depend on content, and content-dependent boundaries rule out the position-only masks that make training and serving efficient (the released LightThinker code builds dense masks and is limited to 4K tokens).
R-KV matches our accuracy only by writing nearly 40% more tokens, so R-KV is slower than full KV. Against R-KV at budget 4096 we claim an advantage in cost, not in accuracy. R-KV reaches our accuracy with a slightly larger budget, and a third of R-KV's samples run into the 32K cap against 3% of our samples. Every extra token is a decode step. At half the budget R-KV falls 4 points below our model at a comparable budget, and more than half of R-KV's samples hit the cap, so R-KV degrades faster than our method as the budget shrinks.

AuthorsOpenReview ↗

Author Response Part 2/3

W2: non-inferiority and seeds

With two training seeds the accuracy cost is 1.8 points. We trained a second seed of both the per-layer and the uniform model (different initialization, data order and hardware) and evaluated 16 samples per problem per seed. Over both seeds the per-layer model is 1.8 points below full KV, with a 95% interval of [−3.2, −0.5] that includes the variation between training seeds. Uniform allocation is 2.7 points below full KV, and per-layer allocation beats uniform allocation in both seeds (+0.9

+0.1,+1.7+0.1, +1.7

pooled). The abstract and text now report the measured gap and the interval in place of "matches full KV".
The 1.8 points buy 3.5x less KV and 3.2x faster completion, and neither baseline offers a better trade. Both figures are measured under saturated serving. LightThinker loses about 10 points, and R-KV keeps accuracy only by writing nearly 40% more tokens, so R-KV runs slower than full KV.
The serving setup and the training cost are answered under Q2 and Q3.

Q2: one serving setup, p95 latency and R-KV's compaction interval

Under one serving protocol our model is the fastest method. We re-timed full KV and our model under exactly the protocol used for LightThinker (synchronous scheduling, real prompts, sampling, the same 512 problems). Our model takes 0.44x full KV's time per problem, LightThinker takes 0.63x, and R-KV under the same scheduling takes 1.8x. The p95 completion time agrees with the mean: our model's p95 is 0.30x full KV's p95 at each method's best concurrency and 0.34x at a common concurrency of 64. Under a KV memory cap our advantage grows. One appendix table will show all of the settings side by side.
The ranking does not depend on the serving setup, because the speed-up comes from fitting more requests in memory. Each request holds 3.5x less KV and writes no more tokens than with full KV, so more requests run at once. The serving setup changes the size of the speed-up, not the direction. A single request on an idle GPU decodes at about full-KV speed, as the paper states.
At the reference interval of 128 steps R-KV still takes 1.8x full KV's time. Moving from 64 to 128 steps cuts R-KV's time per problem by 40%, and the paper will report both settings. R-KV's accuracy is unchanged and still ties our model (43.2 against 43.0). R-KV's cost is the extra output tokens, and a faster implementation cannot shorten the output.

AuthorsOpenReview ↗

Author Response Part 3/3

Q3: training cost and versions

Training costs about a third more than ordinary fine-tuning, at the same peak memory. On 8 H100 GPUs with the same data and token budget, full-KV fine-tuning takes about 93 GPU-hours and per-layer training takes about 123 GPU-hours, with peak memory within 2%. The code tokens add 25% to 50% more positions during training. The implementation uses stock attention kernels and has not been optimized.
The per-layer schedule is cheap to compute. Computing the schedule needs the uniform model (about 100 GPU-hours; the uniform model is also a reported arm) and about one GPU-hour of calibration, with no retraining. At the measured serving speeds the whole pipeline pays for itself after about 133K problems served.
Versions. Release repository commit 794f330, torch 2.8.0, vLLM 0.13.0 with our overlay, and the R-KV authors' port (vLLM 0.25.1, commit 6715468b) run unmodified. We will release the checkpoints on acceptance.

Q4: recall beyond 8 blocks

Exact recall does not collapse with distance, and most of the loss occurs at the first compression. We extended the probe to 24 compressions (about 12K tokens). The probe is a worst case for compression, because a random four-digit value is stated once and the solution never uses the value again. Full KV recalls the value in 98% of cases. Our model recalls the exact value in 73% of cases after one compression, 70% after 8 and 57% after 24, and never when the block's code rows are removed. The first compression therefore costs 25 points of recall, and the next 23 compressions cost 16 more.
In a failed recall the model reads the right block and returns a near copy of the value. Of the 106 failed recalls, 87 get three of the four digits right under teacher forcing, most often missing the leading digit (1607 recalled as 2607). At least three of the four digits are right in every case after one compression and in 89% of cases after 24. Failed recalls attend to the right block as strongly as successful recalls. The model therefore finds the right block in the failed cases too, and the code rows return a slightly degraded copy of the value.
Problem constraints stay exact, and the accuracy gap does not grow with solution length. The prompt is never compressed, so the model reads every constraint stated in the problem verbatim at every step. Only intermediate results pass through compression, and intermediate results that the solution reuses are carried well: when the next block reuses a number, the model assigns probability 0.99 to the number's later digits with the code rows and 0.69 without the code rows. On the task itself the gap to full KV does not widen with output length (+0.1 points per 8K tokens of output

−0.8,+1.0−0.8, +1.0

), so longer solutions do not lose more accuracy. The new figure and the failure cases go in the appendix.
We hope the measured gap, the matched-budget baselines and the single serving protocol resolve the concerns about soundness, and we are glad to answer any further questions.

Dear Reviewer CzDQ,

Thank you again for your detailed review. As the discussion period closes soon, we wanted to briefly follow up on our response above. In short: at a matched resident budget, LightThinker remains about 10 points less accurate, and our ablations attribute the gap to segmentation and position; R-KV matches our accuracy only by generating nearly 40% more tokens, which makes it slower than full KV; and with two training seeds the accuracy cost of AbridgeKV is 1.8 points for 3.5x less KV memory and 3.2x faster completion. We would be grateful to know whether these address your concerns, and we are happy to clarify anything that remains.

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response for All Reviewers

We thank all three reviewers for careful and constructive reviews. The reviews agree that the paper's strengths are the controlled ablations and the working serving implementation. The reviews also share three concerns: the statistical strength of the main claim, the fairness of the baseline comparisons, and the breadth of the evaluation. In response we trained second seeds of the main models, retrained LightThinker at our row budget, re-ran R-KV at its reference compaction interval, timed all methods under one serving protocol, and trained models for a larger backbone, a second domain and a new block-size control.
Statistical strength: compression costs 1.8 points and buys 3.5x less KV and 3.2x faster completion. With a second training seed of each model and 16 samples per problem per seed, the per-layer model is 1.8 points below full KV [−3.2, −0.5]. Per-layer allocation beats uniform allocation in both seeds (+0.9

+0.1,+1.7+0.1, +1.7

), so the gain from per-layer allocation is not seed noise. The paper now states the measured cost in place of "matches full KV".
Fair baselines: under matched conditions LightThinker is still less accurate and R-KV is still slower. LightThinker retrained at our row budget is unchanged, about 10 points below full KV, so the row budget does not explain the gap. R-KV at its reference compaction interval ties our accuracy only by writing nearly 40% more tokens, and takes 1.8x full KV's time. Timed under one serving protocol, our model is the fastest method, and the p95 completion time shows the same advantage over full KV as the mean.
Breadth: the result carries over to a larger model, to code generation and to the longest outputs. The per-layer model is 0.9 points below full KV on Qwen3-8B

−2.4,+0.5−2.4, +0.5

and 0.3 points below full KV on LiveCodeBench code generation

−1.5,+0.9−1.5, +0.9

. The gap to full KV does not grow with output length, and recall through the compressed state declines only gradually out to 24 compressions.
Block size: the benefit of larger blocks is the larger exact window, not fewer compressions. A new control compresses every 512 tokens and also keeps the most recent 512 tokens exact. The control recovers the whole gain of 1024-token blocks, in two training seeds.
All of the new results will be added to the paper. Detailed answers follow under each review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.