acceptodds
Under review as a conference paper at ICLR 2027

ARC-KV: Amortizing Anchor Search for Reconstruction-Based KV Cache Compaction

Abstract

Long-context large language model inference is bottlenecked by KV caches that grow linearly with sequence length. This burden is especially severe for long, reusable context prefixes, whose cache must serve many downstream queries. Reconstruction-based methods such as Attention Matching achieve strong downstream task performance with compact KV caches. However, iterative anchor search dominates the compaction cost of OMP-based Attention Matching. This motivates our selective amortization principle of learning a reusable anchor-selection policy across contexts while retaining context-specific reconstruction. In this work, we propose ARC-KV, a novel reconstruction-based KV cache compaction method that follows this principle. To this end, we first train a value-aware indexer to select real-key anchors in a single scoring pass. ARC-KV then applies convex-hull-constrained key merging and fits an attention-mass bias and compact values against the full cache. At inference time, ARC-KV builds the compact cache once per context using the frozen indexer and reuses it for all subsequent queries. Extensive experiments demonstrate that ARC-KV outperforms reported compaction methods in most settings across QuALITY, RULER, and LongBench on Llama-3.1-8B-Instruct. In particular, at 10% KV retention on QuALITY, ARC-KV improves accuracy from 0.6409 to 0.6474 over Attention Matching while reducing compaction time by a factor of 25.73, from 959.8 s to 37.3 s.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache compression, long-context inference, large language models, efficient inference, KV cache compaction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pCKaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ARC-KV, a reconstruction-based KV-cache compaction method for long-context LLM inference. Its key idea is to amortize the expensive per-context anchor search in Attention Matching: a lightweight value-aware indexer is trained once per base model to select real-key anchors in a single pass, while context-specific reconstruction is retained through attention-response-based key merging and closed-form fitting of attention-mass biases and compact values. The method is evaluated on QuALITY, RULER, and LongBench using Llama-3.1-8B-Instruct, with additional experiments on Qwen3-8B and Gemma-3-12B-IT, and compared against several cache eviction and reconstruction baselines across retention ratios. ARC-KV ranks first among reported compacted methods in 13 of 15 main settings; for example, at 10% retention on QuALITY it improves accuracy from 0.6409 to 0.6474 over Attention Matching while reducing compaction time from 959.8 s to 37.3 s. Ablation and profiling studies further support the value-aware selection, key merging, and nonuniform per-head allocation, although the reported efficiency measurements primarily concern cache compaction rather than end-to-end inference latency.

Critical strengths

ARC-KV makes a technically coherent contribution by separating anchor selection from cache reconstruction. Its reusable value-aware indexer removes the expensive iterative search in Attention Matching, while context-specific key merging and closed-form fitting preserve the flexibility needed for different contexts.

The empirical evaluation is broad and convincing. Across QuALITY, RULER, and LongBench, ARC-KV outperforms reported compacted baselines in most settings and remains competitive across additional model architectures. It also achieves substantial compaction-time reductions—for example, a 25.73× speedup at 10% retention on QuALITY while slightly improving accuracy—supported by ablations and runtime profiling.

Critical weaknesses

The efficiency claim is not yet demonstrated at the end-to-end inference level. Reported speedups measure cache-compaction time only and exclude full decoding latency; moreover, at long contexts, reference-query construction accounts for most of ARC-KV’s remaining runtime. Thus, the 25.73× compaction speedup may not translate into a comparable serving or latency improvement, especially when a prefix is reused only a small number of times.

The evaluation does not fully establish generality. Most results use Llama-3.1-8B-Instruct, while the additional model study tests only one retention ratio and two models. The method is also evaluated mainly for one-shot compression of static prefixes, with limited evidence for generated-state recompression, much longer contexts, or dynamic workloads. In LongBench, the default OMP-based Attention Matching baseline is replaced by AM-HA because of its cost, which further weakens the fairness of the strongest comparison.

Questions for authors
  1. Can the authors report end-to-end serving results, including prefill, compaction, decoding, and repeated queries? Since reference-query construction dominates ARC-KV’s cost at long contexts, the practical break-even point for prefix reuse is important.

  2. Why is AM-HA used instead of the default OMP-based Attention Matching on LongBench? Could the authors provide a direct comparison with OMP-based AM on a smaller matched subset, or report the computational budget and failure rate for both implementations?

  3. Are the reported speedups measured with identical hardware, kernels, precision, batching, and reference-query construction costs for all methods? Please clarify whether baseline implementations were independently optimized or used as released.

  4. How are the nominal retention ratios converted into actual memory usage? Please report the total footprint of compact keys, values, attention-mass biases, temporary tensors, and any metadata, since per-head allocation and chunking may make nominal ratios differ from actual compression.

  5. Were the values of the key-merging coefficient, loss weights, per-head budgets, and other hyperparameters selected using a validation set that is disjoint from all reported test benchmarks? This is important because the accuracy improvements over AM are sometimes relatively small.

  6. How statistically robust are the reported gains? Please provide confidence intervals or results over multiple seeds, particularly for the 0.23–3.63 point improvements on QuALITY and the 50-example-per-task LongBench evaluation.

  7. Does the indexer training corpus overlap in documents, domains, or distributions with QuALITY, RULER, or LongBench? Even without QA labels, overlap could make it difficult to distinguish general cache-selection ability from benchmark-specific adaptation.

  8. For Qwen3 and Gemma, were new indexers trained separately using the same data and compute budget as the Llama indexer, or was the Llama indexer transferred? Clarifying this would help assess whether ARC-KV is broadly reusable or requires substantial model-specific training.

6 comments
AuthorsOpenReview ↗

Response by authors (1/4)

We sincerely thank the reviewer for the detailed and constructive review, and for finding the decomposition of anchor selection from reconstruction technically coherent and the evaluation broad and convincing. We ran new experiments and addressed each question below; all results and clarifications will be added to the revision.

W1 / Q1. End-to-end serving and the break-even point for prefix reuse.

This is a well-founded concern about the real end-to-end serving environment. We integrated the compaction methods into the production serving framework vLLM and measured full serving-pipeline performance on a single NVIDIA B200 GPU. At 32K and 64K context lengths, we record the wall-clock time of three key stages: cache build (context prefill, repeat prefill, selection, merging, fitting), per-query prefill, and 128-token decoding.

  • Build: 1,231.9 / 2,003.8 s for OMP-AM vs. 77.9 / 137.9 s for ARC-KV (about 15× faster), including the reference-query construction the reviewer highlighted. This repeat prefill of the context is not specific to ARC-KV: AM builds the same reference queries, and KVzip scores keys with the same kind of context-reconstruction pass. ARC-KV removes the iterative search on top of it, which is why it now dominates the remaining build time.
  • Serving: each query takes 15% / 22% less time than with the full cache one at a time, and is 4.2× / 5.3× faster with 100 concurrent queries and 6.9× / 8.9× faster with 768.
  • Break-even: against OMP-AM, ARC-KV is cheaper for any number of queries, since both serve from the same-size cache; including the one-time indexer training (147 B200 GPU-hours), it overtakes OMP-AM after about 280 contexts at 64K. Against the full cache, the build is repaid after about 820 / 870 queries served one at a time, after which every query is cheaper and the cache occupies about 20× less memory at ρ = 5%.
ContextBuild [s]: ARC-KV / OMP-AMPer query, one at a time [s]: full / ARC-KV100 concurrent queries [s]: full / ARC-KV
16K38.1 / 613.10.567 / 0.5213.50 / 1.15 (3.0×)
32K77.9 / 1,231.90.613 / 0.5186.24 / 1.48 (4.2×)
64K137.9 / 2,003.80.707 / 0.54811.66 / 2.21 (5.3×)

We will state explicitly that the 25.73× figure refers to compaction time and add these end-to-end results in the revised version.

AuthorsOpenReview ↗

Response by authors (2/4)

W2. Generality.

  • More budgets and models. We extended the cross-model study to five budgets (ρ = 0.01–0.20) on RULER-4K for Qwen3-8B and Gemma-3-12B-IT. ARC-KV is the best compacted method in 7 of 10 settings: every budget on Gemma, and ρ ≤ 0.02 on Qwen. On Qwen at ρ ≥ 0.05 the original KVzip leads, consistent with Table 2; with the same per-head allocation as ARC-KV, however, KVzip drops sharply (27.8 vs. 53.7 at ρ = 0.05) and ARC-KV leads in all 10 settings, indicating that the remaining gap on Qwen stems from KVzip's own budget allocation, which is orthogonal to ARC-KV's anchor selection and could be combined with it. ARC-KV outperforms AM, SnapKV and EA in every setting. We also found that the RULER results in Figure 2b and Table 2 used strict string matching; we now report the official metric, and will correct both accordingly in the revised version; on Llama RULER-4K at ρ = 5%, ARC-KV's lead over AM holds under the official metric (87.0 vs. 58.5, vs. 73.94 vs. 48.00 under strict matching).
  • Longer contexts. On RULER at 16K / 32K / 64K (ρ = 5%), ARC-KV scores 73.4 / 70.6 / 65.3 vs. 54.3 / 43.3 / 47.7 for the best baseline (KVzip), with non-overlapping 95% CIs, even though the indexer was trained only on contexts of at most 16,884 tokens. With a larger budget (16K, ρ = 10%), ARC-KV reaches 93.6, statistically indistinguishable from full context (94.5) and ahead of OMP-AM (90.1) and KVzip (80.1).
  • Generated-state recompression and dynamic workloads. These are outside the current scope, which targets one-shot compaction of static reusable prefixes (Section 3.5, Conclusion). Because the indexer selects anchors in a single pass without per-context search, it is a natural fit for repeated recompression, and we plan to extend ARC-KV to chain-of-thought traces and multi-turn agent histories in future work.

Cross-model results on RULER-4K (%, official metric; full context 99.5 for Qwen3-8B and 99.3 for Gemma-3-12B-IT) against KVzip, the strongest baseline in every setting; AM, SnapKV and EA are below ARC-KV everywhere:

ρQwen3-8B: KVzipQwen3-8B: KVzip (per-head)Qwen3-8B: ARC-KVGemma-3-12B-IT: KVzipGemma-3-12B-IT: ARC-KV
0.0114.917.522.715.615.9
0.0218.619.830.218.729.5
0.0553.727.846.229.146.4
0.1091.856.684.853.567.3
0.2099.094.095.691.193.6

Long-context RULER (official metric, 95% CI):

Method16K, ρ = 5%16K, ρ = 10%32K, ρ = 5%64K, ρ = 5%
Full context94.594.589.488.2
ARC-KV73.4 [68.0, 78.2]93.6 [89.8, 96.7]70.6 [65.3, 75.3]65.3 [60.0, 70.3]
OMP-AM41.6 [36.1, 47.2]90.1 [85.4, 94.3]31.8 [27.9, 36.2]36.2 [30.9, 41.7]
KVzip54.3 [48.1, 60.4]80.1 [76.0, 84.2]43.3 [37.2, 49.3]47.7 [42.4, 53.1]
SnapKV39.4 [34.5, 44.0]49.1 [44.3, 53.6]40.8 [36.1, 45.6]45.4 [41.2, 49.8]

Q2. AM-HA instead of OMP-based AM on LongBench.

We ran OMP-AM directly on a matched subset of 50 LongBench instances (10 each from NarrativeQA, HotpotQA, 2WikiMQA, GovReport and TriviaQA, spanning four task categories), with all methods on identical instances (LongBench score ×100):

Methodρ = 0.02ρ = 0.05ρ = 0.10
ARC-KV48.147.553.2
OMP-AM44.847.449.7
AM-HA39.747.749.1

ARC-KV matches or exceeds OMP-AM at every budget. AM-HA tracks OMP-AM within 0.6 points at ρ = 0.05 and 0.10, so the substitution did not inflate ARC-KV's margin there; at ρ = 0.02 AM-HA is weaker, but ARC-KV still exceeds OMP-AM by 3.3 points on this subset. The substitution was driven by cost: OMP-AM is 3.6× / 11.5× / 33× slower per instance than AM-HA, and running it on all 1,050 instances at ρ = 0.10 would take about 520 B200 GPU-hours. Neither OMP-AM nor AM-HA failed or timed out on any instance.

AuthorsOpenReview ↗

Response by authors (3/4)

Q3. Identical measurement conditions and baseline implementations.

All methods are timed on the same NVIDIA B200 GPU with the same precision and batching, and every compaction time includes reference-query construction. Baselines use released code with default settings: AM (OMP-fast, k = 4, refit every two steps), AM-HA, SnapKV and KVzip from the official Attention Matching implementation, and Expected Attention from the kvpress repository. We did not further optimize any method beyond its released configuration. For serving, the full cache runs on vLLM 0.33.0 with FlashInfer FlashAttention-2 0.7.3, a fast and well-optimized attention backend; the compact caches of ARC-KV and AM are served with the same FlashAttention-2 kernel, modified only to add the attention-mass bias β.

Q4. From nominal retention to actual memory.

  • Stored footprint: Ck, Cv and β together take 1.00 / 5.02 / 18.51% of the full KV cache at 64K for nominal 1 / 5 / 20%; β adds only about 0.4% on top of Ck and Cv. At equal budget every method stores essentially the same number of bytes. Per-head metadata (one budget length per KV head) is negligible.
  • Realized vs. nominal: rounding (Eq. 16) and chunking keep the number of stored states 0.03–0.60% below nominal at ρ ≤ 5% (16K–64K), and 7.8% below at ρ = 20% because 11 heads are capped at their full length. It never exceeds the nominal budget.
  • Temporaries: the largest per-group temporary (indexer scoring or the Cv solve) is at most 10.3 GB and does not grow with context length; peak compaction memory grows only through the reference-query state. Compacting a 64K context fits on a single 80 GB GPU (66.8 GB in total). At 16K, ARC-KV's peak (38.80 GB) is within 0.58 GB of AM's (38.22 GB), the difference being the indexer weights.
ContextStored Ck + Cv + β, % of full KV (ρ = 1% / 5% / 20%)Stored states vs. nominal (ρ = 1% / 5% / 20%)Total GPU memory to compact, ρ = 5% [GB]
16K1.00 / 5.02 / 18.50−0.60% / −0.07% / −7.84%38.8
32K1.00 / 5.02 / 18.51−0.04% / −0.07% / −7.83%50.8
64K1.00 / 5.02 / 18.51−0.11% / −0.03% / −7.82%66.8

Q5. Hyperparameter selection.

The key-merging coefficient λm and the loss weights were selected on our validation set of 60 held-out development articles (Appendix C.2), disjoint from all test benchmarks. The per-head budgets were not tuned by us: we reuse unchanged the weights released with Attention Matching, and every method, including AM itself, uses the same weights. We also measured accuracy for λm ∈ {0.25, 0.5, 0.75}: the three values lie within 1.4 points on QuALITY, 2.6 points on the LongBench subset, and 1.7 / 0.5 points on RULER-4K at ρ = 0.05 / 0.10; only RULER-4K at ρ = 0.02 favors a small λm, consistent with the trend in Table 5. The default is not the best test-set value on QuALITY at any budget or on RULER-4K at ρ = 0.10, and every λm still outperforms AM (OMP) on all three benchmarks at every budget. The results therefore do not rely on test-set tuning.

Benchmarkρλm = 0.25λm = 0.5λm = 0.75AM (OMP)
QuALITY0.0262.5*62.961.559.1
QuALITY0.0564.4*64.565.160.7
QuALITY0.1064.7*65.665.264.1
RULER-4K0.0261.4*57.752.226.1
RULER-4K0.0587.0*86.185.358.5
RULER-4K0.1098.1*98.598.693.7
LongBench subset0.0247.246.548.1*44.8
LongBench subset0.0549.048.647.5*47.4
LongBench subset0.1052.750.653.2*49.7

* marks the default used in the paper. QuALITY values are accuracy (%) with the same single seed for every λm; RULER-4K values are accuracy (%) under the official metric; LongBench values are mean task scores ×100.

Reviewer_Sv4sOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ARC-KV, a learned KV-cache compression framework for long contexts that are reused across multiple queries. Instead of relying on expensive iterative anchor selection, ARC-KV trains value-aware indexers to predict important cache positions in a single scoring pass, followed by key merging, attention-mass correction, and value reconstruction. The method uses a differentiable surrogate to train the indexers without requiring optimal anchor labels. Experiments across several language models and long-context benchmarks demonstrate competitive compression quality and substantially lower cache-construction costs than optimization-based selection methods.

Critical strengths
  1. This paper replaces iterative anchor selection with a learned scoring mechanism which substantially reduces compression cost.

  2. The training approach is technically interesting. By combining hard anchor selection with a differentiable surrogate, the authors enable end-to-end learning of the indexer without relying on expensive OMP-generated supervision.

  3. The evaluation covers various models, benchmarks, and compression ratios, with detailed measurements of reconstruction quality and computational cost. The paper also reports unfavorable results and discusses several implementation trade-offs, providing useful evidence about the method's practical behavior.

Critical weaknesses
  1. The contribution of the learned value-aware indexer is not sufficiently isolated from other components of the compression pipeline. ARC-KV combines learned selection, non-uniform head budgets, key merging, and value reconstruction, but the experiments do not clearly establish which components drive the improvements.

  2. The non-uniform head allocation has a substantial effect on quality, but the procedure for determining its allocation weights is not sufficiently specified.

Questions for authors
  1. How do the one-time indexer training cost, per-context compression cost, and per-query inference cost affect the overall efficiency advantage? At what scale of context and query reuse does ARC-KV become more efficient than OMP-based and lightweight selection methods?

  2. How reliably does the training surrogate predict reconstruction quality after inference-time key merging and fitting? Do the quality advantages persist on independent future queries and longer contexts, particularly at 16K–64K, with statistically meaningful improvements?

4 comments
AuthorsOpenReview ↗

Response by authors (1/2)

We thank the reviewer for the thoughtful review and for finding the hard-selection plus differentiable-surrogate training technically interesting. We address each point below with new experiments, all of which will be included in the revision.

W1. Isolating the learned value-aware indexer from the rest of the pipeline.

We ran a controlled ablation in which the per-head budgets, reference queries, key merging and β/Cv fitting are fixed and only one component changes (Llama-3.1-8B-Instruct, averaged over ρ = 0.02 / 0.05 / 0.10; all variants share the same seed). All RULER numbers in this response use the official metric; Figure 2b and Table 2 used strict string matching, which we will correct in the revision; the advantage over AM holds (e.g., RULER-4K at ρ = 5%: 87.0 vs. 58.5 under the official metric, vs. 73.94 vs. 48.00 under strict matching).

VariantQuALITY (%)RULER-4K (%)
Full ARC-KV63.982.2
Without value-aware path (QK-only indexer)63.581.4
KL-only training (no reconstruction loss)63.578.8
Without key merging (λm = 0)63.278.2
OMP-fast anchors, same merging and fitting63.952.3

Each component contributes: the reconstruction objective (−3.4 on RULER-4K), the value-aware path (−0.4 / −0.8) and key merging (−0.7 / −4.0). Replacing the learned indexer with OMP anchors yields the same QuALITY accuracy but 29.9 points lower RULER-4K accuracy, which we attribute to OMP's anchors dropping needle keys that only a few queries attend to, whereas the indexer's RMS pooling is designed to retain them. The indexer is not only cheaper than OMP's iterative search but also selects better anchors. The full model also has the lowest held-out attention-output error (0.2835 vs. 0.2840–0.2963 for the variants). Value reconstruction (closed-form β/Cv fitting) is identical in ARC-KV and AM, so the ARC-KV vs. AM comparisons isolate anchor selection and merging.

To separate the indexer from the non-uniform head budgets, we also gave every baseline the same per-head allocation as ARC-KV. In the original Figure 2, AM already used it, while SnapKV, KVzip and Expected Attention followed their default uniform allocation. Under identical allocation, ARC-KV (mean of four seeds) remains the most accurate method on QuALITY (%):

ρAM (OMP)SnapKVKVzipEAARC-KV
0.0259.148.647.152.160.7
0.0560.755.255.358.564.8
0.1064.160.464.561.565.2

We will update Figure 2 to use the same allocation for all methods.

W2. How the per-head allocation weights are determined.

We apologize for the missing detail. We reuse, unchanged, the precomputed per-head weights released with Attention Matching (Zweiger et al., 2026); we did not tune them. AM computes them as follows. Starting from a uniform schedule at ρ = 0.05, the procedure varies the retention ratio of one layer–KV-head pair at a time and records the change in the log-perplexity of the full-context model's own answers, giving a sensitivity curve per pair. Treating these curves as separable, a greedy exchange moves budget shares of 10⁻³ from the pair whose loss increases least to the pair whose loss decreases most, averaging the loss over ρ ∈ {0.01, 0.02, 0.05, 0.10}, until no swap helps. Note that the curves are measured with AM's highest-attention compaction, not with ARC-KV, so the allocation is not tuned to our method. We will add this description to Appendix A.2, together with a pointer to the released weights.

AuthorsOpenReview ↗

Response by authors (2/2)

Q1. Training, compaction and query costs, and the scale at which ARC-KV pays off.

We measured all three costs end to end in vLLM on one B200 (single wall clock, 128-token decoding per query).

  • One-time training: 147 B200 GPU-hours per model, including reference-query construction.
  • Per-context build: 38.1 / 77.9 / 137.9 s at 16K / 32K / 64K for ARC-KV, versus 613 / 1,232 / 2,004 s for OMP-AM.
  • Per query: essentially identical for compact caches of the same size; relative to the full cache at 32K–64K, a query takes 15–22% less time one at a time and runs 4.2–5.3× faster with 100 concurrent queries.
ContextARC-KV build [s]OMP-AM build [s]Build speedupContexts to amortize training
16K38.1613.116.1×~920
32K77.91,231.915.8×~460
64K137.92,003.814.5×~280

Versus OMP-based selection, ARC-KV is cheaper for every number of queries, since it saves 575 / 1,154 / 1,866 s per context at the same per-query cost. Including training, it breaks even after about 920 / 460 / 280 contexts at 16K / 32K / 64K.

Versus lightweight selectors (SnapKV, KVzip, EA), ARC-KV takes longer to build, but all methods store essentially the same bytes (β adds about 0.4%) and serve each query from a cache of the same size, so the comparison reduces to accuracy at equal memory. There ARC-KV leads the best lightweight selector by 19.1 / 27.3 / 17.6 points on RULER at 16K / 32K / 64K (ρ = 5%), and by 8.6 / 6.3 points on QuALITY at ρ = 0.02 / 0.05 (mean of four seeds). Even with twice the cache, KVzip (64.5 at ρ = 10%) stays below ARC-KV at ρ = 5% (64.8). ARC-KV is therefore preferable whenever a prefix is reused enough to amortize one build, or whenever memory is the binding constraint. For reference, against the uncompressed cache, one build is repaid after about 820 / 870 queries at 32K / 64K.

Q2. Surrogate fidelity, independent future queries, longer contexts and significance.

Surrogate fidelity. For all 12,800 article–layer–KV-head cells per ρ on QuALITY, we compared the training surrogate (no merging, training split and solver) with the error of the deployed cache (with merging and the deployment solver), both on the same held-out queries. The Spearman correlation is 0.972 / 0.932 / 0.876 at ρ = 0.02 / 0.05 / 0.10, and adding inference-time merging alone preserves the ranking at 0.998 / 0.994 / 0.994 while lowering the error by 4–6%. Because each merged key stays in the convex hull of its group's real keys, merging corrects a given anchor set almost never reversing which anchor sets are better, so the pre-merge surrogate optimizes the right ordering.

ρSpearman, all cellsSpearman within each (layer, head), median(layer, head) pairs with positive correlationRanking preserved by merging alone
0.020.9720.916100%0.998
0.050.9320.90999.6%0.994
0.100.8760.909100%0.994

Independent future queries. Reference queries come only from repeating the context (Section 3.2), so no benchmark question is ever seen when the cache is built; all reported accuracies are on genuinely future queries.

Longer contexts with significance. On RULER at 16K–64K (99 examples per cell, 95% CIs):

Method16K, ρ = 5%32K, ρ = 5%64K, ρ = 5%
Full context94.589.488.2
ARC-KV73.4 [68.0, 78.2]70.6 [65.3, 75.3]65.3 [60.0, 70.3]
Best baseline (KVzip)54.3 [48.1, 60.4]43.3 [37.2, 49.3]47.7 [42.4, 53.1]

ARC-KV's lower CI bound exceeds every baseline's upper bound at all three lengths, although the indexer was trained only on contexts of at most 16,884 tokens. At 16K with ρ = 10%, ARC-KV reaches 93.6 vs. 94.5 for full context, statistically indistinguishable. On QuALITY, every one of four seeds exceeds AM (OMP) by at least 3.6 points at ρ = 0.05 (mean 64.8 vs. 60.7).

We hope these additions resolve the reviewer's concerns, and we would be glad to provide any further detail.

Thank you for the thorough response. The added experiment results address my main concerns. I am keeping my positive score and support acceptance.

We sincerely thank the reviewer for the constructive comments and for the positive assessment of our work. The suggestions have substantially improved the paper.

Reviewer_X1A5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ARC-KV is a method for reconstruction-based KV cache compression that replaces iterative anchor searching with a learned, value-aware indexer. It uses selected original keys as anchors to merge other keys, while calculating attention-mass bias and compact values ​​tailored to the current context. The indexer is trained for each specific model—with the model’s backbone kept frozen—and the resulting compressed cache can be reused for subsequent queries. The approach is evaluated for accuracy on the QuALITY, RULER, and LongBench benchmarks, and it is shown to reduce the cost of cache construction compared to existing Attention Matching techniques.

Critical strengths
  1. The paper presents a practical improvement that maintains accuracy while reducing the cost of iterative anchor search. Under the QuALITY benchmark with a 10% retention rate, ARC-KV—compared to OMP-based Attention Matching—increases accuracy from 64.09% to 64.74% and reduces cache construction time from 959.8 seconds to 37.3 seconds (Figure 3). This demonstrates the effectiveness of a design that replaces anchor selection with a learned process while preserving the reconstruction of the current context. Note, however, that the 25.73x speedup applies specifically to the cache construction time, not the total inference time.

  2. It demonstrates competitive accuracy across various benchmarks and compression budgets. As shown in Figure 2, Llama-3.1-8B-Instruct achieves the highest scores among the compression methods compared in 13 out of 15 benchmark-retention ratio combinations. Notably, under the 5% retention condition for RULER-4K, it scores 73.94, outperforming the 48.00 score achieved by Attention Matching. In additional evaluations using Qwen3-8B and Gemma-3-12B-IT, it also secures the highest scores in three out of the four tested configurations (Table 2).

Critical weaknesses
  1. The independent contributions of the value-aware indexer and the reconstruction-based learning have not been sufficiently disentangled. While Figure 5(a) demonstrates the utility of a selection criterion that incorporates value information, no direct comparison is provided—such as one that removes the value-aware path from the proposed indexer or employs only KL-based learning. Furthermore, Table 7 shows that simply switching the budget allocation to a uniform distribution results in a 10.29-percentage-point drop in accuracy under the 5% retention constraint. Consequently, it is necessary to isolate and verify the specific gains provided by the core learning components while keeping the budget allocation and reconstruction conditions constant.

  2. There is limited evidence evaluating both accuracy and total execution cost in practical settings where long contexts are repeatedly reused. The accuracy evaluation in RULER is restricted to a 4K context, while the analysis for the 4K–64K range primarily focuses on the time required to build the cache for synthetic contexts. The total time presented in Figure 6 is the sum of measurements taken at individual stages, and the decoding speed shown in Table 4 accounts only for attention operations. Consequently, it has not been sufficiently demonstrated to what extent the overall inference time—including the initial cost of building the cache—can be improved while maintaining accuracy in long-context scenarios.

Questions for authors
  1. Regarding Weakness 1: Under identical budget allocation, reference queries, merging, and reconstruction settings, could you compare the full ARC-KV against two ablations: (1) removing the value-aware pathway from the indexer and (2) training the indexer with only the KL-divergence loss? Please report both downstream accuracy and held-out attention-output reconstruction error.

  2. Regarding Weakness 2: For standard configurations (e.g., 32K or 64K), could you report both the accuracy and the total execution time—including cache construction, query processing, and decoding—and demonstrate how the time savings vary based on the number of queries that reuse the same cache?

4 comments
AuthorsOpenReview ↗

Response by authors (1/2)

We thank the reviewer for the careful reading and for recognizing that ARC-KV preserves accuracy while removing iterative anchor search. The two weaknesses point to analyses that strengthen the paper, and we have run both. All new results will be added to the revised paper.

W1 / Q1. Disentangling the value-aware indexer from reconstruction-based training.

As suggested, we fixed the per-head budget allocation, reference queries, key merging and β/Cv fitting, and changed only the selector or its training objective (Llama-3.1-8B-Instruct, averaged over retention ratio ρ = 0.02 / 0.05 / 0.10; QuALITY: 894 questions; RULER-4K: 550 examples; all variants share the same seed). All RULER numbers in this response use the official metric; Figure 2b and Table 2 used strict string matching, which we will correct in the revision; the advantage over AM holds (e.g., RULER-4K at ρ = 5%: 87.0 vs. 58.5 under the official metric, vs. 73.94 vs. 48.00 under strict matching).

VariantQuALITY (%)RULER-4K (%)Held-out attention-output rel. ℓ2
Full ARC-KV63.982.20.2835
(1) Without value-aware path (QK-only indexer)63.581.40.2842
(2) KL-only training63.578.80.2864
Without key merging (λm = 0)63.278.20.2963
OMP-fast anchors, same merging and fitting63.952.30.2840
  • Both learning components contribute. Training with the KL loss alone costs 3.4 points (i.e., 82.2 − 78.8) on RULER-4K, the task most sensitive to retrieving specific values. The value-aware path gives a consistent gain: under the same data and schedule, removing it lowers accuracy on both benchmarks. The full model also has the lowest held-out attention-output error (0.2835, vs. 0.2842 without the value path and 0.2864 with KL-only training).
  • Merging is complementary. Disabling it costs 0.7 points on QuALITY and 4.0 on RULER-4K and raises the held-out error from 0.2835 to 0.2963, confirming the attention-mass reduction of Table 5 at the task level.
  • The learned selector outperforms OMP. Replacing the indexer with OMP-fast anchors under identical merging and fitting gives the same QuALITY accuracy but is 29.9 points worse on RULER-4K. We attribute this to how OMP selects anchors: it greedily fits the attention mass averaged over the reference queries, so it can drop needle keys that only a few queries attend to, whereas the indexer's RMS pooling (Eq. 3) is designed to retain such keys. OMP also builds the cache about 15× more slowly. Amortizing selection therefore costs no quality; the learned indexer selects better anchors than the iterative search it replaces.
  • Budget allocation is held constant across methods. Regarding Table 7, the per-head weights are the ones released with Attention Matching, reused unchanged and shared by every method. With all baselines using the same allocation, ARC-KV (mean of four seeds) still leads AM (OMP) on QuALITY by 1.6 / 4.1 / 1.1 points at ρ = 0.02 / 0.05 / 0.10, so the gain is not an artifact of the allocation.

QuALITY accuracy (%) with every method on the same per-head allocation (ARC-KV: mean of four seeds). In Figure 2, SnapKV, KVzip and EA used uniform allocation; giving them the same per-head allocation as ARC-KV strengthens these baselines, and ARC-KV still leads:

ρAM (OMP)SnapKVKVzipEAARC-KV
0.0259.148.647.152.160.7
0.0560.755.255.358.564.8
0.1064.160.464.561.565.2
AuthorsOpenReview ↗

Response by authors (2/2)

W2 / Q2. Accuracy and total execution time at long context with cache reuse.

ARC-KV outperforms OMP-AM in both accuracy and build time at every context length, and serves each query faster than the full cache while storing about 20× fewer KV states. To measure end-to-end performance, we integrated both compaction methods into the production serving framework vLLM and recorded full-pipeline wall-clock time on real-text contexts on an NVIDIA B200 GPU. Three stages in the pipelines were profiled: cache build (context prefill, repeat prefill, selection, merging, fitting), per-query prefill, and 128-token decoding. All experiments use fast, well-optimized attention kernels: the full cache runs on FlashInfer's FlashAttention-2 kernel, and the compact caches run on the same kernel, modified only to add the mass bias β. Unlike the stage-sum estimate of Figure 6 and the attention-only decode timing of Table 4, these are wall-clock measurements of the whole pipeline.

Context length32K64K
Cache build time [s]: OMP-AM / ARC-KV / Speedup1,231.9 / 77.9 / 15.81×2,003.8 / 137.9 / 14.53×
Per query, one at a time [s]: full cache / ARC-KV / Speedup0.613 / 0.518 / 1.18×0.707 / 0.548 / 1.29×
100 concurrent queries [s]: full cache / ARC-KV / Speedup6.24 / 1.48 / 4.22×11.66 / 2.21 / 5.28×

Accuracy against all baselines at long context (RULER, official metric, 99 examples per cell, 95% CI):

Method16K, ρ = 5%16K, ρ = 10%32K, ρ = 5%64K, ρ = 5%
Full context94.594.589.488.2
ARC-KV73.4 [68.0, 78.2]93.6 [89.8, 96.7]70.6 [65.3, 75.3]65.3 [60.0, 70.3]
OMP-AM41.6 [36.1, 47.2]90.1 [85.4, 94.3]31.8 [27.9, 36.2]36.2 [30.9, 41.7]
KVzip54.3 [48.1, 60.4]80.1 [76.0, 84.2]43.3 [37.2, 49.3]47.7 [42.4, 53.1]
SnapKV39.4 [34.5, 44.0]49.1 [44.3, 53.6]40.8 [36.1, 45.6]45.4 [41.2, 49.8]
EA8.4 [4.9, 11.9]13.0 [9.2, 17.0]5.7 [2.4, 9.4]2.4 [0.2, 5.3]

At ρ = 5%, ARC-KV's lower CI bound is above every baseline's upper bound at all three lengths; at 16K with ρ = 10% it is statistically indistinguishable from full context.

  • How savings scale with reuse. Total time for N queries is the one-time build plus N times the per-query cost. Against OMP-AM, both serve from a compact cache of the same size, so ARC-KV is ahead by its build-time saving (1,154 s at 32K, 1,866 s at 64K) for every N. Against the full cache, ARC-KV takes 15–22% less time per query one at a time and is 4.2–5.3× faster with 100 concurrent queries (6.9–8.9× with 768), so its extra build time is repaid after about 820 / 870 queries served one at a time at 32K / 64K; it also stores about 20× fewer KV states at ρ = 5%.
  • The one-time training cost is amortized quickly. The indexer is trained once per model in 147 B200 GPU-hours; relative to OMP-AM, this is recovered after about 280 contexts at 64K.

We will clarify in the revision that the 25.73× figure refers to compaction time, and add these end-to-end results. We hope they address the reviewer's concerns, and we would be grateful if the reviewer would take them into account.

Thank you for the detailed rebuttal and extensive additional experiments. The controlled ablations, fair baseline comparisons, and long-context end-to-end evaluations have substantially addressed my concerns.

I appreciate the authors' thorough efforts, and the additional results further strengthen my positive assessment of the paper. I will maintain my rating of 3 (Weak Accept).

We sincerely thank the reviewer for the constructive comments and for the positive assessment of our work. The suggestions have substantially improved the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.