- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes ARC-KV, a reconstruction-based KV-cache compaction method for long-context LLM inference. Its key idea is to amortize the expensive per-context anchor search in Attention Matching: a lightweight value-aware indexer is trained once per base model to select real-key anchors in a single pass, while context-specific reconstruction is retained through attention-response-based key merging and closed-form fitting of attention-mass biases and compact values. The method is evaluated on QuALITY, RULER, and LongBench using Llama-3.1-8B-Instruct, with additional experiments on Qwen3-8B and Gemma-3-12B-IT, and compared against several cache eviction and reconstruction baselines across retention ratios. ARC-KV ranks first among reported compacted methods in 13 of 15 main settings; for example, at 10% retention on QuALITY it improves accuracy from 0.6409 to 0.6474 over Attention Matching while reducing compaction time from 959.8 s to 37.3 s. Ablation and profiling studies further support the value-aware selection, key merging, and nonuniform per-head allocation, although the reported efficiency measurements primarily concern cache compaction rather than end-to-end inference latency.
Critical strengths
ARC-KV makes a technically coherent contribution by separating anchor selection from cache reconstruction. Its reusable value-aware indexer removes the expensive iterative search in Attention Matching, while context-specific key merging and closed-form fitting preserve the flexibility needed for different contexts.
The empirical evaluation is broad and convincing. Across QuALITY, RULER, and LongBench, ARC-KV outperforms reported compacted baselines in most settings and remains competitive across additional model architectures. It also achieves substantial compaction-time reductions—for example, a 25.73× speedup at 10% retention on QuALITY while slightly improving accuracy—supported by ablations and runtime profiling.
Critical weaknesses
The efficiency claim is not yet demonstrated at the end-to-end inference level. Reported speedups measure cache-compaction time only and exclude full decoding latency; moreover, at long contexts, reference-query construction accounts for most of ARC-KV’s remaining runtime. Thus, the 25.73× compaction speedup may not translate into a comparable serving or latency improvement, especially when a prefix is reused only a small number of times.
The evaluation does not fully establish generality. Most results use Llama-3.1-8B-Instruct, while the additional model study tests only one retention ratio and two models. The method is also evaluated mainly for one-shot compression of static prefixes, with limited evidence for generated-state recompression, much longer contexts, or dynamic workloads. In LongBench, the default OMP-based Attention Matching baseline is replaced by AM-HA because of its cost, which further weakens the fairness of the strongest comparison.
Questions for authors
-
Can the authors report end-to-end serving results, including prefill, compaction, decoding, and repeated queries? Since reference-query construction dominates ARC-KV’s cost at long contexts, the practical break-even point for prefix reuse is important.
-
Why is AM-HA used instead of the default OMP-based Attention Matching on LongBench? Could the authors provide a direct comparison with OMP-based AM on a smaller matched subset, or report the computational budget and failure rate for both implementations?
-
Are the reported speedups measured with identical hardware, kernels, precision, batching, and reference-query construction costs for all methods? Please clarify whether baseline implementations were independently optimized or used as released.
-
How are the nominal retention ratios converted into actual memory usage? Please report the total footprint of compact keys, values, attention-mass biases, temporary tensors, and any metadata, since per-head allocation and chunking may make nominal ratios differ from actual compression.
-
Were the values of the key-merging coefficient, loss weights, per-head budgets, and other hyperparameters selected using a validation set that is disjoint from all reported test benchmarks? This is important because the accuracy improvements over AM are sometimes relatively small.
-
How statistically robust are the reported gains? Please provide confidence intervals or results over multiple seeds, particularly for the 0.23–3.63 point improvements on QuALITY and the 50-example-per-task LongBench evaluation.
-
Does the indexer training corpus overlap in documents, domains, or distributions with QuALITY, RULER, or LongBench? Even without QA labels, overlap could make it difficult to distinguish general cache-selection ability from benchmark-specific adaptation.
-
For Qwen3 and Gemma, were new indexers trained separately using the same data and compute budget as the Llama indexer, or was the Llama indexer transferred? Clarifying this would help assess whether ARC-KV is broadly reusable or requires substantial model-specific training.
6 comments
Response by authors (2/4)
W2. Generality.
- More budgets and models. We extended the cross-model study to five budgets (ρ = 0.01–0.20) on RULER-4K for Qwen3-8B and Gemma-3-12B-IT. ARC-KV is the best compacted method in 7 of 10 settings: every budget on Gemma, and ρ ≤ 0.02 on Qwen. On Qwen at ρ ≥ 0.05 the original KVzip leads, consistent with Table 2; with the same per-head allocation as ARC-KV, however, KVzip drops sharply (27.8 vs. 53.7 at ρ = 0.05) and ARC-KV leads in all 10 settings, indicating that the remaining gap on Qwen stems from KVzip's own budget allocation, which is orthogonal to ARC-KV's anchor selection and could be combined with it. ARC-KV outperforms AM, SnapKV and EA in every setting. We also found that the RULER results in Figure 2b and Table 2 used strict string matching; we now report the official metric, and will correct both accordingly in the revised version; on Llama RULER-4K at ρ = 5%, ARC-KV's lead over AM holds under the official metric (87.0 vs. 58.5, vs. 73.94 vs. 48.00 under strict matching).
- Longer contexts. On RULER at 16K / 32K / 64K (ρ = 5%), ARC-KV scores 73.4 / 70.6 / 65.3 vs. 54.3 / 43.3 / 47.7 for the best baseline (KVzip), with non-overlapping 95% CIs, even though the indexer was trained only on contexts of at most 16,884 tokens. With a larger budget (16K, ρ = 10%), ARC-KV reaches 93.6, statistically indistinguishable from full context (94.5) and ahead of OMP-AM (90.1) and KVzip (80.1).
- Generated-state recompression and dynamic workloads. These are outside the current scope, which targets one-shot compaction of static reusable prefixes (Section 3.5, Conclusion). Because the indexer selects anchors in a single pass without per-context search, it is a natural fit for repeated recompression, and we plan to extend ARC-KV to chain-of-thought traces and multi-turn agent histories in future work.
Cross-model results on RULER-4K (%, official metric; full context 99.5 for Qwen3-8B and 99.3 for Gemma-3-12B-IT) against KVzip, the strongest baseline in every setting; AM, SnapKV and EA are below ARC-KV everywhere:
| ρ | Qwen3-8B: KVzip | Qwen3-8B: KVzip (per-head) | Qwen3-8B: ARC-KV | Gemma-3-12B-IT: KVzip | Gemma-3-12B-IT: ARC-KV |
|---|---|---|---|---|---|
| 0.01 | 14.9 | 17.5 | 22.7 | 15.6 | 15.9 |
| 0.02 | 18.6 | 19.8 | 30.2 | 18.7 | 29.5 |
| 0.05 | 53.7 | 27.8 | 46.2 | 29.1 | 46.4 |
| 0.10 | 91.8 | 56.6 | 84.8 | 53.5 | 67.3 |
| 0.20 | 99.0 | 94.0 | 95.6 | 91.1 | 93.6 |
Long-context RULER (official metric, 95% CI):
| Method | 16K, ρ = 5% | 16K, ρ = 10% | 32K, ρ = 5% | 64K, ρ = 5% |
|---|---|---|---|---|
| Full context | 94.5 | 94.5 | 89.4 | 88.2 |
| ARC-KV | 73.4 [68.0, 78.2] | 93.6 [89.8, 96.7] | 70.6 [65.3, 75.3] | 65.3 [60.0, 70.3] |
| OMP-AM | 41.6 [36.1, 47.2] | 90.1 [85.4, 94.3] | 31.8 [27.9, 36.2] | 36.2 [30.9, 41.7] |
| KVzip | 54.3 [48.1, 60.4] | 80.1 [76.0, 84.2] | 43.3 [37.2, 49.3] | 47.7 [42.4, 53.1] |
| SnapKV | 39.4 [34.5, 44.0] | 49.1 [44.3, 53.6] | 40.8 [36.1, 45.6] | 45.4 [41.2, 49.8] |
Q2. AM-HA instead of OMP-based AM on LongBench.
We ran OMP-AM directly on a matched subset of 50 LongBench instances (10 each from NarrativeQA, HotpotQA, 2WikiMQA, GovReport and TriviaQA, spanning four task categories), with all methods on identical instances (LongBench score ×100):
| Method | ρ = 0.02 | ρ = 0.05 | ρ = 0.10 |
|---|---|---|---|
| ARC-KV | 48.1 | 47.5 | 53.2 |
| OMP-AM | 44.8 | 47.4 | 49.7 |
| AM-HA | 39.7 | 47.7 | 49.1 |
ARC-KV matches or exceeds OMP-AM at every budget. AM-HA tracks OMP-AM within 0.6 points at ρ = 0.05 and 0.10, so the substitution did not inflate ARC-KV's margin there; at ρ = 0.02 AM-HA is weaker, but ARC-KV still exceeds OMP-AM by 3.3 points on this subset. The substitution was driven by cost: OMP-AM is 3.6× / 11.5× / 33× slower per instance than AM-HA, and running it on all 1,050 instances at ρ = 0.10 would take about 520 B200 GPU-hours. Neither OMP-AM nor AM-HA failed or timed out on any instance.
Response by authors (3/4)
Q3. Identical measurement conditions and baseline implementations.
All methods are timed on the same NVIDIA B200 GPU with the same precision and batching, and every compaction time includes reference-query construction. Baselines use released code with default settings: AM (OMP-fast, k = 4, refit every two steps), AM-HA, SnapKV and KVzip from the official Attention Matching implementation, and Expected Attention from the kvpress repository. We did not further optimize any method beyond its released configuration. For serving, the full cache runs on vLLM 0.33.0 with FlashInfer FlashAttention-2 0.7.3, a fast and well-optimized attention backend; the compact caches of ARC-KV and AM are served with the same FlashAttention-2 kernel, modified only to add the attention-mass bias β.
Q4. From nominal retention to actual memory.
- Stored footprint: Ck, Cv and β together take 1.00 / 5.02 / 18.51% of the full KV cache at 64K for nominal 1 / 5 / 20%; β adds only about 0.4% on top of Ck and Cv. At equal budget every method stores essentially the same number of bytes. Per-head metadata (one budget length per KV head) is negligible.
- Realized vs. nominal: rounding (Eq. 16) and chunking keep the number of stored states 0.03–0.60% below nominal at ρ ≤ 5% (16K–64K), and 7.8% below at ρ = 20% because 11 heads are capped at their full length. It never exceeds the nominal budget.
- Temporaries: the largest per-group temporary (indexer scoring or the Cv solve) is at most 10.3 GB and does not grow with context length; peak compaction memory grows only through the reference-query state. Compacting a 64K context fits on a single 80 GB GPU (66.8 GB in total). At 16K, ARC-KV's peak (38.80 GB) is within 0.58 GB of AM's (38.22 GB), the difference being the indexer weights.
| Context | Stored Ck + Cv + β, % of full KV (ρ = 1% / 5% / 20%) | Stored states vs. nominal (ρ = 1% / 5% / 20%) | Total GPU memory to compact, ρ = 5% [GB] |
|---|---|---|---|
| 16K | 1.00 / 5.02 / 18.50 | −0.60% / −0.07% / −7.84% | 38.8 |
| 32K | 1.00 / 5.02 / 18.51 | −0.04% / −0.07% / −7.83% | 50.8 |
| 64K | 1.00 / 5.02 / 18.51 | −0.11% / −0.03% / −7.82% | 66.8 |
Q5. Hyperparameter selection.
The key-merging coefficient λm and the loss weights were selected on our validation set of 60 held-out development articles (Appendix C.2), disjoint from all test benchmarks. The per-head budgets were not tuned by us: we reuse unchanged the weights released with Attention Matching, and every method, including AM itself, uses the same weights. We also measured accuracy for λm ∈ {0.25, 0.5, 0.75}: the three values lie within 1.4 points on QuALITY, 2.6 points on the LongBench subset, and 1.7 / 0.5 points on RULER-4K at ρ = 0.05 / 0.10; only RULER-4K at ρ = 0.02 favors a small λm, consistent with the trend in Table 5. The default is not the best test-set value on QuALITY at any budget or on RULER-4K at ρ = 0.10, and every λm still outperforms AM (OMP) on all three benchmarks at every budget. The results therefore do not rely on test-set tuning.
| Benchmark | ρ | λm = 0.25 | λm = 0.5 | λm = 0.75 | AM (OMP) |
|---|---|---|---|---|---|
| QuALITY | 0.02 | 62.5* | 62.9 | 61.5 | 59.1 |
| QuALITY | 0.05 | 64.4* | 64.5 | 65.1 | 60.7 |
| QuALITY | 0.10 | 64.7* | 65.6 | 65.2 | 64.1 |
| RULER-4K | 0.02 | 61.4* | 57.7 | 52.2 | 26.1 |
| RULER-4K | 0.05 | 87.0* | 86.1 | 85.3 | 58.5 |
| RULER-4K | 0.10 | 98.1* | 98.5 | 98.6 | 93.7 |
| LongBench subset | 0.02 | 47.2 | 46.5 | 48.1* | 44.8 |
| LongBench subset | 0.05 | 49.0 | 48.6 | 47.5* | 47.4 |
| LongBench subset | 0.10 | 52.7 | 50.6 | 53.2* | 49.7 |
* marks the default used in the paper. QuALITY values are accuracy (%) with the same single seed for every λm; RULER-4K values are accuracy (%) under the official metric; LongBench values are mean task scores ×100.
Response by authors (1/4)
We sincerely thank the reviewer for the detailed and constructive review, and for finding the decomposition of anchor selection from reconstruction technically coherent and the evaluation broad and convincing. We ran new experiments and addressed each question below; all results and clarifications will be added to the revision.
W1 / Q1. End-to-end serving and the break-even point for prefix reuse.
This is a well-founded concern about the real end-to-end serving environment. We integrated the compaction methods into the production serving framework vLLM and measured full serving-pipeline performance on a single NVIDIA B200 GPU. At 32K and 64K context lengths, we record the wall-clock time of three key stages: cache build (context prefill, repeat prefill, selection, merging, fitting), per-query prefill, and 128-token decoding.
We will state explicitly that the 25.73× figure refers to compaction time and add these end-to-end results in the revised version.