acceptodds
Under review as a conference paper at ICLR 2027

RAPID: Extreme KV Cache Compression At Near-Full Quality from the Context Alone

Abstract

The size of KV caches is a dominant bottleneck in long-context model serving. Modern KV compression algorithms fall short on at least one of the following objectives. 1. To maintain uncompressed generation quality. 2. To compress fast, without heavy training, data generation, or model use (beyond context itself). 3. To achieve best possible compression: SOTA as of mid-2026 is over 10× rate as compared to full KV cache storage at half-precision floating points. We introduce RAPID algorithm (Rank And Precision allocation with Independent per-layer Distillation) which our evaluations show to be the closest to satisfying all three objectives. RAPID combines layer-wise latent representations, cross layer budget allocation, per token dimension reduction, per dimension quantization, and intra-layer optimization using prefill query vectors only. RAPID doesn't need training data and access to the model beyond each layer's attention projections. A variant of our algorithm, RAPID++, is always within 2.5 points of the full cache quality at 20× compression, on every evaluated benchmark and model, including Qwen3-4B, a 117B-parameter mixture-of-experts GPT-OSS-120B, and an MLA model whose cache is already compressed.

Primary area:
optimization
Keywords:
KV cache compression, long-context inference, efficient LLM serving, rate–distortion optimization, low-rank approximation, quantization, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NVwCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work concentrates on extreme KV cache compression (20x level) and claims that the proposed method can achieve near-full quality. RAPID jointly adopts strategies such as layer-wise latent representation and cross layer budget allocation, which is free from massive training. Extensive experiments with analyses have verified the effectiveness of the proposed model.

Critical strengths
  1. The authors have conducted extensive evaluations and carefully analyzed the results, making the conclusion more solid.
  2. Some challenging settings (e.g., with MLA) have been considered.
Critical weaknesses
  1. Some essential hyper-parameters should be clearly introduced with parameter analyses, for example, the prefix floor and the probe rank.
Questions for authors
  1. Attention matching is supposed to be compared on other backbones, e.g., GPT-OSS.
  2. The cost of RAPID++ is rather too high.
2 comments
AuthorsOpenReview ↗

Response to Reviewer NVwC

Thanks for the detailed comments that materially improved the paper!

W1: the prefix floor and the probe rank. Both are gone in the revision. The revision replaces probe ranks by the sensitivity of each latent entry under a second-order model of the objective, computed with one backward pass over the reference queries (Section 3 before 3.1, Eqn. 2). The floor of 8 coordinates per token is removed as well, so the allocation now sets every prefix and may evict a token (Appendix B.2). One table lists every remaining setting, its value, and where it is varied (Appendix B.4, Table 2).

Q1: Attention Matching on GPT-OSS-120B. Attention Matching gives each KV head its own budget, computed once per model from per-head sensitivity curves on calibration contexts (Zweiger et al., 2026). Its authors provide these budgets for the models of their paper, which include Qwen3-4B and Llama-3.1-8B, but not for GPT-OSS-120B, and calibrating them takes substantial GPU time. We therefore compare with it on these two models, where the comparison covers both of its versions (repeat-context and self-study queries), RULER as well as LongHealth, and its own answer protocol (Table 1 and Appendices D.1, D.5 and E). In Table 1, RAPID++ is above it at every ratio on both models, by 12 to 16 points on LongHealth at 50× and by 12 to 33 points on RULER at 20×.

On GPT-OSS-120B, the baselines are KVzip, Ada(KVzip), and Ada(KVzip) + KIVI-2. At 20×, RAPID++ scores 75.5 on LongHealth and 68.1 on RULER, against 66.9 and 46.5 for the best baseline on each and 76.6 and 70.4 for the uncompressed cache (Table 1).

Q2: the cost of RAPID++. We take this point and now measure the cost . For a 64K-token context at 20× on one H200, RAPID++ takes 153s on GPT-OSS-120B, 313 s on GLM-4.7-Flash, 347 s on Qwen3-4B, and 356 s on Llama-3.1-8B. Beyond the prefill and the read that captures the queries, compression does not need the model's weights (only each layer's attention projections), so the time does not grow with the model: the 117B-parameter GPT-OSS-120B is the fastest of the four. For comparison, Attention Matching in its authors’ code takes 300s and 255s in the same setting on Qwen3-4B and Llama-3.1-8B. Training a Cartridge often takes several GPU-hours per context.

For reference, the time of the fitted baselines in the same setting, in seconds (Appendix K, Table 29):

MethodQwen3-4BLlama-3.1-8B
RAPID++ without refinement255277
RAPID++347356
Attention Matching300255
Attention Matching, self-study474396
KV-Surgeon318300
KV-Surgeon, self-study634575

Following reviewers’ remarks about this cost, we slightly revised the calculation of the sensitivity (that improved both performance and accuracy). The revised sensitivity calculation also cut the refinement from 50 steps to 8. At 10× and 20× the refinement can even be skipped: without it, RAPID++ takes 109 to 277 s and stays within 1.4 points of the uncompressed cache on LongHealth and 2.2 points on RULER, on every model (Table 1). The cost is paid once per context, and the compressed cache then serves every query on it.

Thanks for the rebuttal, which addressed most of my concerns. I will keep my positive score.

Reviewer_4oF4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RAPID, a training-free KV-cache compression framework targeting extreme compression ratios (20×–100×) while preserving generation quality. RAPID jointly optimizes low-rank representations, global budget allocation across layers, tokens, dimensions, and precision levels, and per-layer refinement using context-derived queries. The central contribution is a unified rate-distortion formulation for KV-cache compression, implemented through a three-stage pipeline combining variable-rank representation, mixed-precision allocation, and lightweight per-layer distillation. The method does not require model fine-tuning or generated training data and operates without access to the full model weights beyond the attention projections. The authors evaluate RAPID and RAPID++ across diverse model architectures, including Qwen3-4B, Llama-3.1-8B, GLM-4.7-Flash, Gemma-4-31B, and GPT-OSS-120B, using LongHealth, MTOB, and QASPER. Overall, the paper presents a technically strong and practically relevant approach to extreme KV-cache compression, supported by extensive experiments across models, compression ratios, and evaluation settings.

Critical strengths
  1. A principled and effective compression framework RAPID combines several complementary techniques, including low-rank factorization, token-wise rank allocation, mixed-precision quantization, layer-wise budget allocation, and per-layer optimization, within a unified rate-distortion framework. Although many individual components have appeared in prior work, their integration into a unified budget allocation mechanism is a meaningful contribution. In particular, Equation (1) directly optimizes normalized attention-output reconstruction error rather than relying solely on surrogate objectives such as token importance, singular-value energy, or attention sparsity. This provides a principled basis for allocating compression budgets according to their impact on attention outputs and distinguishes RAPID from approaches that optimize individual compression dimensions independently.
  2. Extensive evaluation across diverse architectures and settings The experimental evaluation is unusually broad, covering five model families, multiple attention architectures (including GQA, MLA, and hybrid local/global attention), and a range of benchmarks and compression ratios. This diversity strengthens confidence in the method's practical applicability and provides more convincing evidence of robustness than evaluation on a single architecture or benchmark would.
  3. Strong results at extreme compression ratios RAPID++ achieves substantial improvements over the reported baselines on LongHealth at 50× compression: These results suggest that RAPID++ offers substantial quality advantages at aggressive compression ratios. The result on GLM-4.7-Flash is particularly noteworthy because MLA already uses a compressed latent KV representation, yet RAPID++ still achieves a considerable improvement over the reported baseline.
  4. Evaluation beyond standard retrieval tasks Section 4.2 is a particularly valuable part of the paper. Beyond standard benchmark accuracy, the authors evaluate more challenging scenarios involving near-miss questions, cross-patient reasoning, timeline reasoning, and unanswerable questions. These tests expose generalization failures that conventional retrieval-style evaluation may overlook. For example, at 20× compression on Qwen3-4B, RAPID++ achieves an extension-average accuracy of 46.0, compared with 35.3 for Attention Matching, 10.2 for Ada(KVzip), and 4.8 for KVzip. This evaluation strengthens the evidence that RAPID++ preserves useful contextual information beyond the specific queries used during compression.
  5. An important insight into compression objectives A particularly interesting finding is that attention-output reconstruction error appears to be more informative for downstream performance than raw low-rank reconstruction error. Figure 9 shows a weak relationship between SVD reconstruction quality and downstream attention-output quality. This observation offers a plausible explanation for why preserving latent reconstruction quality alone may be insufficient for effective KV-cache compression. It also provides a useful direction for future research: compression objectives should be evaluated in terms of their impact on attention computation, rather than latent reconstruction error alone.
Critical weaknesses
  1. Limited novelty at the component level Many of RAPID's individual components, including SVD-based compression, per-token rank allocation, mixed-precision quantization, Lagrangian budget allocation, and attention-matching objectives, have precedents in prior work. The main novelty therefore lies in their integration and joint optimization rather than in a fundamentally new compression primitive. While the resulting framework is effective, the paper's conceptual novelty is less pronounced than its empirical gains.
  2. Compression-time cost is insufficiently characterized The paper suggests that compression requires approximately one or two prefills plus refinement. However, the procedure also involves 50 gradient steps per layer, five probe evaluations per layer, fitting multiple Lloyd–Max quantizers, and several optimization passes. The actual wall-clock cost and peak memory usage are not sufficiently characterized, particularly for large models such as GPT-OSS-120B. Parallelization efficiency and the practical overhead of the compression pipeline are also important considerations for deployment. Reporting compression latency, peak memory consumption, and scaling behavior would help establish whether the method's quality improvements translate into practical end-to-end benefits.
  3. Limited comparison with amortized and learned compressors The paper provides extensive comparisons with eviction-based and attention-matching methods, but empirical comparisons with recent amortized or learned compression approaches, such as Still, RestoreKV, LCLM, and GradMem, appear limited. These methods occupy an important part of the KV-cache compression design space, particularly when compression costs can be amortized across repeated deployments. Stronger comparisons under matched compression ratios, quality targets, and computational budgets would clarify RAPID's relative advantages and limitations.
  4. The mechanism behind RAPID++'s performance remains unclear One intriguing result is that RAPID++ initialization enables subsequent self-study fine-tuning to outperform the full-context baseline. For example, RAPID++ reportedly achieves 60.0% after only 16 self-study steps, compared with 53.0% for the full-context baseline. This is an interesting observation, but the underlying mechanism is not sufficiently explained. It remains unclear whether the improvement arises from regularization, denoising, information concentration, or another effect. Further analysis would strengthen the scientific contribution and help distinguish the benefits of KV-cache compression itself from those of the subsequent self-study procedure.
Questions for authors

Q1. Validity of the attention-output reconstruction objective The paper argues that attention-output reconstruction error is a more appropriate optimization target than raw low-rank reconstruction error. Can the authors quantify the correlation between this metric and downstream task performance across models, compression ratios, and benchmarks? Such an analysis would help validate the central motivation for the proposed objective. Q2. Practical compression-time overhead What are the actual wall-clock compression latency and peak memory requirements, relative to a single prefill, for Qwen3-4B, Llama-3.1-8B, and GPT-OSS-120B? Quantitative results would help assess the practicality of RAPID in real-world serving scenarios. Q3. Trade-off against amortized compression methods Methods such as Still, RestoreKV, and LCLM amortize compression costs through learned or pretrained components. Under what context-reuse frequency or deployment conditions does RAPID become less cost-effective than these approaches? A comparison that accounts for both one-time compression overhead and repeated inference costs would clarify the method's practical operating regime. Q4. Generalization to unseen queries RAPID++ generalizes strongly to challenging questions, whereas Attention Matching exhibits substantially larger performance degradation. Can the authors explain why context-derived queries are sufficient to preserve information useful for these unseen queries? In particular, what properties of the proposed objective or optimization procedure account for this difference?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 4oF4 (1/3)

Thanks for the detailed comments that materially improved the paper!

W1: novelty.

We agree that each primitive of RAPID has a precedent (and tried our hardest to discuss all relevant prior art), and we do not claim a new primitive. The contribution is indeed that the integration is surprisingly effective (even against heavy-training methods like Cartridge), and that all the components of the algorithm arise from considering one objective, the attention-output error on the context's own queries (Eqn. 1). In contrast, the closest methods take the basis and the allocation from other signals. We make this explicit in the revision (Section 3 before 3.1, and new paragraph "How RAPID differs" in Section 2).

W2 and Q2: compression cost.

We agree that the submission did not measure the cost. We now do so in the revision: 153 s for a 64K-token context at 20× on GPT-OSS-120B (table below). Furthermore, in response to your cost concerns, we refined how the allocation weighs each latent entry, computing the sensitivity directly from a second-order model of the same objective (Eqn. 1). (The submission used a plain SVD, a product of a token weight, its largest attention, and a coordinate weight measured by five truncation probes per layer.) This improved both quality and performance: e.g., refinement now needs 8 steps instead of 50. The rest of the algorithm remains mostly unchanged: the objective, the latent representation, the allocation solver, the refinement procedure, and the reference queries. (A few small parts changed: a token may now keep its exact row, the values projection gets a second refit, the prefix floor is gone, and the menu of bit widths is shorter; see Appendix B.)

Here are results for a 64K-token context at 20× on one H200. We also added Appendix K (Tables 29 to 32) reporting wall-clock time from 16K to 128K tokens on all four models and against the main baselines, time per stage, and memory.

ModelUncompressed cachePrefillRAPID++ without refinementRAPID++RAPID++ in prefillsPeak GPU memory, with weights
GPT-OSS-120B2.4 GB4.4 s109 s153 s3574 GiB
GLM-4.7-Flash3.5 GB23.3 s175 s313 s1373 GiB
Qwen3-4B9.7 GB12.8 s255 s347 s2756 GiB
Llama-3.1-8B8.6 GB11.5 s277 s356 s3158 GiB

A few notes:

  • From 64K to 128K tokens the time grows by 2.7 to 3.3 times.
  • The cost follows the size of the cache, not of the model.
  • For reference, Attention Matching fast variant (highest-attention keys) takes 300s and 255s on Qwen3-4B and Llama-3.1-8B.
  • At 10× and 20× the cache without refinement is within 1.8 points of the refined one on every model and benchmark, so the 8 refinement steps matter mainly at 50×.
  • The memory that compression adds on top of the weights comes from storing intermediate calculations, and it can be traded for a slightly longer run when needed; please see Appendix K for more details.
AuthorsOpenReview ↗

Response to Reviewer 4oF4 (2/3)

W3, Q3: comparison to learned/amortized compressors.

We agree that this comparison was missing. Our revision adds RestoreKV and LCLM as baselines, each in its authors' code (Table 1; Appendix E, Table 8; Appendix D.5, Table 7). Accuracy on LongHealth at 10× / 20× / 50× and RULER score at 20× is:

MethodQwen3-4B, LongHealthQwen3-4B, RULERLlama-3.1-8B, LongHealthLlama-3.1-8B, RULER
Uncompressed cache77.588.374.879.9
RestoreKV72.9 / 69.2 / 57.254.869.9 / 62.5 / 51.547.2
LCLM, at 8× / 16×73.8 / 66.253.9 at 16×no checkpointno checkpoint
RAPID++77.6 / 77.5 / 77.287.774.2 / 73.9 / 74.480.1

LCLM has released checkpoints for Qwen3-4B at 8× and 16× only. We did not manage to add the other two methods you named, Still and GradMem. The reasons are that: Still needs a compactor trained for each base model, and neither a compactor nor its training code are released, and GradMem trains the model's own weights so that test-time writing works, and is demonstrated with models of up to 160M parameters on contexts of a few hundred tokens, hence it does not apply to our setting.

On Q3: once trained, a learned compressor is cheaper per context than RAPID++ (up to about three prefills in our runs, against 13 to 35), so RAPID++ is the less cost-effective choice when a context is read only a few times and the accuracy of the learned compressor suffices. It is the better choice when a context is reused many times or compressed further: RAPID++ at 50× (77.2 and 74.4 on LongHealth) is above RestoreKV at 10× (72.9 and 69.9).

W4: the self-study result. This is of course a great question, and you point at the likely cause: the gain comes from the self-study signal, not from compression. Cartridge paper was already implying this when referencing the prefix tuning paper (Li, Liang 2021).

We add experiments to further test this in Appendix I. First, even the uncompressed cache improves when refined with self-study (Table 23, Figure 10). Accuracy on a 110K-token context with Llama-3.2-3B, at 53×:

Starting cacheNo self-study16 steps32 steps
Uncompressed cache (11.6 GiB)51.860.060.5
RAPID++ (224 MiB)54.860.062.5

(The uncompressed cache scores 51.8 in this run and 53.0 in the earlier one you quote, on the same 200 questions; the two are within one standard error.)

Second, the uncompressed model on shorter contexts (Table 24): it answers the same questions at 69.0 percent with one patient's record in context, 67.8 with two, 58.2 with five, and 52.0 with all ten. So the 52 percent is a by-product of a small model reading a long context, and self-study conversations generated from short parts of the context help it. We revised the claim in the introduction accordingly: RAPID++ followed by self-study is level with the uncompressed cache under the same training, at 1/53 of its size. We also see that the better the initialization, the better the resulting cache: after 64 steps RAPID++ goes from 54.8 to 62.2 percent, while caches of the same size that start near 34 percent end lower: Ada(KVzip) at 50.2 and a Cartridge at 36.2 (46.3 after a full run of 2,048 steps).

AuthorsOpenReview ↗

Response to Reviewer 4oF4 (3/3)

Q1, validity of the objective . We added the analysis you suggested: the attention-output error on the question tokens is rank-correlated with the loss of accuracy with a Spearman coefficient of 0.79, and 0.76 to 0.82 within each model, over the 178 caches of the four models, three ratios (Appendix E.2, Figure 5). The reconstruction error of the keys and values, on the other hand, does not track the attention-output error: across the layers of a cache, the ranking of those error metrics shows negative correlation across layers in 10 of the 12 combinations of model and compression ratio (Appendix L, Table 34, Figure 11).

The increased performance of our revised algorithm further validates this. Its SVD is taken in a metric derived from the objective, so the coordinates are ordered by how much they reduce the attention-output error, not by how much of the keys and values they reconstruct. Every prefix therefore keeps the directions the layer's output depends on most, and each stored coordinate buys more accuracy. For instance, on Llama-3.2-3B with a 110K-token LongHealth context at 20×, the resulting cache (before refinement) has a higher reconstruction error than a plain SVD of the same size (0.49 and 0.81 for keys and values, against 0.27 and 0.31), an attention-output error 5.8 times lower, and an accuracy of 53.0 percent against 46.5 (the uncompressed cache scores 51.8 there; Appendix L, Table 33).

Q4, generalization to unseen queries.

We do not have a complete explanation, and report our intuition and findings so far. First, our intuition is that the repeat read of the context gives every token a query, whereas a set of (self-study) questions emphasizes the parts of the context it asks about. In other words, a set of questions can over-specialize the cache to its own query distribution. In contrast, the context-derived queries cover every token of the context, so the algorithm does not over-specialize to a subset of it.

We test our intuition by fitting a cache to the test questions themselves, with the whole model's KL as the objective, and scoring it on questions it has not seen (new Appendix M, Tables 35 to 37):

Cache fitted to test questionsRAPID++
KL from the uncompressed model (lower is better, two folds split)0.124, 0.1020.022, 0.021
Answers identical to the uncompressed model's (%)83.0, 80.089.8, 89.8
Bits per token on the passages the questions cite, relative to the average token1.31.00

The question-fitted cache concentrates its bits on the tokens its questions read and does not transfer; RAPID++ spreads them evenly and hence transfers.

The queries alone do not explain the gap to Attention Matching, which is also fitted to repeat-context queries. The difference likely arises from the allocation: RAPID sets one budget across layers, tokens, and coordinates from the same objective. Removing any part of this costs quality (Appendix F, Table 18). At 50× on GPT-OSS-120B, an equal budget for every layer, a single prefix length per layer, and a single bit width raise the attention-output error after the 8 steps by 18, 26, and 12 percent, and accuracy on the extended questions falls from 40.3 to 36.0, 32.3, and 36.3. Adding our refinement to Attention Matching's cache does not close the gap either: its own 50× cache rises from 67.5 to 70.9 percent on Qwen3-4B, against 77.2 for RAPID++ (Appendix H, Figure 8).

Thank you for the detailed responses and additional validation. These have addressed my questions and clarified the concerns raised in my review. My original assessment was already positive, and I will maintain my rating of 3 (Weak acceptance). I have no further questions.

Reviewer_ukjVOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RAPID, a KV-cache compression framework that combines layer-wise low-rank factorization, per-token rank selection, mixed-precision quantization, and global memory-budget allocation. Unlike conventional reconstruction-based compression, RAPID further refines the compressed representations by minimizing attention-output errors using queries extracted from the original context. Its extension, RAPID++, incorporates an additional set of reference queries obtained by replaying the context. The authors report strong quality preservation at aggressive compression ratios. At 20x KV-cache compression, RAPID++ achieves near-lossless accuracy across several model architectures, including GQA, MLA, and MoE-based models, without external training data or end-to-end model fine-tuning.

Critical strengths

TBD

Critical weaknesses
  • Novelty is insignificant. The core ingredients [low-rank KV compression, adaptive rank allocation, mixed-precision quantization, Lagrangian bit allocation, and attention-output matching] are individually established already in previous work, such as Palu, MixedDimKV, and Attention Matching. RAPID's contribution is the integration with per-token variable-rank latents and independent layer-wise refinement. This is a meaningful extension, but the paper would benefit from more direct comparisons demonstrating its advantage over recent methods addressing similar rank-allocation and attention-matching problems.

  • Inference efficiency is not discussed. Although RAPID claims substantial reductions in KV-cache memory and bandwidth, the paper does not provide a complete decoding implementation. The authors explicitly acknowledge that they have not implemented the full decoding kernel. Consequently, there are no measurements of actual decoding latency, throughput, or maximum serving batch size. Per-token variable ranks, mixed-precision coordinates, and per-coordinate non-uniform codebook lookups incur irregular memory accesses and computational overhead - achieving 20x storage compression does not necessarily translate into meaningful inference speedup.

  • The compression overhead is not sufficiently evaluated. One of the RAPID's main objectives is efficient compression without expensive training. However, RAPID performs SVD, allocation, and 50 gradient-based refinement steps per layer over the context queries; RAPID++ additionally requires replaying the context through the model. Although each layer can be optimized independently, these operations are not necessarily inexpensive, particularly for 60K-100K-token contexts. This paper provides no convincing wall-clock measurements evaluating the compression overhead due to all these additional operations/steps.

Questions for authors

See "Critical Weaknesses" for my concerns and questions.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ukjV (1/2)

Thanks for the detailed comments that materially improved the paper!

Novelty and evaluations on more direct comparisons.

We agree that each primitive of RAPID has a precedent (and tried our hardest to discuss all relevant prior art), and we do not claim a new primitive. The contribution is indeed that the integration is surprisingly effective (even against heavy-training methods like Cartridge), and that all the components of the algorithm arise from considering one objective, the attention-output error on the context's own queries (Eqn. 1). In contrast, the closest methods take the basis and the allocation from other signals. We make this explicit in the revision (Section 3 before 3.1, and new paragraph "How RAPID differs" in Section 2).

As you point out, the components of RAPID have precedents, so we expanded the evaluation to the methods you name and to related ones. Palu, MixedDimKV, and KV-Surgeon are new baselines, and following Reviewer 4oF4 we added the learned compressors RestoreKV and LCLM.

At 20×, on the two models these methods run on, RAPID++ is the only one that stays close to the uncompressed cache on both benchmarks; the others are close on LongHealth at best, and at least 11.9 points below on RULER (6.8 when considering all evaluated methods):

Method, at 20×Qwen3-4B, LongHealthQwen3-4B, RULERLlama-3.1-8B, LongHealthLlama-3.1-8B, RULER
Uncompressed cache77.588.374.879.9
No context34.9n/a39.2n/a
RAPID++77.587.773.980.1
Attention Matching75.154.672.868.0
Attention Matching, self-study75.650.269.962.2
KV-Surgeon75.444.071.658.5
KV-Surgeon, self-study77.655.270.559.8
RestoreKV69.254.862.547.2
MixedDimKV32.811.341.517.4
LCLM (16×)66.253.9no checkpointno checkpoint
Palu (19.3×)not runnot run19.80.0

Remarks: MixedDimKV scores tokens by the attention of the last prompt tokens. Here those are the end of the context, because the cache is built before the question. Palu scores 67.0 at 10.2x, the range its paper evaluates, and collapses beyond it.

All results are in Section 4 (Table 1) and Appendix E (Tables 8 and 9). Notably, the result is more than the sum of its parts: adding refinement or quantization to the closest baselines does not bring them to RAPID's quality. On Qwen3-4B at 50× on LongHealth, our gradient refinement raises Attention Matching's own 50× cache (Figure 8) from 67.5 to 70.9 percent, and quantizing its 10× cache to 52.7× with KIVI-2 gives 68.0. Even a baseline with all three steps, eviction with Ada(KVzip), quantization with KIVI-2, and our refinement, reaches 74.2, against 77.2 for RAPID++ (Appendix H).

AuthorsOpenReview ↗

Response to Reviewer ukjV (2/2)

Inference efficiency.

We agree that a 20× smaller cache does not by itself give faster decoding: the straight-forward kernel implementation is slower. We therefore also added a decode kernel for a kernel-aligned version of RAPID, where the optimization problem for the bit allocation is further constrained: it restricts the stored format precisely with the three overheads you mention (codebook, mixed precision, and per-token rank). These constraints affect the accuracy only by ~1 point. See exact details in new Section 5 and Appendix N.

While the new kernel is (slightly) faster, we highlight the benefit when serving multiple contexts on a GPU. Namely, consider an H100 serving a number C of contexts of 51K to 59K (independent of each other) from LongHealth. Since under RAPID compression many more contexts fit, we are able to serve more tokens/second (per GPU). Below are comparison numbers. We serve GLM-4.7-Flash (MLA) and GPT-OSS-120B (mixture of experts) at 20×. The uncompressed cache is served by FlashMLA (GLM-4.7-Flash) and FlashInfer (GPT-OSS-120B). (See further details in Figure 3, and Tables 38 to 41 in Appendix N).

ModelCacheContexts CTokens per secondAccuracy (%)
GLM-4.7-FlashUncompressed156.566.0
GLM-4.7-FlashKernel-aligned RAPID++159.064.9
GLM-4.7-FlashUncompressed7 (largest that fits)267.866.0
GLM-4.7-FlashKernel-aligned RAPID++641,652 (6.2×)64.9
GPT-OSS-120BUncompressed1126.966.3
GPT-OSS-120BKernel-aligned RAPID++1127.464.9
GPT-OSS-120BUncompressed6 (largest that fits)429.366.3
GPT-OSS-120BKernel-aligned RAPID++642,322 (5.4×)64.9

The uncompressed cache runs out of memory beyond 7 contexts on GLM-4.7-Flash and 6 on GPT-OSS-120B, where it reached 268 and 429 tokens per second. The kernel-aligned cache at 64 contexts generates 6.2× and 5.4× more. Accuracy is averaged over the original and the extended LongHealth (800 questions).

Compression cost.

We agree that the submission did not measure the cost. We now do so in the revision: 153 s for a 64K-token context at 20× on GPT-OSS-120B (table below). Furthermore, in response to your cost concerns, we refined how the allocation weighs each latent entry, computing the sensitivity directly from a second-order model of the same objective (Eqn. 1). (The submission used a plain SVD, a product of a token weight, its largest attention, and a coordinate weight measured by five truncation probes per layer.) This improved both quality and performance: e.g., refinement now needs 8 steps instead of 50. The rest of the algorithm remains mostly unchanged: the objective, the latent representation, the allocation solver, the refinement procedure, and the reference queries. (A few small parts changed: a token may now keep its exact row, the values projection gets a second refit, the prefix floor is gone, and the menu of bit widths is shorter; see Appendix B.)

Here are results for a 64K-token context at 20× on one H200. We also added Appendix K (Tables 29 to 32) reporting wall-clock time from 16K to 128K tokens on all four models and against the main baselines, time per stage, and memory.

ModelUncompressed cachePrefillRAPID++ without refinementRAPID++RAPID++ in prefillsPeak GPU memory, with weights
GPT-OSS-120B2.4 GB4.4 s109 s153 s3574 GiB
GLM-4.7-Flash3.5 GB23.3 s175 s313 s1373 GiB
Qwen3-4B9.7 GB12.8 s255 s347 s2756 GiB
Llama-3.1-8B8.6 GB11.5 s277 s356 s3158 GiB

A few notes:

  • From 64K to 128K tokens the time grows by 2.7 to 3.3 times.
  • The cost follows the size of the cache, not of the model.
  • For reference, Attention Matching fast variant (highest-attention keys) takes 300s and 255s on Qwen3-4B and Llama-3.1-8B.
  • At 10× and 20× the cache without refinement is within 1.8 points of the refined one on every model and benchmark, so the 8 refinement steps matter mainly at 50×.
  • The memory that compression adds on top of the weights comes from storing intermediate calculations, and it can be traded for a slightly longer run when needed; please see Appendix K for more details.
Reviewer_ukjVOpenReview ↗

Increase ,y score from 1 to 2

The rebuttal is overall good. The main concern remaining is the technical novelty/contribution of this work integrating multiple existing approaches - I will increase my score from 1 to 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response and summary of changes

We thank the reviewers and the AC for engaging with our paper, and thoughtful comments that improved the paper. Below we highlight some important points, as well as list of changes in the pdf reflecting the updates to the paper.

  1. Novelty: We agree with the reviewers that each primitive of RAPID has a precedent. As reviewers observed, the contribution is indeed that the integration is surprisingly effective (even against heavy-training methods like Cartridge), and that all the components of the algorithm arise from considering one objective, the attention-output error on the context's own queries (Eqn. 1). We make this explicit in the revision (Section 3 before 3.1, and new paragraph "How RAPID differs" in Section 2).

  2. Compression cost: We agree that the submission did not measure the cost. We now do so in the revision: 153 s for a 64K-token context at 20× on GPT-OSS-120B. Furthermore, in response to reviewers’ cost concerns, we refined how the allocation weighs each latent entry, computing the sensitivity directly from a second-order model of the same objective (Eqn. 1). See Appendix K.

  3. Inference efficiency: we agree with the reviewers that a smaller cache does not by itself give faster decoding, and hence implemented a decode kernel. While the straight-forward kernel implementation is slower, we also added a decode kernel for a kernel-aligned version of RAPID, where the optimization problem for the bit allocation is further constrained. (It restricts the stored format precisely with the three overheads the reviewers mention as being worrisome.) These constraints affect the accuracy only by ~1 point. Exact details in new Section 5 and Appendix N.

  4. We added additional comparisons/experiments as requested.

List of changes to the submitted pdf of the paper, aiming to address the reviewer’s questions:

  • Decode time evaluation: we added Section 5 (instead of original Sec 3.4).
  • Compression time and storage: now reported in new Appendix K.
  • Revised sensitivity computation: in response to reviewers’ concern, we streamlined sensitivity computation in the algorithm. Changes propagate to: Sec 3 to 3.1, cost term in 3.2. Appendix B. This initialization reduced the number of required GD from ~50 to ~8, or even none at all (we make relevant updates throughout the manuscript).
  • Extended experimental section (in Sec 4): added Ruler and additional baselines (Palu, MixedDimKV, RestoreKV, LCLM, KV-Surgeon).
  • Extended ablations for algorithm components and hyper parameters (Appendix F)
  • Comparison to cartridge, end to end (Appendix I)
  • Polishing writing: related work (Sec 2) / clarifying methodological innovation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.