- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work concentrates on extreme KV cache compression (20x level) and claims that the proposed method can achieve near-full quality. RAPID jointly adopts strategies such as layer-wise latent representation and cross layer budget allocation, which is free from massive training. Extensive experiments with analyses have verified the effectiveness of the proposed model.
Critical strengths
- The authors have conducted extensive evaluations and carefully analyzed the results, making the conclusion more solid.
- Some challenging settings (e.g., with MLA) have been considered.
Critical weaknesses
- Some essential hyper-parameters should be clearly introduced with parameter analyses, for example, the prefix floor and the probe rank.
Questions for authors
- Attention matching is supposed to be compared on other backbones, e.g., GPT-OSS.
- The cost of RAPID++ is rather too high.
2 comments
Thanks for the rebuttal, which addressed most of my concerns. I will keep my positive score.
Response to Reviewer NVwC
Thanks for the detailed comments that materially improved the paper!
W1: the prefix floor and the probe rank. Both are gone in the revision. The revision replaces probe ranks by the sensitivity of each latent entry under a second-order model of the objective, computed with one backward pass over the reference queries (Section 3 before 3.1, Eqn. 2). The floor of 8 coordinates per token is removed as well, so the allocation now sets every prefix and may evict a token (Appendix B.2). One table lists every remaining setting, its value, and where it is varied (Appendix B.4, Table 2).
Q1: Attention Matching on GPT-OSS-120B. Attention Matching gives each KV head its own budget, computed once per model from per-head sensitivity curves on calibration contexts (Zweiger et al., 2026). Its authors provide these budgets for the models of their paper, which include Qwen3-4B and Llama-3.1-8B, but not for GPT-OSS-120B, and calibrating them takes substantial GPU time. We therefore compare with it on these two models, where the comparison covers both of its versions (repeat-context and self-study queries), RULER as well as LongHealth, and its own answer protocol (Table 1 and Appendices D.1, D.5 and E). In Table 1, RAPID++ is above it at every ratio on both models, by 12 to 16 points on LongHealth at 50× and by 12 to 33 points on RULER at 20×.
On GPT-OSS-120B, the baselines are KVzip, Ada(KVzip), and Ada(KVzip) + KIVI-2. At 20×, RAPID++ scores 75.5 on LongHealth and 68.1 on RULER, against 66.9 and 46.5 for the best baseline on each and 76.6 and 70.4 for the uncompressed cache (Table 1).
Q2: the cost of RAPID++. We take this point and now measure the cost . For a 64K-token context at 20× on one H200, RAPID++ takes 153s on GPT-OSS-120B, 313 s on GLM-4.7-Flash, 347 s on Qwen3-4B, and 356 s on Llama-3.1-8B. Beyond the prefill and the read that captures the queries, compression does not need the model's weights (only each layer's attention projections), so the time does not grow with the model: the 117B-parameter GPT-OSS-120B is the fastest of the four. For comparison, Attention Matching in its authors’ code takes 300s and 255s in the same setting on Qwen3-4B and Llama-3.1-8B. Training a Cartridge often takes several GPU-hours per context.
For reference, the time of the fitted baselines in the same setting, in seconds (Appendix K, Table 29):
Following reviewers’ remarks about this cost, we slightly revised the calculation of the sensitivity (that improved both performance and accuracy). The revised sensitivity calculation also cut the refinement from 50 steps to 8. At 10× and 20× the refinement can even be skipped: without it, RAPID++ takes 109 to 277 s and stays within 1.4 points of the uncompressed cache on LongHealth and 2.2 points on RULER, on every model (Table 1). The cost is paid once per context, and the compressed cache then serves every query on it.