- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies KV cache offloading on long context problems that require the model to recover many pieces of information from the prompt (instead of one or few “needles”). The argument is that the standard evaluations such as RULER underrepresent this regime.
They introduce Text2JSON, a 500 example structured extraction benchmark, and evaluate alongside other long context benchmarks such as MultiNeedle-128K, LongProc, etc. Empirically, the authors find that several offloading methods suffer from accuracy degradation on these workloads.
Through ablations on components for these methods, they attribute much of this degradation to 1) aggressive low-rank approximations of keys, and 2) grouped KV selection. Following from this, they propose YAKV, a simple alternative that uses 4-bit HIGGS quantization for offloaded KVs and approximately 2-bit per key representations for selection. This avoids SVD and token grouping. Across Qwen3 and Llama3 models, YAKV generally improves accuracy over baselines such as ShadowKV, ArkVale, LRQK, and InfiniGen at matched transfer budget, although the gap to full attention varies considerably depending on the model and task.
Aside: all figures, especially Fig 1, could be significantly improved. There also seems to be significant \vspace hacking which gives an ugly looking result.
Critical strengths
The paper exposes a consequential evaluation blind spot and validates it.
The observation that commonly used KV-offloading benchmarks mostly require retrieving only a small amount of information is important. This is operationalized through “context intensity” by the number of required “needles” and evaluated on four different workloads. The finding that large degradation appears across multiple offloading algorithms across both model families makes this compelling.
The component ablation is strong and leads to a fruitful, actionable alternative.
The paper separately studies key compression, grouping, landmarks, and individual key selection. The experiments show that increasing SVD rank or eliminating SVD improves retrieval, that group landmarks can remain far from an oracle even with smaller groups, and that low-bit per key HIGGS representations can provide better selection. This gives the paper value beyond the proposed method, YAKV, which can lead to design of the future offloading methods.
Critical weaknesses
“Context intensity” explanation is not causally isolated.
The paper defines high context intensity as a threshold of at least 10 needles, but the evaluated tasks simultaneously vary in other factors such as output length, output structure, base task difficulty, context length, and required reasoning. Text2JSON is helpful in this regard since it reduces some confounds, but it is itself a templated synthetic benchmark, where entries are GPT-generated and mixed with FineWeb-Edu passages.
Despite the experiments convincingly establishing the failure of existing methods on the selected workloads, it less convincingly establishes that the causal variable is the amount of context that must be recovered. A controlled sweep over 1 to >16 required items at fixed context length, task format, and preferably comparable output length would be very helpful for providing evidence for this causality.
The practical comparison is incomplete.
The main accuracy comparisons equalize PCle traffic, which is logical but is only one system resource. Importantly, the headline throughput gains come from allowing YAKV to run at much larger batch sizes. The detailed runtime results show that this can incur large latency increases, with the example of YAKV increasing throughput from 25.3 to 76.2 tokens per second but TPOT from 38 to 420 ms and TTFT from 3.1s to 6.2s for Qwen3-30B-A3B-Instruct.
This demonstrates a useful trade off between throughput and latency rather than an unconditional serving improvement. I normally wouldn’t think this as a critical weakness but the appendix also shows that MagicPIG is competitive with or better than YAKV in accuracy on several Qwen settings, but no matched end-to-end latency comparison is provided. Accordingly, a matched throughput vs. TPOT/TTFT comparison would make the practical conclusion stronger.
Questions for authors
Can the authors please provide experimental results for the concerns in weaknesses? I would be willing to increase my score, if even only the first weakness is addressed.
4 comments
(Part 2) Throughput/Latency Comparison
This demonstrates a useful trade off between throughput and latency rather than an unconditional serving improvement.
While we do not claim unconditional improvement, we agree that the paper can benefit from stating this more clearly. In the final version, we will rewrite that specific claim to clearly state that we are performing large-batch high-throughput inference, e.g.:
(L463) “... showing that YAKV can maintain high inference throughput while being more accurate on context-intensive tasks” will be rewritten as “... showing that YAKV can perform high-throughput inference by fitting a larger batch into the same device memory. Thus, YAKV offloading provides a useful trade-off between throughput and latency while being more accurate than other offloading algorithms on context-intensive tasks.”
Accordingly, a matched throughput vs. TPOT/TTFT comparison would make the practical conclusion stronger.
To address this, we prioritized evaluating MagicPIG throughput in the same setup we use for Section 4.5, using K=11, L=300. We run it on the same H200 GPU and tune batch size to maximize throughput. Since the method does not release optimized inference code, we initially observed very poor runtime (>1s latency) and had to optimize inference code to get speedups. We also note that MagicPIG heavily depends on CPU usage (it computes attention on CPU), so we lifted restrictions so it could use all cores and 300GiB RAM. The results are summarized in Table C below:
Table C: Qwen3-30B-A3B-Thinking-2507, real data setup from Table 4 (bottom):
| Setup | Batch | avg. TPOT (ms) | Throughput (tok/s) | Rel. Speedup |
|---|---|---|---|---|
| Full Attn | 1 | 39 | 25.64 | 1.00 |
| MagicPIG | 4 | 198 | 20.3 | 0.79 |
| MagicPIG | 8 | 261 | 30.65 | 1.20 |
| YAKV | 4 | 108 | 37.04 | 1.44 |
| YAKV | 8 | 122 | 65.57 | 2.56 |
| YAKV | 32 | 252 | 127.0 | 4.95 |
To summarize, YAKV offloading outperforms MagicPIG primarily because the MagicPIG computes attention on CPU. With larger batch sizes, it shows less gain with batch size because CPU operations become the bottleneck. We could not run MagicPIG with batch size 32 because the LSH table ran out of 512gb system RAM (not GPU memory), whereas YAKV uses <50GiB. Note, however, that poor performance on H200 does not conflict with the main claims of MagicPIG. In their evaluations, the main gains were observed on much weaker GPUs (e.g. RTX 4090), where the CPU was less of a bottleneck. We offer some discussion of this in L1385-1403 and will include these additional evaluations to better explain the trade-offs between the two methods.
The detailed runtime results show that this can incur large latency increases … TPOT from 38 to 420 ms and TTFT from 3.1s to 6.2s for Qwen3-30B-A3B-Instruct.
Please also note that our TPOT includes the prefill time (as stated in L452-453). In the Instruct scenario we have TPOT 420, which is largely affected by frequent (due to short output lengths) large (up to 128k) prefills. We state this issue not as YAKV’s but rather fundamental for this prefill-heavy scenario. To prove our point we simulated unlimited VRAM for full attention baseline by serving it with dummy-weights and keeping KV-cache shared between layers, then compared it against YAKV under the exact same load.
Qwen3-30B-A3B-Instruct-2507
| batch size | 4 | 8 | 16 | 32 |
|---|---|---|---|---|
| full attention | 59ms | 91ms | 151ms | 277ms |
| YAKV | 124ms | 163ms | 247ms | 420ms |
Qwen3-30B-A3B-Thinking-2507
| batch size | 4 | 8 | 16 | 32 |
|---|---|---|---|---|
| full attention | 36ms | 55ms | 92ms | 156ms |
| YAKV | 108ms | 122ms | 150ms | 252ms |
As you can see, in both prefill-heavy and decode-heavy scenarios we have comparable latency with large-batch full attention.
We hope that these clarifications and additional experiments address your concerns. If you have follow-up suggestions, please leave a follow-up comment so that we can continue the discussion.
Thank you for the new experiments and clarifications.
Context intensity. The needle sweep resolves my main concern. With the context length, task format and hyperparameters all fixed, ShadowKV matches full attention at 1-3 needles (and the gap widens as the amount of needles required increases). I believe this is good evidence that context intensity drives the degradation. Notably, YAKV also degrades with more needles, but more slowly (i.e., about 2.5 points by 31 needles). I would describe this as moderate degradation rather than near-lossless. Output length still grows with needle count, so it isn’t perfectly controlled, but the trend is sufficiently convincing. Please add this sweep to the paper.
Practical comparison. The revised L463 wording fixes the overclaim. The MagicPIG results at matched batch size and the simulated full-attention baseline clarify the trade-off. Much of the latency increase here seems to come from batching itself, and not from YAKV. However, your tables still show YAKV is 1.5-3 times slower per token than full attention at the same batch size. I think this is an non-negligible overhead, not “comparable latency”. The final version should explicitly acknowledge so and also present MagicPIG’s accuracy advantage on LongProc alongside its runtime disadvantage.
Thank you for your thoughtful review and valuable comments. Your feedback has helped us clarify the positioning of our work and improve its quality. As suggested, we will include the statements mentioned above in the final revision and acknowledge the limitations of our method.
(Part 1) Isolating Context Intensity
Thank you for the detailed feedback and insightful comments. We conducted additional experiments to address your concerns and report them below.
To address this, we ran modified variants of the (synthetic) MultiNeedle-128K benchmark where we change the number of needles inserted at the time of data generation. The default evaluation uses 11 needles. Following your suggestion, we sweep between 1 and 31 needles and keep all other parameters equal, with the exception of maximum output length: for 21 and 31 needles, we extend the maximum output length from 1024 to 2048 tokens after we noticed that some answers were cut short. While output length naturally changes with the number of needles, the evaluation accuracy averages over sub-questions. We evaluate YAKV, ShadowKV, and the original full attention baseline on Llama-3.1-8B-Instruct and Qwen-4B-Instruct-2507 models. We use the same hyperparameters as in Table 3, except for the number of needles and max output length.
Table A: Llama-3.1-8B-Instruct Accuracy on MultiNeedle-128k w/ varying number of needles
Table B: Qwen3-4B-Instruct-2507 Accuracy on MultiNeedle-128k w/ varying number of needles
The results demonstrate that, in this controlled setting with explicit context-intensity, the two KV offloading methods perform similarly for 1-5 needles, where the task is similar to ordinary NIAH, but the accuracy drawdown increases as we reach the full context intensity of MultiNeedle-128K. We agree that a clean, controlled evaluation helps to decouple context intensity from other factors and thank you for suggesting this evaluation.