acceptodds
Under review as a conference paper at ICLR 2027

KV Cache Offloading for Context-Intensive Tasks

Abstract

With the growing demand for long-context LLMs across a wide range of applications, the key–value (KV) cache has become a critical bottleneck for both latency and memory usage. Recently, KV-cache offloading has emerged as a promising approach to reduce memory footprint and inference latency while preserving accuracy. Most recent works on KV offloading evaluate on tasks that do not require extracting large amounts of information from the context. In this work, we study KV-cache offloading on context-intensive tasks: problems where the solution requires looking up a lot of information from the input prompt. We create and release the Text2JSON benchmark, a highly context-intensive task that requires extracting structured knowledge from raw text. We evaluate modern KV offloading on Text2JSON and other context-intensive tasks and find significant performance degradation on both Llama 3 and Qwen3 models. Our analysis identifies two key reasons for poor accuracy: low-rank projection of keys and unreliable selection, and propose a simpler alternative strategy that significantly improves accuracy across multiple LLM families and benchmarks. These findings highlight the need for a comprehensive and rigorous evaluation of long-context compression techniques.

Primary area:
foundation or frontier models, including LLMs
Keywords:
efficiency, large language models, kv cache, long context

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_41vkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies KV cache offloading on long context problems that require the model to recover many pieces of information from the prompt (instead of one or few “needles”). The argument is that the standard evaluations such as RULER underrepresent this regime.

They introduce Text2JSON, a 500 example structured extraction benchmark, and evaluate alongside other long context benchmarks such as MultiNeedle-128K, LongProc, etc. Empirically, the authors find that several offloading methods suffer from accuracy degradation on these workloads.

Through ablations on components for these methods, they attribute much of this degradation to 1) aggressive low-rank approximations of keys, and 2) grouped KV selection. Following from this, they propose YAKV, a simple alternative that uses 4-bit HIGGS quantization for offloaded KVs and approximately 2-bit per key representations for selection. This avoids SVD and token grouping. Across Qwen3 and Llama3 models, YAKV generally improves accuracy over baselines such as ShadowKV, ArkVale, LRQK, and InfiniGen at matched transfer budget, although the gap to full attention varies considerably depending on the model and task.

Aside: all figures, especially Fig 1, could be significantly improved. There also seems to be significant \vspace hacking which gives an ugly looking result.

Critical strengths

The paper exposes a consequential evaluation blind spot and validates it.

The observation that commonly used KV-offloading benchmarks mostly require retrieving only a small amount of information is important. This is operationalized through “context intensity” by the number of required “needles” and evaluated on four different workloads. The finding that large degradation appears across multiple offloading algorithms across both model families makes this compelling.

The component ablation is strong and leads to a fruitful, actionable alternative.

The paper separately studies key compression, grouping, landmarks, and individual key selection. The experiments show that increasing SVD rank or eliminating SVD improves retrieval, that group landmarks can remain far from an oracle even with smaller groups, and that low-bit per key HIGGS representations can provide better selection. This gives the paper value beyond the proposed method, YAKV, which can lead to design of the future offloading methods.

Critical weaknesses

“Context intensity” explanation is not causally isolated.

The paper defines high context intensity as a threshold of at least 10 needles, but the evaluated tasks simultaneously vary in other factors such as output length, output structure, base task difficulty, context length, and required reasoning. Text2JSON is helpful in this regard since it reduces some confounds, but it is itself a templated synthetic benchmark, where entries are GPT-generated and mixed with FineWeb-Edu passages. 

Despite the experiments convincingly establishing the failure of existing methods on the selected workloads, it less convincingly establishes that the causal variable is the amount of context that must be recovered. A controlled sweep over 1 to >16 required items at fixed context length, task format, and preferably comparable output length would be very helpful for providing evidence for this causality.

The practical comparison is incomplete.

The main accuracy comparisons equalize PCle traffic, which is logical but is only one system resource. Importantly, the headline throughput gains come from allowing YAKV to run at much larger batch sizes. The detailed runtime results show that this can incur large latency increases, with the example of YAKV increasing throughput from 25.3 to 76.2 tokens per second but TPOT from 38 to 420 ms and TTFT from 3.1s to 6.2s for Qwen3-30B-A3B-Instruct.

This demonstrates a useful trade off between throughput and latency rather than an unconditional serving improvement. I normally wouldn’t think this as a critical weakness but the appendix also shows that MagicPIG is competitive with or better than YAKV in accuracy on several Qwen settings, but no matched end-to-end latency comparison is provided. Accordingly, a matched throughput vs. TPOT/TTFT comparison would make the practical conclusion stronger.

Questions for authors

Can the authors please provide experimental results for the concerns in weaknesses? I would be willing to increase my score, if even only the first weakness is addressed.

4 comments
AuthorsOpenReview ↗

(Part 1) Isolating Context Intensity

Thank you for the detailed feedback and insightful comments. We conducted additional experiments to address your concerns and report them below.

A controlled sweep over 1 to >16 required items at fixed context length, task format, and preferably comparable output length would be very helpful for providing evidence for this causalty.

To address this, we ran modified variants of the (synthetic) MultiNeedle-128K benchmark where we change the number of needles inserted at the time of data generation. The default evaluation uses 11 needles. Following your suggestion, we sweep between 1 and 31 needles and keep all other parameters equal, with the exception of maximum output length: for 21 and 31 needles, we extend the maximum output length from 1024 to 2048 tokens after we noticed that some answers were cut short. While output length naturally changes with the number of needles, the evaluation accuracy averages over sub-questions. We evaluate YAKV, ShadowKV, and the original full attention baseline on Llama-3.1-8B-Instruct and Qwen-4B-Instruct-2507 models. We use the same hyperparameters as in Table 3, except for the number of needles and max output length.

Table A: Llama-3.1-8B-Instruct Accuracy on MultiNeedle-128k w/ varying number of needles

num. needles1357911(original)162131
Original10010098.396.495.994.593.387.258.6
YAKV(ours)10010097.895.695.393.491.985.556.2
ShadowKV10010095.890.386.685.780.477.146.8

Table B: Qwen3-4B-Instruct-2507 Accuracy on MultiNeedle-128k w/ varying number of needles

num. needles1357911(original)162131
Original10010010098.998.298.995.694.083.1
YAKV(ours)10010099.298.497.898.494.392.780.6
ShadowKV10010098.696.392.692.687.885.074.4

The results demonstrate that, in this controlled setting with explicit context-intensity, the two KV offloading methods perform similarly for 1-5 needles, where the task is similar to ordinary NIAH, but the accuracy drawdown increases as we reach the full context intensity of MultiNeedle-128K. We agree that a clean, controlled evaluation helps to decouple context intensity from other factors and thank you for suggesting this evaluation.

AuthorsOpenReview ↗

(Part 2) Throughput/Latency Comparison

This demonstrates a useful trade off between throughput and latency rather than an unconditional serving improvement.

While we do not claim unconditional improvement, we agree that the paper can benefit from stating this more clearly. In the final version, we will rewrite that specific claim to clearly state that we are performing large-batch high-throughput inference, e.g.:

(L463) “... showing that YAKV can maintain high inference throughput while being more accurate on context-intensive tasks” will be rewritten as “... showing that YAKV can perform high-throughput inference by fitting a larger batch into the same device memory. Thus, YAKV offloading provides a useful trade-off between throughput and latency while being more accurate than other offloading algorithms on context-intensive tasks.”

Accordingly, a matched throughput vs. TPOT/TTFT comparison would make the practical conclusion stronger.

To address this, we prioritized evaluating MagicPIG throughput in the same setup we use for Section 4.5, using K=11, L=300. We run it on the same H200 GPU and tune batch size to maximize throughput. Since the method does not release optimized inference code, we initially observed very poor runtime (>1s latency) and had to optimize inference code to get speedups. We also note that MagicPIG heavily depends on CPU usage (it computes attention on CPU), so we lifted restrictions so it could use all cores and 300GiB RAM. The results are summarized in Table C below:

Table C: Qwen3-30B-A3B-Thinking-2507, real data setup from Table 4 (bottom):

SetupBatchavg. TPOT (ms)Throughput (tok/s)Rel. Speedup
Full Attn13925.641.00
MagicPIG419820.30.79
MagicPIG826130.651.20
YAKV410837.041.44
YAKV812265.572.56
YAKV32252127.04.95

To summarize, YAKV offloading outperforms MagicPIG primarily because the MagicPIG computes attention on CPU. With larger batch sizes, it shows less gain with batch size because CPU operations become the bottleneck. We could not run MagicPIG with batch size 32 because the LSH table ran out of 512gb system RAM (not GPU memory), whereas YAKV uses <50GiB. Note, however, that poor performance on H200 does not conflict with the main claims of MagicPIG. In their evaluations, the main gains were observed on much weaker GPUs (e.g. RTX 4090), where the CPU was less of a bottleneck. We offer some discussion of this in L1385-1403 and will include these additional evaluations to better explain the trade-offs between the two methods.

The detailed runtime results show that this can incur large latency increases … TPOT from 38 to 420 ms and TTFT from 3.1s to 6.2s for Qwen3-30B-A3B-Instruct.

Please also note that our TPOT includes the prefill time (as stated in L452-453). In the Instruct scenario we have TPOT 420, which is largely affected by frequent (due to short output lengths) large (up to 128k) prefills. We state this issue not as YAKV’s but rather fundamental for this prefill-heavy scenario. To prove our point we simulated unlimited VRAM for full attention baseline by serving it with dummy-weights and keeping KV-cache shared between layers, then compared it against YAKV under the exact same load.

Qwen3-30B-A3B-Instruct-2507

batch size481632
full attention59ms91ms151ms277ms
YAKV124ms163ms247ms420ms

Qwen3-30B-A3B-Thinking-2507

batch size481632
full attention36ms55ms92ms156ms
YAKV108ms122ms150ms252ms

As you can see, in both prefill-heavy and decode-heavy scenarios we have comparable latency with large-batch full attention.

We hope that these clarifications and additional experiments address your concerns. If you have follow-up suggestions, please leave a follow-up comment so that we can continue the discussion.

Thank you for the new experiments and clarifications.

Context intensity. The needle sweep resolves my main concern. With the context length, task format and hyperparameters all fixed, ShadowKV matches full attention at 1-3 needles (and the gap widens as the amount of needles required increases). I believe this is good evidence that context intensity drives the degradation. Notably, YAKV also degrades with more needles, but more slowly (i.e., about 2.5 points by 31 needles). I would describe this as moderate degradation rather than near-lossless. Output length still grows with needle count, so it isn’t perfectly controlled, but the trend is sufficiently convincing. Please add this sweep to the paper.

Practical comparison. The revised L463 wording fixes the overclaim. The MagicPIG results at matched batch size and the simulated full-attention baseline clarify the trade-off. Much of the latency increase here seems to come from batching itself, and not from YAKV. However, your tables still show YAKV is 1.5-3 times slower per token than full attention at the same batch size. I think this is an non-negligible overhead, not “comparable latency”. The final version should explicitly acknowledge so and also present MagicPIG’s accuracy advantage on LongProc alongside its runtime disadvantage.

Thank you for your thoughtful review and valuable comments. Your feedback has helped us clarify the positioning of our work and improve its quality. As suggested, we will include the statements mentioned above in the final revision and acknowledge the limitations of our method.

Reviewer_nndYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies KV-cache offloading on context-intensive tasks that require extracting information from many input positions. It introduces Text2JSON and analyzes failures caused by key compression and approximate token selection. Based on these findings, YAKV combines quantized per-token key proxies with quantized KV offloading. Experiments evaluate accuracy across four task suites and report continuous-batching throughput.

Critical strengths

The paper provides useful component-level analysis of KV-offloading failures beyond conventional retrieval benchmarks. Experiments varying key compression, grouping, and selection—including oracle-selection controls—help distinguish representation errors from retrieval errors. Validation on several existing benchmarks strengthens the contribution beyond the newly introduced dataset.

Critical weaknesses
  1. The near-lossless accuracy claim is overstated. On Qwen3-30B-A3B, LongProc performance drops from 59.70 with full attention to 38.00 with YAKV, while MagicPIG achieves 56.78. Substantial degradation also remains on Llama-based models. These results limit the claim that the proposed changes adequately address context-intensive workloads.
  2. The end-to-end efficiency advantage remains insufficiently established. Runtime comparisons primarily use full attention rather than competing offloading or GPU-resident quantized-KV methods. Higher throughput comes with substantially increased reported TPOT—for example, 38 to 420 ms on the Instruct workload. Matched PCIe transfer budgets alone do not establish a favorable accuracy–latency–memory trade-off.
Questions for authors
  1. How should the near-lossless claim be reconciled with the substantial LongProc and Llama-model degradation? What explains the remaining gap, particularly where MagicPIG preserves considerably higher accuracy?
  2. Under comparable accuracy and latency constraints, what throughput and GPU/CPU-memory advantages does YAKV provide over competing offloading methods and GPU-resident quantized-KV baselines?
4 comments
AuthorsOpenReview ↗

Response to Weaknesses/Questions

Thank you for your encouraging feedback and questions. We answer your questions and address your concerns below.

(weakness 1 & question 1) How should the near-lossless claim be reconciled with the substantial LongProc and Llama-model degradation?

We believe that a slight revision to lines 73–74 would make the claim clearer. Our intended meaning is that YAKV achieves near-lossless accuracy on many context-intensive tasks for which previous offloading methods introduced significant accuracy gaps. However, there are cases in which the base model itself performs poorly or the output format breaks, as well as reduced-budget settings in which no algorithm is lossless. We therefore propose revising the claim to “YAKV achieves near-lossless accuracy on most context-intensive tasks.” We believe this clarification preserves our core contribution. We welcome feedback on whether this revised wording adequately addresses your concern.

What explains the remaining gap, particularly where MagicPIG preserves considerably higher accuracy?

To the best of our knowledge, this affects YAKV and other offloading methods only on LongProc (Table 9), whereas other evaluations have much smaller accuracy gaps. After conducting an error analysis of Qwen3-4B on LongProc, we found that the default configuration of YAKV seems to affect output formatting on some examples. In about a third of additional errors, the model changed the order of columns or (in 2 cases) added unnecessary columns. Also, there were a few cases where the model changed the numeric formatting (fractions vs decimal). Under the LongProc grader’s scoring rules, an incorrectly formatted column invalidates the entire row, and changing the column order invalidates the entire sample.

We also note that both MagicPIG, YAKV and full attention make similar errors in different samples, but MagicPIG made less of them. On closer inspection, we found that the official system prompt (LongProc/data/html_to_tsv/prompts.yaml) does not explain that these changes (e.g. column order) will cause the model to be scored at 0, so a small change to model behavior introduced by offloading could feasibly cause it to choose a different seemingly valid order that gets rejected by the grader. On other datasets, MagicPIG achieves accuracy comparable to YAKV’s, but its practical applicability is limited by its substantial system-memory requirements and CPU-based attention computation (see lines 1385–1402).

(weakness 2, question 2) Under comparable accuracy and latency constraints, what throughput and GPU/CPU-memory advantages does YAKV provide over competing offloading methods and GPU-resident quantized-KV baselines?

To address this, we ran accuracy and latency evaluations for three other offloading methods and summarize our findings below. To better compare accuracy to throughput trade-offs, we also include Text2JSON accuracy for Qwen3-30B-A3B-Instruct-2507 in the last column.

Table A: Qwen3-30B-A3B inference throughput and accuracy across methods

SetupBatch sizeMean TPOT (ms)Throughput (tokens/s)Relative speedup (×)Text2JSON accuracy (%)
Full attention139.025.61.097.9
ArkVale16246.665.02.546.1
MagicPIG8261.430.61.297.3
ShadowKV16104.4153.86.080.4
YAKV32252.0127.04.996.7

We compare all methods on a single H200 GPU, Intel Xeon Platinum 8558, using the official pytorch/pytorch:2.9.1-cuda12.8-cudnn9-devel Docker container. For each method, we tune batch size to maximize throughput in the reasoning on real data evaluation setup (Table 4, bottom) while fitting into memory. For MagicPIG, which computes attention on the CPU, we found that the original paper used relatively slow GPUs (L20, RTX 4090) where the method can provide significant speedup. However, on the higher performance H200 GPU, we found that the method is severely bottlenecked by its CPU attention code (lines 1390–1402) and used over 300 GiB system RAM for the LSH tables even with batch size 8. We implemented several output-preserving code optimizations and tuned hyperparameters to obtain speedup above 1, but the algorithm appears to be more practical for slow GPU / strong CPU setups than the high-end GPUs we evaluate on.

Overall, YAKV provides a good trade-off between accuracy and latency, with significantly higher throughput than the only method of comparable accuracy. In the revised manuscript, we will include an accuracy-throughput plot with alternative configurations (i.e., lower throughput but lower latency) for each method.

We are happy to continue the discussion if you have any further suggestions or follow-up questions.

Thank you for the clarification and additional experiments. Table A provides useful evidence for YAKV’s throughput advantage over competing offloading methods.

Two points remain unclear:

  1. Does the formatting-based explanation also account for the larger LongProc gap on Qwen3-30B-A3B and the degradation on Llama-based models?
  2. Could you provide GPU/CPU memory usage for the compared methods and clarify whether GPU-resident quantized-KV baselines were evaluated? The proposed alternative configurations would also help assess the latency–throughput trade-off.
AuthorsOpenReview ↗

Addressing the additional two points

Does the formatting-based explanation also account for the larger LongProc gap on Qwen3-30B-A3B and the degradation on Llama-based models?

On Qwen3-30B-A3B: yes, but there are slightly fewer errors overall. On Llama-based models, it’s not quite the same. Instead, the two Llama-based models we tested often lose track of their assignment after generating the first twenty to fifty items. This seems to affect both baseline full attention and offloading methods, including MagicPIG.

Could you provide GPU/CPU memory usage for the compared methods and clarify whether GPU-resident quantized-KV baselines were evaluated?

We compiled per-algorithm memory usage under sustained load. In the table below, VRAM is the H200 GPU memory, and RAM is system memory (ps RSS).

Table B: GPU memory usage across different algorithms, H200

SetupBatchVRAMRAM
Full Attn1106.7 GiB1.6 GiB
ArkVale1692.9 GiB193.9 GiB
MagicPIG898.0 GiB308.0 GiB
ShadowKV16130.0 GiB105.0 GiB
YAKV32109.9 GiB101.6 GiB

On GPU-resident quantized-KV baselines. We did not evaluate GPU-based quantization in our original work, as it would require rewriting and re-optimizing our reference implementation to provide a fair comparison. That said, we conducted additional experiments to help estimate how they compare. Namely, we implemented KIVI [1], a popular training-free KV quantization method that quantizes keys and values along different dimensions to reduce error. The table below compares 2-bit and 4-bit KIVI on one of the models to establish how different quantization bit widths score on context-intensive tasks.

Table C: Accuracy on context-intensive tasks for meta-llama/Llama-3.1-8B-Instruct*

MethodText2JSONMultiNeedle
Original46.7594.55
KIVI 4-bit44.4294.36
KIVI 2-bit11.2980.09

These points of comparison show that 4-bit quantization is competitive in accuracy, but 2-bit quantization degrades accuracy far past offloading methods. The theoretical upper bound speed-up for quantization is 4× for 4-bit and 8× for 2-bit from fitting larger batch sizes and perfectly optimized computations. The real throughput speed-ups observed in the original KIVI paper are 2.35×–3.47× for 2-bit KIVI on A100 (see Figure 5 in [1]); the paper does not report 4-bit speed-ups. In turn, the offloading speed-ups we observe in Section 4.5 and the baseline methods evaluated for this discussion obtain up to 4.9× higher throughput against bf16 full attention (up to 6× for ShadowKV at the cost of some accuracy), which is higher than the upper bound for 4-bit resident quantization.

Please note that this is a very limited comparison, as there are other quantization schemes and evaluation scenarios. We hope that our findings above show, if little else, that offloading has at least some niches where it is the optimal choice. To summarize, GPU-resident quantization algorithms are generally easier to implement than offloading and, unlike offloading, do not seem to struggle with context-intensive tasks. In turn, offloading has greater potential speed-ups and can fit extremely long context lengths where GPU-resident methods would not work at all.

The proposed alternative configurations would also help assess the latency–throughput trade-off.

We updated the submission PDF to include the additional configurations in Appendix M, Figure 13 on page 29 (L1542–1565).

[1] Liu, Zirui, et al. “Kivi: A tuning-free asymmetric 2-bit quantization for KV cache.” arXiv preprint arXiv:2402.02750 (2024).

Thank you for the clarifications and additional experiments. The memory measurements, KIVI results, and additional configurations help address my concerns and better characterize the method’s trade-offs. I maintain my original rating of 3 (Weak acceptance).

Reviewer_TGUROpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how existing KV cache offloading methods perform when tasks require extracting substantial information from the context. It introduces Text2JSON and shows cases where these methods suffer large accuracy drops. The authors trace these failures to low-rank key compression and group-based selection, then improve accuracy with YAKV, which selects tokens individually using quantized keys.

Critical strengths

S1. Dataset usefulness: The paper introduces Text2JSON to evaluate tasks requiring extensive information extraction from the context, an aspect not adequately covered by existing benchmarks.

S2. Clear positioning: The paper clearly explains how its approach differs from existing offloading methods and supports these design choices with experiments on key compression and selection.

Critical weaknesses

W1. Limited runtime comparison: The paper compares accuracy against several offloading methods under matched PCIe transfer budgets, but compares speed only against Full Attention. Even with the same transfer volume, the time spent selecting and reconstructing KV entries can differ. Direct throughput and latency comparisons with existing offloading methods are needed to assess YAKV’s practical benefits.

W2. Limited evaluation across model families: The abstract claims gains “across multiple LLM families,” but experiments cover only Llama and Qwen3. Please add results on other model families such as Mistral.

Questions for authors

See weakness

2 comments
AuthorsOpenReview ↗

Response to Weaknesses/Questions

Thank you for your feedback! We are glad that you value the core idea of our approach. We address your concerns below in the order you raised them.

(W1) Direct throughput and latency comparisons with existing offloading methods are needed to assess YAKV’s practical benefits.

We agree that additional comparisons would improve our work. Following suggestions from you and the other reviewers, we compare the runtime throughput of YAKV, ShadowKV, and ArkVale in the real data inference setup from Section 4.5 (real data, reasoning). We also include MagicPIG, an accurate but CPU-heavy LSH-based offloading method that computes attention on CPU (overview in Appendix K). To better compare accuracy–throughput trade-offs, we also include Text2JSON accuracy for Qwen3-30B-A3B-Instruct-2507 in the last column.

We compare all methods on a single H200 GPU, Intel Xeon Platinum 8558, using the official pytorch/pytorch:2.9.1-cuda12.8-cudnn9-devel Docker container. For each method, we tune batch size to maximize throughput in the reasoning on real data evaluation setup (Table 4, bottom) while fitting into memory. For MagicPIG, which computes attention on the CPU, we found that the original paper used relatively slow GPUs (L20, RTX 4090) where the method can provide significant speedup. In our H200 setup, MagicPIG was limited by CPU attention computation and required over 300 GiB of system RAM for its LSH tables at batch size 8. We applied output-preserving code optimizations and tuned its hyperparameters to improve throughput. These results are specific to our hardware configuration; the relative performance may differ on systems with a different balance of CPU and GPU capabilities. The table below reports the highest-throughput configurations for each method:

Table A: Qwen3-30B-A3B inference throughput and accuracy across methods

SetupBatch sizeMean TPOT (ms)Throughput (tokens/s)Relative speedup (×)Text2JSON accuracy (%)
Full attention139.025.61.097.9
ArkVale16246.665.02.546.1
MagicPIG8261.430.61.297.3
ShadowKV16104.4153.86.080.4
YAKV32252.0127.04.996.7

At the throughput-maximizing configurations reported here, YAKV achieves higher Text2JSON accuracy than ArkVale and ShadowKV, with throughput between the two methods. Compared with MagicPIG, YAKV achieves approximately 4.2× higher throughput, with Text2JSON accuracy of 96.7% versus 97.3%.

The abstract claims gains “across multiple LLM families,” but experiments cover only Llama and Qwen3. Please add results on other model families such as Mistral.

We agree that more extensive evaluation across model families would strengthen the positioning of the paper.

Following your suggestion, we adapted YAKV to MistralAI’s Ministral-3 models,specifically: mistralai/Ministral-3-8B-Instruct-2512 and mistralai/Ministral-3-14B-Instruct-2512. We chose these models as they are have native support of long contexts. Below, we evaluate them in the same configuration we use for Llama-3.1-8B-Instruct in Table 3.

Table B: Ministral-3-8B-Instruct-2512

MethodText2JSONMultiNeedleLongProcLoongAverage
Full attention97.4498.5543.5449.1972.18
YAKV96.1396.5537.7641.7368.04
ShadowKV88.9591.9126.6939.0361.65

Table C: Ministral-3-14B-Instruct-2512

MethodText2JSONMultiNeedleLongProcLoongAverage
Full attention97.0799.9158.554.5677.51
YAKV97.8510052.647.1974.41
ShadowKV92.199842.6844.9269.45

To summarize, YAKV consistently outperforms ShadowKV across benchmarks and scores closer to full attention in terms of average accuracy. Similar to our Llama-3.1 experiments, both methods reduce accuracy on LongProc HTML to TSV, though YAKV has significantly smaller accuracy gaps than ShadowKV.

We hope that the additional experiments help alleviate your concerns. If you have any follow-up suggestions, we are eager to continue our discussion.

Thank you for the additional experiments, which address my concerns. I have also read Reviewer nndY’s follow-up questions and hope the authors will address them. I will maintain my positive score (weak accept).

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their rebuttals and updated their manuscript/responses. Please carefully read the authors' responses and engage in the discussion to clarify any remaining concerns.

In light of the rebuttal, please let the authors know whether your concerns have been adequately addressed, and update your review comments and scores if appropriate. Active participation is essential for us to reach a well-calibrated decision.

Please note that the author-reviewer discussion phase ends on November 18 at 11:59 PM AoE, after which no further responses can be posted. Thank you for your continued time and effort.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.