- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
BudgetKV reduces KV-cache memory during chunked prefill when the task query is known in advance. It combines a fixed lexical policy with scores from the model’s frozen input embeddings, then exchanges equally sized token blocks.
Under protected-block feasibility, persistent KV residency is bounded by B after selection and B+p between selections. This bound excludes temporary allocations and does not automatically extend to decoding.
Experiments cover two small models, five LongBench tasks, adapted RULER diagnostics through 32K, and an 8GB Jetson. Semantic refinement significantly improves one of four primary model–budget settings. The clearest benefit is retrieval; reasoning improvements are inconsistent.
Critical strengths
-
Practical motivation BudgetKV addresses requests that may fail during prefill even when their final compressed cache would fit. Reusing resident embeddings avoids an additional encoder or profiling pass.
-
Controlled and transparent evaluation The semantic ablation matches realized final-prefill counts. Qwen3-1.7B at (B=1024) gains 6.81 points. The authors appropriately acknowledge inconclusive results elsewhere.
Critical weaknesses
-
Competitiveness remains unresolved Query-visible SnapKV is stronger in all four controlled diagnostic settings (Figure 9). The larger favorable comparison uses query-hidden SnapKV and different decoding policies. EpiCache, a close bounded-prefill method that also studies target-model embeddings, is not experimentally compared. Recent lightweight selectors such as Fast KVzip are also absent. These gaps leave the advantage under the intended memory constraint unclear.
-
Evidence for accuracy preservation is narrow Only one of four primary semantic comparisons is significant after correction. At 32K, retrieval remains below Full KV, while reasoning gains are inconsistent. The five-task LongBench subset and three adapted RULER tasks do not establish broad long-context performance. NoLiMa would be particularly relevant because it tests retrieval with little lexical overlap. The Jetson capacity experiment also omits lexical-only retention and StreamingLLM, so it does not isolate the deployment benefit of the semantic component.
Questions for authors
-
How does BudgetKV compare with query-visible SnapKV and EpiCache when query access, precision, inputs, decoding policy, and peak-memory constraints are matched?
-
Does the semantic gain persist on low-overlap query–evidence pairs? Please provide per-task results and evidence-retention measurements to clarify where it helps.
-
Do lexical-only retention and StreamingLLM also complete the six 8K Jetson requests? What accuracy do all methods achieve on those same requests?
-
How were the lexical rules, margins, and swap cap selected? Does the main improvement survive reasonable changes, including removal of the literal “KV” bonus?
8 comments
W2
Reviewer comment
Evidence for accuracy preservation is narrow.
Response
We are very sorry that the original discussion did not connect desktop accuracy to edge deployment, making the semantic contribution difficult to judge. We explain those outcomes first, then test the edge balance with the requested controls.
1. What the earlier desktop results mean. The four semantic contrasts and 32K quality diagnostic ran on an RTX 4090 D. The 400-document LongBench confirmation uses matched inputs/realized KV and task-stratified paired bootstrap intervals:
Table 23 (p. 20).
| Model | Gain | 95% CI | Family-adjusted CI | |
|---|---|---|---|---|
| Qwen3-1.7B | 512 | 2.64 | [, 6.42] | [, 7.52] |
| Qwen3-1.7B | 1024 | 6.81 | [3.31, 10.41] | [2.37, 11.45] |
| Llama-3.2-1B | 512 | 0.67 | [, 2.73] | [, 3.32] |
| Llama-3.2-1B | 1024 | 2.15 | [, 4.62] | [, 5.28] |
The early, non-contextual relevance signal helps select distant evidence, but its benefit depends on model, budget and task. Qwen/ gains 6.81 points, adjusted CI [2.37,11.45]; the other intervals include zero. Full KV preserves the entire context when desktop memory permits: at 32K, BudgetKV/Full-KV retrieval is 66.25/80.00 on Qwen and 41.25/76.25 on Llama. Compressing that evidence does not ensure equal QA/tracking quality, whose complete outcomes remain in Table 26.
2. The paper's deployment objective. We seek useful answers under an 8 GB edge device's memory and execution costs. Desktop quality is a supplementary selection test; on-device value requires both completing the long prompt and retaining evidence for its answer. Chunked retention addresses prefill residency, while semantic refinement improves the choice within that retained budget.
3. Do lexical retention and StreamingLLM also fit? Yes. The complete Jetson comparison includes both:
Table 7 (p. 15).
| Method | Completed / 12: 8K | 16K | 32K | Delivered: 16K | 32K |
|---|---|---|---|---|---|
| BudgetKV | 12 | 12 | 12 | 55.42 | 51.25 |
| Lexical control | 12 | 12 | 12 | 34.58 | 30.42 |
| StreamingLLM | 12 | 12 | 12 | 16.67 | 33.33 |
| EpiCache block | 12 | 12 | 12 | 25.00 | 25.00 |
| Query-visible SnapKV | 12 | 6 | 0 | 16.67 | 0 |
| Active PyramidKV | 12 | 8 | 6 | 8.33 | 16.67 |
| Expected Attention | 12 | 7 | 0 | 0 | 0 |
| Full KV | 6 | 0 | 0 | 0 | 0 |
BudgetKV, lexical and StreamingLLM complete 12/12 actual-32K requests, but deliver 51.25, 30.42 and 33.33 points. BudgetKV/lexical retrieval is 68.75/6.25, with equal QA/tracking scores: the 20.83-point macro gain comes from relevance selection, beyond making requests fit. Full KV (higher desktop retrieval reference) and normal SnapKV (higher LongBench scores) both complete 0/12 at 32K on Jetson. Thus the edge balance combines executable length and useful answers.
4. Requested low-overlap and broader-task checks. NoLiMa has 174 cases per backbone/condition. The complete paragraph results are:
NoLiMa paragraph results (Appendix G.2, pp. 23–24), organized for inspection.
| Condition | Llama-3.2-1B | Qwen3-1.7B |
|---|---|---|
| BudgetKV, 8K background | 0.00 | 0.00 |
| Lexical, 8K background | 0.00 | 0.00 |
| Full KV, 8K background | 0.00 | 4.02 |
| Query-visible SnapKV, 8K background | 0.00 | 4.02 |
| Full KV, short context | 41.95 | 43.68 |
This test gives no semantic gain; short-context controls clarify the input condition. Q2 analyzes the visible-evidence slice. The same edge policy also runs summarization/code:
Table 14 (p. 17).
| Model | Task | BudgetKV | Lexical |
|---|---|---|---|
| Qwen3-1.7B | QMSum | 12.85 | 12.07 |
| Llama-3.2-1B | QMSum | 11.97 | 10.99 |
| Qwen3-1.7B | RepoBench-P-E | 13.60 | 12.67 |
| Llama-3.2-1B | RepoBench-P-E | 7.33 | 8.35 |
Both policies complete 80/80 requests across Qwen/Llama and 0/1 GiB competing-memory tiers. QMSum uses ROUGE-L; code uses its frozen similarity metric.
Revised manuscript text. Section 4.2, p. 7, states:
“The improved macro score therefore comes from preserving nonlocal retrieval evidence within the same cache size.”
Appendix E/Table 23 (p. 20) preserves all corrected desktop intervals. Section 4.2/Figure 3 and Table 7 (pp. 6 and 15) add the device controls. Appendix G.2 (pp. 23–24) adds NoLiMa and states:
“The visible-evidence slice and short-context controls identify the input conditions of these retrieval tests.”
Section 4.6 (p. 9) and Appendix C.2/Table 14 (p. 17) add the natural-task evidence.
Q1
Reviewer comment
How does BudgetKV compare with query-visible SnapKV and EpiCache when query access, precision, inputs, decoding policy, and peak-memory constraints are matched?
Response
Please accept our sincere apologies that the original protocol did not make equivalent query access, decoding and memory conditions clear. This made a fair comparison difficult. We now identify the matched resources explicitly and provide normal query-visible controls, native-kernel results and a common external-memory load.
Query, precision and quality. The desktop study uses the same 400 LongBench documents, task queries, FP16 KV, 8,192-token input cap and greedy EOS up to 64 outputs. SnapKV sees the query in its normal observation window; BudgetKV supplies its IDs to the scorer. Median prefill KV matches per cell; exact-byte SnapKV pairs retain the ranking. The complete common-decoder scores are:
Table 25 (p. 21).
| Model | BudgetKV | Lexical | SnapKV | EpiCache block | Gate | |
|---|---|---|---|---|---|---|
| Llama-3.2-1B | 512 | 12.34 | 11.59 | 19.73 | 16.69 | -- |
| Llama-3.2-1B | 1024 | 19.88 | 17.62 | 20.31 | 16.97 | -- |
| Qwen3-1.7B | 512 | 30.26 | 27.63 | 41.72 | 30.30 | -- |
| Qwen3-1.7B | 1024 | 36.98 | 30.12 | 46.25 | 35.46 | -- |
| Qwen3-4B | 512 | 32.64 | 27.35 | 51.05 | 23.64 | 33.28 |
| Qwen3-4B | 1024 | 42.72 | 34.65 | 52.69 | 31.55 | 38.40 |
Gate is the released Fast KVzip selector with uniform-head gathering. EpiCache block here is the common-decoder adaptation, separate from its native cache path.
Controlled output work. The two-source, check forces 64 outputs, with three repetitions and actual complete lengths of 10,868/11,539 tokens:
Table 19 (p. 19).
| Method | 8K time (s) | Full-input time (s) |
|---|---|---|
| BudgetKV | 9.16 | 11.27 |
| Lexical control | 8.34 | 10.29 |
| StreamingLLM | 7.80 | 9.55 |
| Query-visible SnapKV | 6.92 | 8.35 |
| Native EpiCache | 64.32 | 86.69 |
BudgetKV, lexical, StreamingLLM and normal SnapKV share a growing-cache decoder. Native EpiCache retains its original kernel on its compatible stack. A separate 20-source Llama-3.2-1B FP16 native study matches post-query KV; at , BudgetKV/EpiCache take 9.16/63.89 seconds for 64 outputs and score 12.93/4.54 in separate natural-EOS runs (Table 16, p. 18).
What memory is matched? Pairs match actual cache tensors after the query is included. A nominal example retains 1,011 tokens, 33,128,448 bytes; controls match that realized footprint. The 20-source native study reports the following GiB medians:
Table 17 (p. 18).
| CUDA allocated: BudgetKV | EpiCache | Process RSS: BudgetKV | EpiCache | |
|---|---|---|---|---|
| 512 | 2.339 | 2.360 | 5.119 | 3.037 |
| 1024 | 2.366 | 2.385 | 5.120 | 3.047 |
BudgetKV's CUDA allocation is similar and process RSS is higher. Thus retained KV is matched, while whole-process peaks are measured outcomes. These are not identical measured-RSS ceilings.
Same device and external pressure. A six-source native study uses 0/1 GiB locked competing memory at 8K and full lengths of 9,331–16,384 tokens, with matched KV and greedy EOS. All 120 requests complete. Its complete 1 GiB results are:
Table 18 (p. 19).
| Method | 8K time (s) | Full-input time (s) | Full-input score |
|---|---|---|---|
| BudgetKV | 6.73 | 10.47 | 16.67 |
| Lexical control | 6.13 | 8.86 | 18.25 |
| StreamingLLM | 6.13 | 8.54 | 5.69 |
| Query-visible SnapKV | 4.61 | 7.19 | 0.88 |
| Native EpiCache | 61.97 | 97.31 | 5.36 |
At full length, BudgetKV delivers 16.67 points in 10.47 seconds, versus native EpiCache 5.36 in 97.31 and normal SnapKV 0.88 in 7.19. The matched conditions are retained KV and external residency; backends and measured peaks remain disclosed. Separately, the actual-32K Jetson matrix gives BudgetKV 12/12 completions, 51.25 points, versus SnapKV 0/12 (Figure 3).
Revised manuscript text. Appendix A, p. 14, states:
“Fixed-work comparisons match retained KV bytes and output count. CUDA allocator statistics and process RSS are measured separately. Synchronized profiles use separate runs.”
Its timing paragraph includes relevance scoring, prefill, generation and EpiCache clustering/routing, starting after model loading. Appendix D/Tables 16–19 (pp. 18–19) supplies the native results, while F.1/Table 25 (p. 21) gives common-decoder quality. Appendix D, p. 19, states:
“All 120 requests in this pressure study match their actual KV targets. ... Equal retained KV and equal external residency are the comparison conditions; RSS is a measured outcome.”
Thank you for recognizing our practical motivation and matched-cache semantic evaluation.
W1
Reviewer comment
Response
We sincerely apologize that the desktop results and edge objective were not clearly distinguished, making the practical advantage difficult to assess. We now explain that balance.
1. Why the desktop result favors query-visible SnapKV. These were RTX 4090 D desktop quality tests. When full-prompt processing fits, normal SnapKV selects using query-visible contextual attention. BudgetKV selects earlier from input embeddings and lexical cues, before those contextual statistics exist. This is a quality/cost trade-off. The revised manuscript reruns the larger comparison with matched query access and common decoding:
Table 25 (p. 21).
This uses 400 LongBench documents, FP16 KV, an 8,192-token cap and greedy EOS up to 64 outputs. Median KV matches per cell; exact-byte SnapKV pairs preserve the ranking. Gate uses released Fast KVzip with uniform-head gathering.
2. Our objective is the edge quality–memory–time balance. The desktop study isolates selection quality when execution is possible. On an 8 GB device, the full prompt can exhaust memory during prefill even if the final compressed cache fits. BudgetKV retains a bounded state after each chunk, seeking useful answers as long inputs pass through a small cache.
3. What changes on Jetson? Qwen3-1.7B FP16, B=1024; each completion column is out of 12, with unsuccessful requests scored zero:
Table 7 (p. 15).
The desktop-higher-quality SnapKV completes 6/12 at 16K and 0/12 at actual 32K here. BudgetKV completes 12/12 throughout, delivering 51.25 points at 32K versus lexical 30.42 and StreamingLLM 33.33, which also finish all requests. This shows both feasibility and better useful retention within the edge budget.
4. Close bounded-prefill and recent baselines. A separate same-Jetson Llama-3.2-1B FP16 study uses 20 paired 8K sources and matched post-query KV. EpiCache runs its original attention/cache kernels in a single-episode document adaptation, including clustering/routing:
Table 16 (p. 18).
All requests complete. Fixed work uses 64 outputs and three trials; natural-EOS quality is measured separately. At B=1024, BudgetKV combines 6.98× speedup with +8.39 quality points. EpiCache block in Table 7 is a separate adaptation.
The native Fast KVzip study uses RTX 4090D, Qwen3-4B BF16 and 20 paired 8K sources; all 40 pairs meet 0.1% measured-byte tolerance. Appendix F.2 (p. 21) reports:
Revised manuscript text. Appendix F.1, p. 21, states:
Section 4.2/Figure 3 (p. 6), Table 7 (p. 15), Section 4.5/Figure 5 (pp. 8–9) and Appendix D/Table 16 (p. 18) give the device evidence. Section 4.2, p. 7, explains: