acceptodds
Under review as a conference paper at ICLR 2027

BudgetKV: Pre-Attention KV Retention with Resident Input Embeddings

27 views

Abstract

Long-context inference on memory-constrained devices must preserve nonlocal evidence while the key–value (KV) cache grows throughout prompt prefill. A request can therefore exceed available memory even when the model weights and intended final cache fit. Block-wise prefill can bound persistent state, but retention policies may still require contextual attention profiles, patched-prompt forwards, or auxiliary semantic encoders before deciding what matters. We introduce BudgetKV, a training-free method for requests whose task query is known before shared-context prefill. BudgetKV scores prompt tokens against the target model's resident input embeddings before contextual attention statistics are available, then refines a frozen lexical anchor through guarded equal-size block exchanges. This reuse of model parameters provides a query-conditioned pre-attention signal without another Transformer forward or learned encoder; under protected-block feasibility, persistent-KV token residency is at most B after selection and B+p between selections. On a frozen 400-document LongBench study, resident-embedding refinement yields a statistically supported +6.81-point gain for Qwen3-1.7B at B=1024, while the other tested model–budget cells remain inconclusive after adjustment. At 32K, the largest observed advantage is on needle retrieval rather than reasoning. On an 8GB Jetson, BudgetKV completes all six tested 8K requests at both budgets, whereas Full KV completes two; relative to successful Full-KV runs, persistent-KV footprint falls by 85.9–92.2%. These results support resident input embeddings as a useful pre-attention signal when contextual profiling is constrained by prefill memory, but benefits remain model- and task-dependent and scheduling adds overhead.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache compression, query-conditioned retention, pre-attention scoring, resident input embeddings, long-context inference, bounded prefill

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.60 over 5 reviews, above 10% of the others.
Reviewer_V3ZoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BudgetKV reduces KV-cache memory during chunked prefill when the task query is known in advance. It combines a fixed lexical policy with scores from the model’s frozen input embeddings, then exchanges equally sized token blocks.

Under protected-block feasibility, persistent KV residency is bounded by B after selection and B+p between selections. This bound excludes temporary allocations and does not automatically extend to decoding.

Experiments cover two small models, five LongBench tasks, adapted RULER diagnostics through 32K, and an 8GB Jetson. Semantic refinement significantly improves one of four primary model–budget settings. The clearest benefit is retrieval; reasoning improvements are inconsistent.

Critical strengths
  1. Practical motivation BudgetKV addresses requests that may fail during prefill even when their final compressed cache would fit. Reusing resident embeddings avoids an additional encoder or profiling pass.

  2. Controlled and transparent evaluation The semantic ablation matches realized final-prefill counts. Qwen3-1.7B at (B=1024) gains 6.81 points. The authors appropriately acknowledge inconclusive results elsewhere.

Critical weaknesses
  1. Competitiveness remains unresolved Query-visible SnapKV is stronger in all four controlled diagnostic settings (Figure 9). The larger favorable comparison uses query-hidden SnapKV and different decoding policies. EpiCache, a close bounded-prefill method that also studies target-model embeddings, is not experimentally compared. Recent lightweight selectors such as Fast KVzip are also absent. These gaps leave the advantage under the intended memory constraint unclear.

  2. Evidence for accuracy preservation is narrow Only one of four primary semantic comparisons is significant after correction. At 32K, retrieval remains below Full KV, while reasoning gains are inconsistent. The five-task LongBench subset and three adapted RULER tasks do not establish broad long-context performance. NoLiMa would be particularly relevant because it tests retrieval with little lexical overlap. The Jetson capacity experiment also omits lexical-only retention and StreamingLLM, so it does not isolate the deployment benefit of the semantic component.

Questions for authors
  1. How does BudgetKV compare with query-visible SnapKV and EpiCache when query access, precision, inputs, decoding policy, and peak-memory constraints are matched?

  2. Does the semantic gain persist on low-overlap query–evidence pairs? Please provide per-task results and evidence-retention measurements to clarify where it helps.

  3. Do lexical-only retention and StreamingLLM also complete the six 8K Jetson requests? What accuracy do all methods achieve on those same requests?

  4. How were the lexical rules, margins, and swap cap selected? Does the main improvement survive reasonable changes, including removal of the literal “KV” bonus?

8 comments

Thank you for recognizing our practical motivation and matched-cache semantic evaluation.

W1

Reviewer comment

Competitiveness remains unresolved.

Response

We sincerely apologize that the desktop results and edge objective were not clearly distinguished, making the practical advantage difficult to assess. We now explain that balance.

1. Why the desktop result favors query-visible SnapKV. These were RTX 4090 D desktop quality tests. When full-prompt processing fits, normal SnapKV selects using query-visible contextual attention. BudgetKV selects earlier from input embeddings and lexical cues, before those contextual statistics exist. This is a quality/cost trade-off. The revised manuscript reruns the larger comparison with matched query access and common decoding:

Table 25 (p. 21).

ModelBBBudgetKVLexicalSnapKVEpiCache blockFast KVzip (Gate)
Llama-3.2-1B51212.3411.5919.7316.69--
Llama-3.2-1B102419.8817.6220.3116.97--
Qwen3-1.7B51230.2627.6341.7230.30--
Qwen3-1.7B102436.9830.1246.2535.46--
Qwen3-4B51232.6427.3551.0523.6433.28
Qwen3-4B102442.7234.6552.6931.5538.40

This uses 400 LongBench documents, FP16 KV, an 8,192-token cap and greedy EOS up to 64 outputs. Median KV matches per cell; exact-byte SnapKV pairs preserve the ranking. Gate uses released Fast KVzip with uniform-head gathering.

2. Our objective is the edge quality–memory–time balance. The desktop study isolates selection quality when execution is possible. On an 8 GB device, the full prompt can exhaust memory during prefill even if the final compressed cache fits. BudgetKV retains a bounded state after each chunk, seeking useful answers as long inputs pass through a small cache.

3. What changes on Jetson? Qwen3-1.7B FP16, B=1024B=1024; each completion column is out of 12, with unsuccessful requests scored zero:

Table 7 (p. 15).

MethodCompleted / 12: 8K16K32KDelivered: 16K32K
BudgetKV12121255.4251.25
Lexical control12121234.5830.42
StreamingLLM12121216.6733.33
EpiCache block12121225.0025.00
Query-visible SnapKV126016.670
Active PyramidKV12868.3316.67
Expected Attention127000
Full KV60000

The desktop-higher-quality SnapKV completes 6/12 at 16K and 0/12 at actual 32K here. BudgetKV completes 12/12 throughout, delivering 51.25 points at 32K versus lexical 30.42 and StreamingLLM 33.33, which also finish all requests. This shows both feasibility and better useful retention within the edge budget.

4. Close bounded-prefill and recent baselines. A separate same-Jetson Llama-3.2-1B FP16 study uses 20 paired 8K sources and matched post-query KV. EpiCache runs its original attention/cache kernels in a single-episode document adaptation, including clustering/routing:

Table 16 (p. 18).

BBFixed 64 outputs: BudgetKV (s)EpiCache (s)SpeedupNatural-EOS score: BudgetKVEpiCache
5128.8558.166.57$\times$8.128.96
10249.1663.896.98$\times$12.934.54

All requests complete. Fixed work uses 64 outputs and three trials; natural-EOS quality is measured separately. At B=1024B=1024, BudgetKV combines 6.98× speedup with +8.39 quality points. EpiCache block in Table 7 is a separate adaptation.

The native Fast KVzip study uses RTX 4090D, Qwen3-4B BF16 and 20 paired 8K sources; all 40 pairs meet 0.1% measured-byte tolerance. Appendix F.2 (p. 21) reports:

BBBudgetKVNative Fast KVzip
51233.171.20
102432.8010.89

Revised manuscript text. Appendix F.1, p. 21, states:

“Table 25 gives every method the same task query. SnapKV includes it in its normal observation window; BudgetKV supplies its IDs to the scorer. Median prefill KV bytes match within each cell.”

Section 4.2/Figure 3 (p. 6), Table 7 (p. 15), Section 4.5/Figure 5 (pp. 8–9) and Appendix D/Table 16 (p. 18) give the device evidence. Section 4.2, p. 7, explains:

“Chunk-wise retention makes long requests executable, while semantic refinement improves retrieval quality within that constraint. The lexical and StreamingLLM controls also complete all inputs, so the quality comparison separates relevance selection from completion alone.”

W2

Reviewer comment

Evidence for accuracy preservation is narrow.

Response

We are very sorry that the original discussion did not connect desktop accuracy to edge deployment, making the semantic contribution difficult to judge. We explain those outcomes first, then test the edge balance with the requested controls.

1. What the earlier desktop results mean. The four semantic contrasts and 32K quality diagnostic ran on an RTX 4090 D. The 400-document LongBench confirmation uses matched inputs/realized KV and task-stratified paired bootstrap intervals:

Table 23 (p. 20).

ModelBBGain95% CIFamily-adjusted CI
Qwen3-1.7B5122.64[−1.08-1.08, 6.42][−2.13-2.13, 7.52]
Qwen3-1.7B10246.81[3.31, 10.41][2.37, 11.45]
Llama-3.2-1B5120.67[−1.26-1.26, 2.73][−1.74-1.74, 3.32]
Llama-3.2-1B10242.15[−0.22-0.22, 4.62][−0.91-0.91, 5.28]

The early, non-contextual relevance signal helps select distant evidence, but its benefit depends on model, budget and task. Qwen/B=1024B=1024 gains 6.81 points, adjusted CI [2.37,11.45]; the other intervals include zero. Full KV preserves the entire context when desktop memory permits: at 32K, BudgetKV/Full-KV retrieval is 66.25/80.00 on Qwen and 41.25/76.25 on Llama. Compressing that evidence does not ensure equal QA/tracking quality, whose complete outcomes remain in Table 26.

2. The paper's deployment objective. We seek useful answers under an 8 GB edge device's memory and execution costs. Desktop quality is a supplementary selection test; on-device value requires both completing the long prompt and retaining evidence for its answer. Chunked retention addresses prefill residency, while semantic refinement improves the choice within that retained budget.

3. Do lexical retention and StreamingLLM also fit? Yes. The complete Jetson comparison includes both:

Table 7 (p. 15).

MethodCompleted / 12: 8K16K32KDelivered: 16K32K
BudgetKV12121255.4251.25
Lexical control12121234.5830.42
StreamingLLM12121216.6733.33
EpiCache block12121225.0025.00
Query-visible SnapKV126016.670
Active PyramidKV12868.3316.67
Expected Attention127000
Full KV60000

BudgetKV, lexical and StreamingLLM complete 12/12 actual-32K requests, but deliver 51.25, 30.42 and 33.33 points. BudgetKV/lexical retrieval is 68.75/6.25, with equal QA/tracking scores: the 20.83-point macro gain comes from relevance selection, beyond making requests fit. Full KV (higher desktop retrieval reference) and normal SnapKV (higher LongBench scores) both complete 0/12 at 32K on Jetson. Thus the edge balance combines executable length and useful answers.

4. Requested low-overlap and broader-task checks. NoLiMa has 174 cases per backbone/condition. The complete paragraph results are:

NoLiMa paragraph results (Appendix G.2, pp. 23–24), organized for inspection.

ConditionLlama-3.2-1BQwen3-1.7B
BudgetKV, 8K background0.000.00
Lexical, 8K background0.000.00
Full KV, 8K background0.004.02
Query-visible SnapKV, 8K background0.004.02
Full KV, short context41.9543.68

This test gives no semantic gain; short-context controls clarify the input condition. Q2 analyzes the visible-evidence slice. The same edge policy also runs summarization/code:

Table 14 (p. 17).

ModelTaskBudgetKVLexical
Qwen3-1.7BQMSum12.8512.07
Llama-3.2-1BQMSum11.9710.99
Qwen3-1.7BRepoBench-P-E13.6012.67
Llama-3.2-1BRepoBench-P-E7.338.35

Both policies complete 80/80 requests across Qwen/Llama and 0/1 GiB competing-memory tiers. QMSum uses ROUGE-L; code uses its frozen similarity metric.

Revised manuscript text. Section 4.2, p. 7, states:

“The improved macro score therefore comes from preserving nonlocal retrieval evidence within the same cache size.”

Appendix E/Table 23 (p. 20) preserves all corrected desktop intervals. Section 4.2/Figure 3 and Table 7 (pp. 6 and 15) add the device controls. Appendix G.2 (pp. 23–24) adds NoLiMa and states:

“The visible-evidence slice and short-context controls identify the input conditions of these retrieval tests.”

Section 4.6 (p. 9) and Appendix C.2/Table 14 (p. 17) add the natural-task evidence.

Q1

Reviewer comment

How does BudgetKV compare with query-visible SnapKV and EpiCache when query access, precision, inputs, decoding policy, and peak-memory constraints are matched?

Response

Please accept our sincere apologies that the original protocol did not make equivalent query access, decoding and memory conditions clear. This made a fair comparison difficult. We now identify the matched resources explicitly and provide normal query-visible controls, native-kernel results and a common external-memory load.

Query, precision and quality. The desktop study uses the same 400 LongBench documents, task queries, FP16 KV, 8,192-token input cap and greedy EOS up to 64 outputs. SnapKV sees the query in its normal observation window; BudgetKV supplies its IDs to the scorer. Median prefill KV matches per cell; exact-byte SnapKV pairs retain the ranking. The complete common-decoder scores are:

Table 25 (p. 21).

ModelBBBudgetKVLexicalSnapKVEpiCache blockGate
Llama-3.2-1B51212.3411.5919.7316.69--
Llama-3.2-1B102419.8817.6220.3116.97--
Qwen3-1.7B51230.2627.6341.7230.30--
Qwen3-1.7B102436.9830.1246.2535.46--
Qwen3-4B51232.6427.3551.0523.6433.28
Qwen3-4B102442.7234.6552.6931.5538.40

Gate is the released Fast KVzip selector with uniform-head gathering. EpiCache block here is the common-decoder adaptation, separate from its native cache path.

Controlled output work. The two-source, B=1024B=1024 check forces 64 outputs, with three repetitions and actual complete lengths of 10,868/11,539 tokens:

Table 19 (p. 19).

Method8K time (s)Full-input time (s)
BudgetKV9.1611.27
Lexical control8.3410.29
StreamingLLM7.809.55
Query-visible SnapKV6.928.35
Native EpiCache64.3286.69

BudgetKV, lexical, StreamingLLM and normal SnapKV share a growing-cache decoder. Native EpiCache retains its original kernel on its compatible stack. A separate 20-source Llama-3.2-1B FP16 native study matches post-query KV; at B=1024B=1024, BudgetKV/EpiCache take 9.16/63.89 seconds for 64 outputs and score 12.93/4.54 in separate natural-EOS runs (Table 16, p. 18).

What memory is matched? Pairs match actual cache tensors after the query is included. A nominal B=1024B=1024 example retains 1,011 tokens, 33,128,448 bytes; controls match that realized footprint. The 20-source native study reports the following GiB medians:

Table 17 (p. 18).

BBCUDA allocated: BudgetKVEpiCacheProcess RSS: BudgetKVEpiCache
5122.3392.3605.1193.037
10242.3662.3855.1203.047

BudgetKV's CUDA allocation is similar and process RSS is higher. Thus retained KV is matched, while whole-process peaks are measured outcomes. These are not identical measured-RSS ceilings.

Same device and external pressure. A six-source native study uses 0/1 GiB locked competing memory at 8K and full lengths of 9,331–16,384 tokens, with matched KV and greedy EOS. All 120 requests complete. Its complete 1 GiB results are:

Table 18 (p. 19).

Method8K time (s)Full-input time (s)Full-input score
BudgetKV6.7310.4716.67
Lexical control6.138.8618.25
StreamingLLM6.138.545.69
Query-visible SnapKV4.617.190.88
Native EpiCache61.9797.315.36

At full length, BudgetKV delivers 16.67 points in 10.47 seconds, versus native EpiCache 5.36 in 97.31 and normal SnapKV 0.88 in 7.19. The matched conditions are retained KV and external residency; backends and measured peaks remain disclosed. Separately, the actual-32K Jetson matrix gives BudgetKV 12/12 completions, 51.25 points, versus SnapKV 0/12 (Figure 3).

Revised manuscript text. Appendix A, p. 14, states:

“Fixed-work comparisons match retained KV bytes and output count. CUDA allocator statistics and process RSS are measured separately. Synchronized profiles use separate runs.”

Its timing paragraph includes relevance scoring, prefill, generation and EpiCache clustering/routing, starting after model loading. Appendix D/Tables 16–19 (pp. 18–19) supplies the native results, while F.1/Table 25 (p. 21) gives common-decoder quality. Appendix D, p. 19, states:

“All 120 requests in this pressure study match their actual KV targets. ... Equal retained KV and equal external residency are the comparison conditions; RSS is a measured outcome.”

Reviewer_xAXgOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces BudgetKV, a training-free KV-cache retention method for memory-constrained long-context inference when the task query is available before prefill. It scores tokens using the model's resident input embeddings, combines semantic relevance with a lexical anchor through guarded block exchanges, and performs chunk-wise eviction with a conditional persistent-KV budget guarantee. Evaluation on Qwen3-1.7B and Llama-3.2-1B covers LongBench, retrieval and reasoning diagnostics up to 32K tokens, and an 8GB Jetson device. Results show a statistically significant semantic-scoring benefit in one model-budget setting, improved needle retrieval, and greater request completion under memory constraints, with additional runtime overhead.

Critical strengths
  1. Lightweight and Training-Free Query-Aware KV Cache Selection. BudgetKV introduces a practical approach to estimating token relevance before attention computation by reusing the LLM's existing input embeddings, without requiring additional Transformer forward passes or auxiliary semantic encoders. The method demonstrates a statistically significant 6.81-point improvement over lexical-only retention on Qwen3-1.7B at a 1024-token budget, along with substantial improvements in long-context needle retrieval. Although these gains are not consistent across all experimental settings, the approach offers a potentially useful quality–memory trade-off for resource-constrained inference.

  2. Practical Memory Efficiency with Real-Device Validation. The paper provides a clearly defined conditional KV-cache residency bound during chunk-wise prefill and validates the approach on an 8GB Jetson device. At 8K context length, BudgetKV successfully completes all six tested requests under both cache budgets, whereas Full-KV inference completes only two. The method also achieves an 85.9–92.2% reduction in persistent-KV footprint relative to successful Full-KV runs. These results demonstrate practical benefits for memory-constrained deployment, although the additional scheduling overhead and limited evaluation scale remain concerns.

Critical weaknesses
  1. Limited Methodological Novelty and Inconsistent Empirical Improvements. The proposed BudgetKV framework primarily combines existing techniques, including embedding-based semantic similarity, heuristic lexical scoring, and chunk-wise KV cache eviction. Query-aware KV selection and bounded prefill have already been investigated in prior work, such as SnapKV, EpiCache, and IntentKV. The main methodological contribution appears to be reusing the LLM's input embeddings as a lightweight pre-attention relevance signal, rather than introducing a fundamentally new selection mechanism. More importantly, the effectiveness of this component is insufficiently demonstrated. In the controlled LongBench evaluation (Section 4.2), statistically significant improvements over the lexical-only baseline are observed in only one of four model-budget configurations (Qwen3-1.7B, B = 1024). The other three configurations show no statistically significant improvements after multiple-comparison correction. Furthermore, the secondary common-decoder evaluation (Appendix Figure 9) shows that query-visible SnapKV outperforms BudgetKV across all four tested configurations, although the compared observation windows differ. These results raise concerns about the methodological novelty, effectiveness, and generalizability of the proposed semantic selection mechanism.

  2. Limited Practical Efficiency and Insufficient Experimental Validation. Although BudgetKV substantially reduces persistent KV-cache memory consumption, its practical efficiency remains questionable. The proposed method introduces additional computational overhead from repeated pooling, block selection, and KV gathering. As reported in the Jetson experiments (Section 4.4), BudgetKV increases request latency by approximately 1.75 seconds compared with StreamingLLM at B = 1024, although the implementations use different decoding policies. Moreover, the experimental evaluation is limited to two relatively small LLMs (Qwen3-1.7B and Llama-3.2-1B), and the primary device-capacity results are based on only six 8K requests on a single 8GB Jetson device. The method also assumes that the task query is known before prefill and remains fixed, limiting its applicability to dynamic multi-turn conversations. Finally, the theoretical memory guarantee is conditional and only bounds persistent KV residency, rather than total peak memory or decoding-stage memory usage. These limitations make it difficult to establish whether BudgetKV provides a sufficiently general and practically efficient solution for real-world long-context LLM inference.

Questions for authors

1. Inconsistent Effectiveness of the Proposed Semantic Scoring. In Section 4.2, the proposed semantic refinement achieves a statistically significant improvement over the lexical-only baseline in only one of the four model-budget configurations (Qwen3-1.7B, B = 1024). The remaining three configurations show statistically inconclusive improvements after multiple-comparison correction. What explains this inconsistency, particularly the limited improvements on Llama-3.2-1B? Can the authors provide additional analyses identifying the model characteristics, cache budgets, or task conditions under which the proposed semantic scoring is reliably beneficial?

2. Fair Comparison with Existing Query-Aware KV Compression Methods. The secondary evaluation in Appendix Figure 9 shows that query-visible SnapKV outperforms BudgetKV across all four tested model-budget configurations under a common decoder and matched prefill KV bytes. Although query visibility and observation-window length differ between the compared procedures, these results raise concerns about the effectiveness of the proposed selection mechanism. Could the authors provide more controlled comparisons in which competing methods receive equivalent query information, while accounting for the additional computation and peak memory required to obtain their selection signals?

3. Practical Efficiency and Scalability. Although BudgetKV reduces persistent KV-cache memory usage, its repeated pooling, block selection, and KV gathering introduce non-negligible computational overhead. For example, the Jetson experiments report approximately 1.75 seconds of additional request latency compared with StreamingLLM at B = 1024, although the implementations use different decoding policies. Can the authors provide end-to-end latency and peak-memory comparisons under a consistent decoding implementation? Furthermore, the evaluation only covers two relatively small models and a single 8GB Jetson device, with the primary capacity results based on six 8K requests. Is there evidence that the memory-quality trade-off remains favorable for larger LLMs and longer contexts on memory-constrained hardware?

7 comments

Thank you for your careful review and recognition of two strengths: the lightweight, training-free signal and its controlled 6.81-point gain; and the conditional persistent-KV bound and its Jetson validation. We sincerely appreciate your detailed feedback. It prompted a clearer edge-device focus and substantial new controlled and deployment experiments.

W1

Reviewer comment

Limited methodological novelty and inconsistent empirical improvements.

Response

We sincerely apologize that the original manuscript did not clearly separate desktop quality from edge deployment, obscuring both our contribution and the varying gains. The revised manuscript focuses on quality, memory and time under edge constraints.

A. Methodological novelty. BudgetKV combines two levels: a signal from the target model's resident input embeddings, giving query-conditioned relevance before contextual attention; and a retention policy integrating that signal into bounded chunk-wise prefill:

MethodSelection signalAvailability
SnapKVContextual attentionAfter observation window
EpiCacheProbed attention; semantic organizationAfter block-specific probe
IntentKVInferred-intention attentionThrough contextual attention
BudgetKVInput embeddings; lexical anchorBefore attention profiling

No auxiliary encoder, learned selector or extra Transformer forward is needed for the signal. The novelty is not embedding similarity alone, but its integration into an online, chunk-wise retention policy that operates before contextual attention becomes available. Guarded equal-size block exchanges refine the lexical anchor without increasing retained tokens, preserving sink/recent blocks. Under protected-block feasibility, persistent KV stays within BB after selection and B+pB+p between selections (Sections 2–3, pp. 2–5).

B. Why gains vary. The original 400-document LongBench confirmation measures desktop quality:

Table 23 (p. 21).

ModelBBGain95% CIFamily-adjusted CI
Qwen3-1.7B5122.64[−1.08-1.08, 6.42][−2.13-2.13, 7.52]
Qwen3-1.7B10246.81[3.31, 10.41][2.37, 11.45]
Llama-3.2-1B5120.67[−1.26-1.26, 2.73][−1.74-1.74, 3.32]
Llama-3.2-1B10242.15[−0.22-0.22, 4.62][−0.91-0.91, 5.28]

Qwen/B=1024B=1024 gains 6.81 points, adjusted CI [2.37,11.45]; other intervals include zero. Diagnostics distinguish retaining evidence from recovering answers: on the low-overlap slice, Llama retains 45.74% versus lexical 31.25% of gold tokens with unchanged answer score; Qwen3-1.7B/4B retain more evidence and gain 45.45/36.36 answer points. Better retention thus translates differently into answers across models (Table 28, p. 25; detailed analyses in Q1).

Query-visible SnapKV scores 46.25 versus BudgetKV 36.98 on desktop Qwen/B=1024B=1024, using contextual attention after prompt processing. BudgetKV selects earlier to balance quality, memory and time on edge devices. New Jetson results show:

Table 7 (p. 16).

MethodCompleted / 12: 8K16K32KDelivered: 16K32K
BudgetKV12121255.4251.25
Lexical control12121234.5830.42
StreamingLLM12121216.6733.33
EpiCache block12121225.0025.00
Query-visible SnapKV126016.670.00
Active PyramidKV12868.3316.67
Expected Attention12700.000.00
Full KV6000.000.00

With Qwen3-1.7B FP16/B=1024B=1024 on the 8 GB Jetson, failed requests score zero. At actual 32K, BudgetKV delivers 51.25 versus lexical 30.42 and StreamingLLM 33.33, all completing 12/12. Retrieval is 68.75 versus lexical 6.25; QA/tracking tie. SnapKV and Full KV complete 0/12 at 32K. This shows quality gains among completing policies and longer feasible inputs relative to full-prompt paths.

Revised manuscript text. Introduction, p. 2, states:

“We introduce a query-conditioned retention signal available before contextual attention statistics, obtained directly from the target language model's resident input embeddings.”

Section 4.2, p. 7, states:

“Chunk-wise retention makes long requests executable, while semantic refinement improves retrieval quality within that constraint. The lexical and StreamingLLM controls also complete all inputs, so the quality comparison separates relevance selection from completion alone.”

Table 23 (p. 21), Table 28 (p. 25) and Figure 3/Table 7 (pp. 6 and 16) support this response.

W2

Reviewer comment

Limited practical efficiency and insufficient experimental validation.

Response

We are very sorry that unclear experiment roles and decoding conditions made efficiency and validation difficult to assess. The revised manuscript separates desktop quality from edge deployment. We address cost and validation below.

A. Practical efficiency: what is the cost and what does it buy? The original +1.75 seconds versus StreamingLLM at B=1024B=1024 is a Jetson systems measurement; pooling, selection and gathering add work. The revised fixed-work check uses two paired sources, three repetitions, matched KV and 64 forced outputs. Complete inputs contain 10,868/11,539 tokens:

Table 19 (p. 20).

Method8K time (s)Full-input time (s)
BudgetKV9.1611.27
Lexical control8.3410.29
StreamingLLM7.809.55
Query-visible SnapKV6.928.35
Native EpiCache64.3286.69

The first four policies share a growing-cache decoder; EpiCache keeps its original kernel. BudgetKV adds 0.82/0.98 seconds over lexical retention, about 10%; versus StreamingLLM, the gaps are 1.36/1.72 seconds. It trades selection cost for retained relevance; the separate native EpiCache study gives 6.57–6.98× fixed-work speedup (Tables 16–17).

B. Experimental validation: what has been added?

Long inputs and competing memory. On the 8 GB Jetson, BudgetKV completes 12/12 at each actual 8K/16K/32K length; normal SnapKV completes 12/6/0 and Full KV 6/0/0. At 32K, BudgetKV delivers 51.25 versus lexical 30.42 and StreamingLLM 33.33, both fully completing. Six natural 10,070–16,703-token questions additionally give 6/6 and 43.5 F1 under both zero and 1 GiB locked competing memory (Figures 3–4).

The expanded task/model completion table is:

Table 5 (p. 10).

MethodSummary/code: QwenLlama4B: completedDelivered F1
BudgetKV40/4040/4012/1233.33
Lexical control40/4040/4012/1233.33
StreamingLLM40/40---12/120.00
Query-visible SnapKV19/40---3/1216.67
Full KV------0/120.00

Summary/code covers ten sources per task/model and two memory levels: 80/80 BudgetKV requests across Qwen/Llama. Qwen3-4B uses INT8 weights, FP16 KV, three sources, 8K/full-input pairs and 0/1 GiB load. Full lengths are 11,231, 16,802 and 16,897 tokens; all six loaded requests complete (Table 15).

Multi-turn validation. The new Jetson study uses eight SCBench-derived conversations, five turns each, with actual 16K/32K initial inputs and Qwen3-1.7B FP16/B=1024B=1024:

Table 4 (p. 8).

Method16K completed / 8Score32K completed / 8Score
BudgetKV8/86.458/87.96
Lexical control8/89.208/86.97
StreamingLLM8/87.648/86.74
Query-visible SnapKV6/87.080/80.00
Full KV0/80.000/80.00

At 32K, BudgetKV completes 8/8 sessions, delivers 7.96 versus lexical 6.97 and StreamingLLM 6.74, and keeps at most 112 MiB KV after each turn. Context is prefilled once; later questions/answers reuse the cache with the fixed initial signal. Greedy EOS allows 64 outputs/turn; the 180-second session limit includes setup. This establishes the measured five-turn capability without changing the selector's query contract.

Query and memory scope. The guarantee bounds persistent prefill KV: at most BB after selection and B+pB+p after the next chunk, under protected-block feasibility. Lower CUDA reservation is a measured result for specific controls (Table 21), not a whole-process memory guarantee. CUDA allocated/reserved peaks and RSS are reported separately; generated KV follows the decoder. The query is available before prefill; evicted KV cannot be recovered for later questions.

Revised manuscript text. Section 4.4/Table 4 (p. 8) and Appendix I (p. 26) add dialogue. Appendix I states:

“Each context is prefilled once; subsequent questions and generated answers reuse the cache with a fixed initial query signal.”

Sections 4.2–4.6 (pp. 6–9), Appendix C/Tables 14–15 (pp. 18–19) and D/Tables 16–20 (pp. 19–21) add length, task, 4B, cost and memory evidence. Section 5, p. 10, states:

“These measurements establish the edge-inference benefit; the formal bound characterizes persistent KV during prefill.”

The device evidence concerns the tested 8 GB Jetson and declared workloads.

Q1

Response

Please accept our sincere apologies that mixed desktop/edge evidence obscured model, budget and task differences. We now separate retention, answer recovery and the conditions with measured gains.

A. What varies with model and budget? The desktop confirmation uses 400 documents, five tasks, matched inputs/KV and 20,000 task-stratified paired bootstrap draws with family correction:

Table 23 (p. 21).

ModelBBGain95% CIFamily-adjusted CI
Qwen3-1.7B5122.64[−1.08-1.08, 6.42][−2.13-2.13, 7.52]
Qwen3-1.7B10246.81[3.31, 10.41][2.37, 11.45]
Llama-3.2-1B5120.67[−1.26-1.26, 2.73][−1.74-1.74, 3.32]
Llama-3.2-1B10242.15[−0.22-0.22, 4.62][−0.91-0.91, 5.28]

Gains grow from B=512B=512 to 1024: Qwen 2.64→6.81, Llama 0.67→2.15. Qwen/B=1024B=1024 has the positive adjusted interval.

B. Why is Llama's answer gain smaller? Eleven passage-retrieval documents have visible gold evidence and query-to-gold Jaccard at most 0.30, at B=1024B=1024:

Table 28 (p. 25).

BackboneBudgetKV retentionLexical retentionAnswer-score change
Llama-3.2-1B45.74%31.25%0
Qwen3-1.7B37.47%20.65%+45.45+45.45
Qwen3-4B39.48%21.85%+36.36+36.36

Llama retains 45.74% versus lexical 31.25% but gains no answer points. Qwen3-1.7B/4B retain more evidence and gain 45.45/36.36 points. Retention alone therefore does not explain answer quality; 540 replays check outputs/positions.

Separating evidence availability from recovery. The original 16K/32K decline cases are replayed with four complete entity–value sentences retained as their tokens arrive:

Table 27 (p. 24).

ModelInputBudgetKVComplete evidencePlaceboShort Full KV
Qwen16K1001000100
Qwen32K43.7587.50100
Llama16K7581.250100
Llama32K6.25256100

Cells average four original seeds; all 16 intervention pairs match final prefill KV bytes and replays reproduce outputs/positions. At 32K, complete evidence raises Qwen from 43.75 to 87.50, and Llama from 6.25 to 25.00. Yet Llama's short Full KV control scores 100.00 with the same evidence. Thus the gap involves using evidence within a longer retained context, beyond whether it is preserved. Targets are evaluator-supplied. This diagnoses recovery, not deployable policy quality. Placebo shares the budget ceiling; eight of 16 inputs match bytes.

C. Does the selection policy affect Llama on-device? Thirty development sources at B=1024B=1024 cover ten settings, with 300/300 Jetson requests completed:

Table 31 (p. 26).

SettingScoreSettingScore
Original31.98Lexical control27.26
α=0.25\alpha=0.2533.05α=0.75\alpha=0.7534.92
ε=0\varepsilon=027.26ε=0.30\varepsilon=0.3034.37
δ=0\delta=034.37δ=0.10\delta=0.1035.56
Per-call cap 131.92Per-call cap 431.98

The original policy gains 4.72 points over lexical. With ε=0\varepsilon=0, all 30 outputs match lexical. Changing the per-call cap from two to one/four changes the score by at most 0.06.

D. Under which tested tasks is it beneficial? At desktop 32K/B=1024B=1024, BudgetKV/lexical scores are: Qwen retrieval 66.25/10.00, QA 30.00/25.00, tracking 6.00/6.00; Llama retrieval 41.25/2.50, QA 25.00/30.00, tracking 0.00/0.00 (Table 26, p. 24). Gains concentrate on retrieval, rather than applying uniformly across tasks. On Jetson at actual 32K, macro score rises 30.42→51.25, with both policies completing all requests: retrieval 6.25→68.75, QA 75.00/75.00, tracking 10.00/10.00 (Table 8, p. 16).

Revised manuscript text. Appendix G.1, p. 24, states:

“The 32K intervention raises recall on both models, showing that maintaining complete evidence changes outcomes under the same realized budget. The short control achieves 100 throughout. The short and long controls distinguish evidence availability from answer recovery in a long retained context.”

Appendix H.1, p. 26, states:

“With ε=0\varepsilon=0, all 30 generated outputs match the lexical condition, showing how the lexical guard regulates semantic admissions in this cohort.”

Supporting locations: Appendix E/Table 23 (p. 21), G/Tables 27–28 (pp. 24–25), H.1/Table 31 (p. 26) and Section 4.7 (p. 10).

Reviewer_4pXYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BudgetKV, a training-free KV cache compression method targeting long-context inference on memory-constrained devices, particularly during the prefill stage.

The key idea is to exploit the target model's resident input embedding table to estimate the relevance of prompt tokens to a given task query before contextual attention statistics become available. Specifically, BudgetKV combines query-conditioned embedding similarity with a heuristic lexical importance score to construct a retention policy. It then performs chunk-wise, guarded equal-size block exchanges while protecting sink and recent tokens, aiming to maintain a bounded persistent KV cache throughout prefill.

The method provides a conditional KV residency bound of B tokens after each selection and B+p tokens between selections, where p is the prefill chunk size, assuming protected-block feasibility.

Experiments on Qwen3-1.7B and Llama-3.2-1B-Instruct demonstrate improvements in selected long-context retrieval settings and better memory feasibility on an 8GB Jetson device. However, the benefits of semantic refinement are model- and budget-dependent, and the additional scheduling and KV movement operations introduce non-negligible overhead.

Critical strengths
  1. Practical problem formulation. The paper addresses an important but relatively underexplored aspect of KV cache compression: the memory required during prefill, rather than only the final KV cache size. Estimating token relevance before contextual attention statistics are available is a meaningful direction for memory-constrained inference.

  2. Efficient use of existing model representations. BudgetKV reuses the target model's input embedding table instead of introducing an additional semantic encoder or performing another Transformer forward pass. This provides a lightweight query-conditioned signal without additional learned parameters.

  3. Explicit resource guarantee and deployment evaluation. The authors provide a conditional persistent-KV residency bound and validate the method on an actual Jetson device. The results demonstrate that bounded KV residency can improve the feasibility of processing long prompts under tight memory constraints.

Critical weaknesses
  1. Limited generalization of the proposed semantic signal.

The effectiveness of resident-embedding refinement is not consistent across the evaluated models and budgets. In the controlled 400-document LongBench evaluation, only the Qwen3-1.7B model with a 1024-token budget achieves a statistically significant improvement over the lexical-only baseline after multiple-comparison adjustment. The other three configurations remain inconclusive.

Furthermore, the improvements at 32K context length are primarily concentrated in needle retrieval, while the gains on question answering and variable tracking are limited or inconsistent. This raises questions about whether non-contextual input embeddings can provide sufficiently reliable relevance estimates for more complex reasoning tasks.

  1. The method relies on multiple heuristic components and hyperparameters.

Although the central idea of using input embedding similarity is relatively simple, the complete method introduces several additional design choices, including handcrafted lexical features, score normalization, a fixed semantic mixing coefficient, lexical safety margins, utility improvement thresholds, and limits on the number of exchanges.

These components make the selection procedure more involved, while their individual sensitivity and transferability to different model scales have not been established. In particular, the paper does not convincingly demonstrate that the lexical guard and the various exchange constraints are all necessary for obtaining the reported improvements.

  1. The practical efficiency benefits are not fully convincing.

Avoiding an additional Transformer forward pass does not necessarily translate into lower end-to-end latency. The implementation still incurs repeated pooling, scheduling, selection, and KV gathering operations.

Indeed, the Jetson evaluation shows that BudgetKV can introduce noticeable request-time overhead compared with other methods. At a 1024-token budget, it adds approximately 1.51 seconds compared with Expected Attention and 2.05 seconds compared with Query-hidden SnapKV in the reported native execution configurations.

While the method clearly improves memory feasibility in the tested setting, its overall efficiency advantage under practical inference workloads remains less convincing.

  1. The applicability of the method is relatively restricted.

BudgetKV assumes that the task query is available before shared-context prefill and remains fixed throughout the request. This is reasonable for certain document QA and retrieval-augmented generation scenarios, but less suitable for general interactive inference or multi-turn conversations where user information needs may evolve.

Moreover, the method uses shared retention indices across all layers and KV heads, which prevents it from exploiting layer-specific or head-specific differences in token importance.

Questions for authors
  1. Could the authors provide a sensitivity analysis for the major heuristic components, including the semantic mixing coefficient, lexical safety margin, utility threshold, and admission limits? It would be particularly interesting to understand whether the reported improvements can be maintained with a simpler selection procedure.

  2. How well does the proposed semantic signal transfer to larger models and more diverse tasks? In particular, can the authors demonstrate consistent improvements beyond the retrieval-oriented scenarios where the current method appears most effective?

  3. Could the authors provide a more comprehensive end-to-end efficiency comparison under matched KV memory budgets and consistent decoding policies? This would help clarify whether the additional pooling, selection, and KV movement overhead can be justified by the memory savings in realistic deployment scenarios.

  4. EpiCache also investigates bounded persistent KV residency during prefill, although it relies on different selection signals. Could the authors clarify the practical advantages of BudgetKV over this closely related approach, ideally through a comparison under similar memory constraints?

8 comments

Thank you for your careful review and for recognizing our prefill-memory motivation, reuse of resident representations and Jetson evaluation. We sincerely appreciate your detailed suggestions, which helped strengthen our paper.

W1

Reviewer comment

Limited generalization of the proposed semantic signal.

Response

We sincerely apologize that the original desktop/device distinction was unclear, making the semantic and deployment benefits difficult to assess. We explain those outcomes and the edge objective below.

1. What the desktop results measure. The 400-document confirmation and length/task diagnostic ran on RTX 4090 D, isolating retention quality when execution fits desktop memory. The embedding score supplies early query affinity; the Transformer answers from retained states. Benefits vary with model, budget and task. The full corrected confirmation is:

Table 23 (p. 20).

ModelBBGain95% CIFamily-adjusted CI
Qwen3-1.7B5122.64[−1.08-1.08, 6.42][−2.13-2.13, 7.52]
Qwen3-1.7B10246.81[3.31, 10.41][2.37, 11.45]
Llama-3.2-1B5120.67[−1.26-1.26, 2.73][−1.74-1.74, 3.32]
Llama-3.2-1B10242.15[−0.22-0.22, 4.62][−0.91-0.91, 5.28]

Qwen/B=1024B=1024 gains 6.81 points, adjusted CI [2.37,11.45]; other intervals include zero. Desktop 32K BudgetKV/lexical retrieval is 66.25/10.00 on Qwen and 41.25/2.50 on Llama; Full KV provides references of 80.00/76.25. Table 26 retains QA/tracking outcomes. The common-decoder Qwen3-4B study adds +8.07 points.

2. The research target is an edge balance. Our primary objective is useful long-input delivery within an 8 GB device's memory and execution costs. A full prompt can exhaust prefill memory even when the final compressed cache fits. BudgetKV selects after each chunk, preserving relevant evidence as the input passes through a small persistent cache.

3. What is delivered on Jetson? Qwen3-1.7B FP16, B=1024B=1024; each completion column is out of 12, with unsuccessful requests scored zero:

Table 7 (p. 15).

MethodCompleted / 12: 8K16K32KDelivered: 16K32K
BudgetKV12121255.4251.25
Lexical control12121234.5830.42
StreamingLLM12121216.6733.33
EpiCache block12121225.0025.00
Query-visible SnapKV126016.670.00
Active PyramidKV12868.3316.67
Expected Attention12700.000.00
Full KV6000.000.00

BudgetKV completes 12/12 at every actual length, delivering 51.25 at 32K, versus lexical 30.42 and StreamingLLM 33.33, both also fully completing. Its retrieval score is 68.75 versus lexical 6.25, while QA/tracking tie. Thus the 20.83-point macro gain identifies relevance selection within the same executable footprint.

Normal SnapKV scores 46.25 versus BudgetKV 36.98 on desktop Qwen/B=1024B=1024 LongBench (Table 25). On Jetson it completes 6/12 at 16K and 0/12 at 32K; Full KV also completes 0/12 at 32K. Desktop quality and useful device delivery therefore need joint assessment.

4. Beyond retrieval on-device. The same policy supports summarization and repository completion:

Table 14 (p. 17).

ModelTaskBudgetKVLexical
Qwen3-1.7BQMSum12.8512.07
Llama-3.2-1BQMSum11.9710.99
Qwen3-1.7BRepoBench-P-E13.6012.67
Llama-3.2-1BRepoBench-P-E7.338.35

Both policies complete 80/80 requests across two models and 0/1 GiB load. QMSum uses ROUGE-L and code its task metric. Llama Jetson QA adds 300/300 completions across ten settings; the original scores 31.98 versus lexical 27.26.

Revised manuscript text. Section 4.7, p. 10, states:

“Each pair uses identical input and measured final KV bytes. The gain therefore reflects better retention within the same cache allocation (Appendix E).”

Section 4.2/Figure 3 (p. 6) and Table 7 (p. 15) give the device results. Section 4.2, p. 7, explains:

“Chunk-wise retention makes long requests executable, while semantic refinement improves retrieval quality within that constraint. The lexical and StreamingLLM controls also complete all inputs, so the quality comparison separates relevance selection from completion alone.”

Appendix E/Table 23 (p. 20) preserves all corrected contrasts; F.1/Table 24–25 (p. 21) gives model transfer and baselines. Section 4.6 (p. 9), C.2/Table 14 (p. 17) and H.1/Table 31 (p. 25) add natural tasks and Llama device sensitivity.

W2

Reviewer comment

The method relies on multiple heuristic components and hyperparameters.

Response

We are very sorry that the original ablations did not separate the purpose of each constraint from its effect on answer quality. This made the procedure's complexity and the necessity of its rules difficult to evaluate. We have added direct rule changes and a simpler alternative at matched realized KV.

Why these settings? Query overlap, digit and case cues prioritize direct matches, values and names. The 0.7-max/0.3-mean pooling combines a strong local cue with block-wide support. Separate [0,1] normalization makes α=0.5\alpha=0.5 an equal mixture. ε=0.15\varepsilon=0.15 permits a controlled lexical deficit; δ=0.05\delta=0.05 requires a utility gain. Two swaps per call, sixteen per request and locks regulate repeated admissions. Equal-size exchange preserves token count; sink/recent blocks preserve the designated boundary regions.

Direct rule evidence. Table 29 contains all 14 configurations on 30 development documents, two models and two budgets: 1,680/1,680 completed runs. Entries are task-macro changes from the original; each cell averages 15 2WikiMQA and 15 TriviaQA sources at the nominal budget.

Table 29 (p. 24).

SettingLlama B=512B=512B=1024B=1024Qwen B=512B=512B=1024B=1024
Original+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00
Semantic scoring disabled−0.45-0.45−4.73-4.73+4.97+4.97−1.94-1.94
Lexical guard removed+2.44+2.44+2.38+2.38+0.58+0.58+0.00+0.00
Combined-utility selector+3.81+3.81+2.42+2.42+1.15+1.15+4.29+4.29
Digit cue removed+5.24+5.24−0.46-0.46+8.84+8.84+2.95+2.95
Uppercase cue removed+0.32+0.32+2.30+2.30−2.46-2.46−3.76-3.76
Short-token cue removed+3.33+3.33+2.88+2.88+0.00+0.00+3.33+3.33
All non-query cues removed−1.34-1.34−0.48-0.48−0.94-0.94+3.23+3.23
Max/mean weights 0.5/0.5+1.11+1.11−0.06-0.06+0.00+0.00+3.33+3.33
Max-only pooling+8.64+8.64+2.94+2.94+6.85+6.85−0.71-0.71
Query cap q=16q=16+0.42+0.42+4.65+4.65+1.78+1.78−2.22-2.22
Query cap q=64q=64+4.88+4.88+2.38+2.38+2.06+2.06−5.40-5.40
Sink protection halved+0.00+0.00−10.14-10.14+0.00+0.00+0.00+0.00
Recent protection halved+2.82+2.82−0.37-0.37−0.37-0.37−2.22-2.22

The matrix tests the guard, simpler selector, cues, pooling, query cap and protection across both backbones and budgets.

Numerical controls. The Qwen study below varies mixing, both margins and the request cap, matching measured final KV bytes; each task has 15 development sources.

Table 30 (p. 24).

Setting2WikiMQATriviaQA
α=0.25\alpha=0.25 / 0.750.75+7.46+7.46 / +6.14+6.14−1.67-1.67 / −6.41-6.41
ε=0\varepsilon=0 / 0.300.30−1.11-1.11 / +5.56+5.56−2.78-2.78 / 0.000.00
δ=0\delta=0 / 0.100.10−0.79-0.79 / 0.000.00−5.56-5.56 / +6.67+6.67
Remove lexical guard+5.56+5.56−5.56-5.56
Global swap cap from 16 to 80.000.000.000.00

For example, removing the guard changes the two tasks by +5.56/−5.56; changing the global cap leaves them unchanged. An additional Llama Jetson sweep completes 300/300 requests across ten settings, with the original policy scoring 31.98 versus lexical 27.26. Its full numerical results are given in our Q1 reply.

Are all constraints necessary for quality? The added simpler selector retains protected blocks, then fills remaining capacity by combined utility, omitting exchange guards, margins and locks. On a separate 100-document cohort, changes from the original at B=512/1024B=512/1024 are −0.61/+3.22 on Llama and +1.48/+3.02 on Qwen. All 400 pairs match measured bytes, and all four adjusted intervals include zero. This gives no statistically resolved quality difference on that cohort; it supports simplification as a viable alternative there. Equal-size replacement and protection still have their stated resource/control roles.

Revised manuscript text. Section 3.3, p. 4, now specifies:

“The exchange must keep the token count unchanged, preserve lexical importance within tolerance ε\varepsilon, and improve combined utility by more than δ\delta.”

Appendix H/Table 29–30 (p. 24) adds the individual tests; H.1/Table 31 (p. 25) adds Llama. The paragraph beginning “The simpler selector” in Appendix H, p. 24, states:

“The simpler selector retains protected blocks and fills the remaining cache by combined utility, without the lexical exchange guard, admission margins or exchange locks. ... All 400 pairs match measured bytes; all four adjusted intervals include zero. This tests simplification independently of the frozen confirmation set.”

W3

Reviewer comment

The practical efficiency benefits are not fully convincing.

Response

We sincerely apologize that the original discussion did not clearly connect measured overhead to deployment value, making the practical advantage difficult to assess. We now explain the resource trade-off, useful delivery and controlled costs.

1. Why the original request time increases. The cited +1.51 seconds versus Expected Attention and +2.05 versus query-hidden SnapKV are Jetson measurements. Pooling, selection and KV gathering add work even without another Transformer pass. At B=1024B=1024, that systems study also records 488/520 MiB lower peak CUDA reservations, respectively (Table 12, p. 16). We retain both observations: a smaller measured CUDA reservation has an additional scheduling/movement cost.

2. The target is useful delivery under edge limits. Desktop selection-quality tests allow full-prompt processing when memory permits. On an 8 GB device, that step can prevent a long request from completing. We therefore evaluate executable input length, delivered answers and request cost together. The complete Qwen3-1.7B FP16/B=1024B=1024 Jetson study is:

Table 7 (p. 15).

MethodCompleted / 12: 8K16K32KDelivered: 16K32K
BudgetKV12121255.4251.25
Lexical control12121234.5830.42
StreamingLLM12121216.6733.33
EpiCache block12121225.0025.00
Query-visible SnapKV126016.670.00
Active PyramidKV12868.3316.67
Expected Attention12700.000.00
Full KV6000.000.00

BudgetKV completes 12/12 actual-32K requests and delivers 51.25 points, versus lexical 30.42 and StreamingLLM 33.33 with full completion. Normal SnapKV, which has higher common-decoder desktop LongBench scores, completes 0/12 at 32K here. Thus its desktop quality advantage does not supply an answer in this tested device condition.

3. What does the selection cost under equal output work? Two sources, three repetitions, B=1024B=1024, 64 forced outputs; complete lengths are 10,868/11,539 tokens:

Table 19 (p. 19).

Method8K time (s)Full-input time (s)
BudgetKV9.1611.27
Lexical control8.3410.29
StreamingLLM7.809.55
Query-visible SnapKV6.928.35
Native EpiCache64.3286.69

The four inexpensive policies use a common growing-cache decoder; EpiCache retains its native cache kernel. BudgetKV adds 0.82/0.98 seconds over lexical retention, about 10%. The table directly reports the latency choice among normal policies.

4. Native bounded-prefill comparison. Twenty paired 8K sources use Llama-3.2-1B FP16 and matched post-query KV. EpiCache includes clustering/routing and its original attention/cache kernels:

Table 16 (p. 18).

BBFixed 64 outputs: BudgetKV (s)EpiCache (s)SpeedupNatural-EOS score: BudgetKVEpiCache
5128.8558.166.57$\times$8.128.96
10249.1663.896.98$\times$12.934.54

All 240 fixed-work and 80 separate natural-EOS requests complete. At B=1024B=1024, BudgetKV combines 6.98× fixed-work speedup and +8.39 quality points. This establishes a measured cost/quality advantage over that native path, alongside the inexpensive alternatives above.

5. Component accounting. Separate synchronized profile medians are:

Table 20 (p. 19).

RegionBudgetKV (s)Lexical (s)
Embedding0.1410.000
Selection1.4580.472
Gather0.2050.199
Lexical0.5880.555

Four profile requests per method supply these medians. Nested regions are not additive; Table 19 uses separate normal executions.

Revised manuscript text. Section 4.5, p. 9, states:

“The fixed-work lexical comparison quantifies the additional relevance-selection cost: 0.82 seconds at 8K and 0.98 seconds on complete 10.9K/11.5K inputs, approximately 10% of lexical request time. Appendix D supplies the normal-policy controls, component timings and complete memory measurements.”

Section 4.2/Figure 3 (p. 6) gives delivery; Appendix D/Tables 16–20 (pp. 18–19) gives cost and memory. Appendix A, p. 14, states:

“Single-request latency measurements start with loaded models and prepared token inputs. It includes relevance scoring, prefill and greedy generation, plus EpiCache clustering and routing. It excludes loading, worker startup and background-memory allocation.”

Reviewer_8vK8OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

BudgetKV is a training-free KV-cache selection method for long-context inference on memory-constrained devices. The problem it addresses: during prompt prefill the KV cache grows until it exceeds memory, even when the weights and the intended final cache would fit. Block-wise prefill bounds persistent state, but conventional retention policies still require contextual attention profiles, patched-prompt forwards, or auxiliary encoders before deciding what to keep.

Critical strengths
  • The proposed method has been tested on a real 8G Jetson device, showing the capability of real deployment on edge devices.
Critical weaknesses
  • chunk-wise selection is not new. reference [1]
  • There are some constants and choose without any explain.
    • In Equation (1), why choose 0.7 and 0.3?
    • uj=(1−α)ℓj+αeju_j = (1-\alpha) \ell_j + \alpha e_j, why choose α=0.5\alpha=0.5?
    • Equation (3), ϵ=0.15\epsilon=0.15 and δ=0.05\delta=0.05, why these value?
  • Experiment comparisons are mainly against StreamingLLM and SnapKV. Please consider add more advance baselines.

[1] MoBA: Mixture of block attention for long-context LLMs

Questions for authors
  • In Figure 4: there is a significant performance decrease of BudgetKV. What cause this?
  • In references: some papers are accepted and should not cite the arxiv version. i.e., SnapKV.
5 comments

Thank you for your careful review and for recognizing our evaluation on a real 8 GB Jetson. We appreciate your constructive feedback and address each comment below.

W1

Reviewer comment

chunk-wise selection is not new. reference [1]

[1] MoBA: Mixture of block attention for long-context LLMs

Response We sincerely apologize that our original explanation did not make the purpose of chunk selection and our contribution relative to MoBA sufficiently clear. Thank you for drawing our attention to this important distinction.

1. Why we select chunks. We target edge requests where model weights and the final compressed cache fit in memory, but the full prompt may exceed memory during prefill. After each prefill chunk, BudgetKV selects KV blocks for subsequent computation, preserving useful distant evidence within a small persistent cache. This requires an early relevance signal, before sufficient contextual attention statistics are available.

2. What distinguishes BudgetKV from MoBA. Both methods organize tokens into blocks, but they make different decisions using different representations. The comparison below summarizes MoBA and our Sections 3.1–3.4; it is an explanatory comparison, rather than an additional experimental table.

AspectMoBABudgetKV
GoalReduce attention computation through sparse block routingPreserve query-relevant evidence during memory-bounded prefill
DecisionWhich KV blocks each attention query attends toWhich KV blocks remain after each prefill chunk
QueryCurrent attention query vectorApplication task query supplied before prefill
Block signalQuery affinity to mean-pooled contextual keysResident input-embedding similarity plus a lexical anchor
SelectionTop-kk attention-block routingGuarded, equal-size exchanges within the retention budget
Resource controlledBlocks participating in attentionPersistent KV positions across prefill chunks

Our contribution is an early query-relevance signal integrated with budget-preserving retention. BudgetKV reuses the pretrained input table without auxiliary encoders, learned selectors, or extra Transformer passes. A lexical anchor preserves direct evidence, while semantic refinement admits relevant alternatives through equal-size exchanges with lexical and utility tests. Sink and recent blocks remain protected. When protected blocks fit the budget, persistent KV stays within BB tokens after selection and B+pB+p after appending a chunk of at most pp tokens, preserving distant evidence beyond cache capacity.

3. What this achieves on the device. On the 8 GB Jetson, Qwen3-1.7B FP16 with B=1024B=1024 completes 12/12 requests at 8K, 16K, and 32K, while query-visible SnapKV completes 12/12, 6/12, and 0/12, respectively (Figure 3, p. 6). At 32K, task-macro scores are 51.25 for BudgetKV, 30.42 for the lexical control, and 33.33 for StreamingLLM. To isolate the relevance-selection effect, we reproduce the complete retrieval table, as shown in Table 2 (p. 7).

Each length reuses four sources; unsuccessful requests score zero. BudgetKV and its semantic-disabled lexical control both complete every retrieval request at the same realized KV size. The +62.50-point 32K difference therefore measures the value of semantic refinement within the same retention framework. Completion and useful evidence retention are both necessary for the application.

4. Revised manuscript text. Section 2, p. 3, paragraph beginning “Quest and MoBA,” now states:

“Quest and MoBA select pages or blocks for attention (Tang et al., 2024; Lu et al., 2025); BudgetKV selects KV entries to retain across prefill chunks. ”

Section 3.2, p. 4, paragraph after Equation (2), specifies:

“Equation 2 uses embedding lookups and matrix products. The representations are taken directly from the pretrained input table. Embedding lookups and matrix products run on the model GPU in 256-token batches before prefill. Their scalar scores are transferred to the central processing unit (CPU) for normalization and selection. This step introduces no extra Transformer forward or learned parameters.”

Sections 3.3–3.4 (pp. 4–5) give the exchanges and cache bound; Section 4.2, Figure 3 and Table 2 (pp. 6–7) provide the deployment results reproduced above.

W2

Reviewer comment

There are some constants and choose without any explain.

  • In Equation (1), why choose 0.7 and 0.3?
  • uj=(1−α)ℓj+αeju_j = (1 - \alpha)\ell_j + \alpha e_j, why choose α=0.5\alpha = 0.5?
  • Equation (3), ϵ=0.15\epsilon = 0.15 and δ=0.05\delta = 0.05, why these value?

Response

We are very sorry that the rationale for these constants was left unexplained in the original manuscript. We understand that listing the values alone does not explain why they are appropriate. Below we explain the design purpose of each value and provide the added sensitivity results directly.

1. Pooling weights 0.7/0.3. Sparse evidence, such as an entity or number, may be diluted by block averaging. We therefore assign 0.7 to the maximum and 0.3 to the mean, balancing salient cues with distributed relevance. The same pooling applies to lexical and semantic scores:

sjraw=0.7max⁡i∈Bjsi+0.3mean⁡i∈Bjsi.s_j^{\mathrm{raw}}=0.7\max_{i\in\mathcal B_j}s_i+0.3\operatorname{mean}_{i\in\mathcal B_j}s_i.

Table 29 compares this setting with equal pooling and max-only pooling. Across Llama/Qwen and B=512/1024B=512/1024, task-macro changes range from −0.06 to +3.33 and −0.71 to +8.64 points, respectively.

2. Mixing weight α=0.5\alpha=0.5. After normalizing lexical and semantic scores to [0,1], we combine them as uj=(1−α)ℓj+αeju_j=(1-\alpha)\ell_j+\alpha e_j. Equal weighting avoids favoring either signal a priori and provides a shared setting across tasks and models.

3. Margins ε=0.15\varepsilon=0.15 and δ=0.05\delta=0.05. A new exchange must satisfy

nc=nv,ℓc+0.15≥ℓv,uc>uv+0.05.n_c=n_v,\qquad\ell_c+0.15\geq\ell_v,\qquad u_c>u_v+0.05.

This permits a limited lexical trade-off while requiring a meaningful utility gain. Equal-size exchange preserves the token budget, and normalized scores make both thresholds scale-consistent. For example, a candidate with lexical score 0.60 can replace a 0.70 victim, but not a 0.80 victim.

4. Added numerical sensitivity experiments. The following is the complete Qwen table. Each task uses 15 development documents; paired changes are relative to the original setting, with measured final KV bytes matched.

Table 30 (p. 24): paired score changes from the fixed Qwen3-1.7B setting; 15 documents per task.

Setting2WikiMQATriviaQA
α=0.25\alpha=0.25 / 0.750.75+7.46+7.46 / +6.14+6.14−1.67-1.67 / −6.41-6.41
ε=0\varepsilon=0 / 0.300.30−1.11-1.11 / +5.56+5.56−2.78-2.78 / 0.000.00
δ=0\delta=0 / 0.100.10−0.79-0.79 / 0.000.00−5.56-5.56 / +6.67+6.67
Remove lexical guard+5.56+5.56−5.56-5.56

For example, increasing α\alpha helps 2WikiMQA but reduces TriviaQA, while removing the lexical guard changes the two tasks by +5.56/−5.56. This supports retaining a shared balance between the signals. We also added an on-device Llama study: ten settings on 30 development sources at B=1024B=1024, with 300/300 completed requests. The complete table is:

Table 31 (p. 25).

SettingScoreSettingScore
Original31.98Lexical control27.26
α=0.25\alpha=0.2533.05α=0.75\alpha=0.7534.92
ε=0\varepsilon=027.26ε=0.30\varepsilon=0.3034.37
δ=0\delta=034.37δ=0.10\delta=0.1035.56
Per-call cap 131.92Per-call cap 431.98

The original policy improves over lexical retention by 4.72 points. With ε=0\varepsilon=0, all 30 outputs match lexical retention, directly showing how the tolerance enables semantic admissions. Changing the per-call exchange cap from two to one or four changes the score by at most 0.06 points. Together, these experiments explain the functional effect of the settings and show their behavior on another backbone and the target device; changes can favor different tasks or settings.

5. Revised manuscript text. Section 3.2, p. 4, paragraph after Equation (2), now makes the common scale explicit:

“We set ℓj=N(ℓraw)j\ell_j=\mathcal N(\ell^{\rm raw})_j and ej=N(eraw)je_j=\mathcal N(e^{\rm raw})_j, then combine them as uj=(1−α)ℓj+αeju_j=(1-\alpha)\ell_j+\alpha e_j, with α=0.5\alpha=0.5. Thus both admission thresholds operate on a common unit scale.”

Section 3.3, p. 4, introduces Equation (3) as follows:

“The exchange must keep the token count unchanged, preserve lexical importance within tolerance ε\varepsilon, and improve combined utility by more than δ\delta.”

Appendix H, Tables 29–31 (pp. 24–25), adds the rule and numerical studies. Section H.1, p. 25, states:

“With ε=0\varepsilon=0, all 30 generated outputs match the lexical condition, showing how the lexical guard regulates semantic admissions in this cohort. Changing the per-call cap from two to one or four leaves the macro score within 0.06 points of the original.”

W3

Reviewer comment

Experiment comparisons are mainly against StreamingLLM and SnapKV. Please consider add more advance baselines.

Response

We sincerely apologize for the incomplete coverage of recent baselines in the original experiments. Thank you for this valuable suggestion; we agree that these comparisons are needed to assess our method fairly. We have added active layer-adaptive PyramidKV, Expected Attention, native-kernel EpiCache, and native Fast KVzip, and use normal query-visible SnapKV. Their measured results are listed below, with each comparison's model and protocol identified.

1. All methods on the 8 GB Jetson. The full original Table 7 compares Qwen3-1.7B FP16 at B=1024B=1024. Each completion count has denominator 12. Delivered quality includes every planned request, assigning zero when no answer is returned.

Table 7 (p. 15).

MethodCompleted: 8K16K32KDelivered score: 16K32K
BudgetKV12121255.4251.25
Lexical control12121234.5830.42
StreamingLLM12121216.6733.33
EpiCache block12121225.0025
Query-visible SnapKV126016.670
Active PyramidKV12868.3316.67
Expected Attention127000
Full KV60000

As also shown in Figure 3 (p. 6), BudgetKV provides 12/12 actual-32K completions and 51.25 delivered points. The table includes the newly added PyramidKV and Expected Attention results, alongside the EpiCache block adaptation. This tests whether the complete inference path fits the device and what quality it delivers.

2. Native EpiCache on the same Jetson. This separate comparison uses Llama-3.2-1B FP16 and 20 paired 8K sources, matching measured post-query KV. EpiCache runs its original CUDA cache/attention kernels in a single-episode document adaptation, including clustering/routing.

Table 16 (p. 18).

BBFixed 64 outputs: BudgetKV (s)EpiCache (s)RatioNatural-EOS score: BudgetKVEpiCache
5128.8558.166.57$\times$8.128.96
10249.1663.896.98$\times$12.934.54

Times are medians over three trials with 64 forced outputs; quality comes from separate natural-EOS runs. Both methods complete all requests. At B=1024B=1024, BudgetKV gives 6.98× fixed-work speedup and +8.39 quality points. At B=512B=512, fixed-work latency is 8.85 versus 58.16 seconds. Figure 5 (p. 9) shows this comparison; Appendix D separately reports CUDA peaks, process RSS and compatible software stacks.

3. Active PyramidKV at matched KV. Appendix F.2 (p. 21) uses unequal layer allocation, 40 documents per model/budget, and exact measured prefill-KV matching in all 160 pairs. The paragraph's complete scores are reproduced here:

ModelBBBudgetKVActive PyramidKV
Llama-3.2-1B5129.8919.08
Llama-3.2-1B102418.7920.69
Qwen3-1.7B51235.4850.33
Qwen3-1.7B102437.6751.42

These quality results and the Jetson completion matrix show the quality–capacity choice directly.

4. Native Fast KVzip. On RTX 4090D, the released Qwen3-4B gate and native cache path are compared in BF16 on 20 paired 8K documents; all 40 pairs meet the 0.1% measured-byte tolerance. Appendix F.2 (p. 21) reports:

BBBudgetKVNative Fast KVzip
51233.171.20
102432.8010.89

Where we revised the paper. Section 4.2/Figure 3 (p. 6) and Table 7 (p. 15) add device completion and quality for all eight methods. Section 4.5 (pp. 8–9), Figure 5 and Appendix D/Table 16 (p. 18) add native EpiCache. Appendix F.2 (p. 21), paragraphs beginning “On the same GPU class” and “The active PyramidKV comparison,” add native Fast KVzip and active PyramidKV. Appendix F.1/Table 25 (p. 21) preserves the full normal query-visible common-decoder comparison.

The added native-comparison protocol in Appendix D, p. 18, reads:

“The native Jetson comparison uses Llama-3.2-1B FP16 and 20 fixed 8K sources: four each from 2WikiMQA, HotpotQA, MuSiQue, passage retrieval and TriviaQA. Both budgets match measured KV after the query enters the cache. The released EpiCache implementation runs its original FlashAttention and cache kernels on Orin sm87. It is adapted to one document episode, using LLM embeddings for clustering and routing.”

Appendix F.2, p. 21, also states:

“The active PyramidKV comparison uses 40 documents per model and budget with unequal layer allocation and exact measured prefill KV matching in all 160 pairs.”

Reviewer_RCASOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

BudgetKV targets long-context inference where the task query is known and fixed before shared-context prefill. Instead of waiting for attention statistics, it scores token blocks using the model’s resident input embeddings plus a frozen lexical anchor, then performs chunk-wise selection with guarded equal-size exchanges. This yields a conditional persistent-KV token-residency contract of at most BB after selection and B+pB+p between selections, assuming protected-block feasibility. Experiments include a frozen 400-document LongBench confirmation set, an adapted RULER diagnostic at 8K/16K/32K, and 8GB Jetson deployment. The main positive quality result is Qwen3-1.7B at B=1024B=1024: +6.81 points over no-semantic retention, with an adjusted 95% CI excluding zero; other model-budget cells are inconclusive. The largest task-specific gain is on needle retrieval rather than reasoning.

Critical strengths
  • BudgetKV uses lexical and semantic scores derived only from resident input embeddings and fixed rules, without relying on LLM-internal attention statistics or additional Transformer passes, to perform effective fixed-budget KV compression.
Critical weaknesses
  • The method requires the query to be known and fixed before shared-context prefill, which limits its applicability. The authors’ own SCBench five-turn results (5.83/5.86 versus 8.66 for Full KV) show weak multi-turn quality.

  • The LongBench main table foregrounds query-hidden SnapKV as a secondary comparator, but this is an unnatural SnapKV configuration: in normal use the query sits inside the observation window and is visible to attention. Figure 9 shows query-visible SnapKV is stronger in all four model-budget cells.

  • Key hyperparameters are frozen without sensitivity analysis, including the 0.7/0.3 block aggregation, lexical coefficients, ϵ\epsilon/δ\delta, q≤32, α=0.50, and sink/recent sizes. Only no-semantic and no-gate ablations are provided.

  • The lexical predicates are clearly domain-biased (e.g., “KV”=5.0, digits, uppercase letters, short tokens), yet no cross-domain or predicate-sensitivity analysis is provided. The RULER diagnostic does stratify by Needle retrieval, QA, and Variable tracking, but this is not an ablation of the lexical predicates.

Questions for authors
  • How were “KV”=5.0 and the other lexical coefficients chosen? Is there any sensitivity or cross-domain evidence, and could this weight dominate the anchor in non-KV-cache domains?
5 comments

Thank you for recognizing the value of obtaining query relevance from resident embeddings without attention statistics or another Transformer pass. We appreciate your careful comments and address each concern below.

W1 — Earlier desktop results and new Jetson dialogue evidence

Reviewer comment

The method requires the query to be known and fixed before shared-context prefill, which limits its applicability. The authors’ own SCBench five-turn results (5.83/5.86 versus 8.66 for Full KV) show weak multi-turn quality.

Response

We sincerely apologize that the original presentation did not clearly distinguish the earlier desktop dialogue diagnostic from our edge-device objective. This made the query assumption and the meaning of the multi-turn scores difficult to assess. Thank you for bringing this to our attention.

Why the earlier scores were lower. The 5.83/5.86 versus 8.66 scores came from the earlier desktop five-turn diagnostic. BudgetKV reused the initial query signal as later questions changed; evicted KV could not be recovered. Full KV kept the uncompressed context and could access later-needed evidence when desktop memory allowed execution. This explains the quality cost of reusing the initially selected cache.

Our deployment objective. We balance executable input length, retained memory and useful answers on an 8 GB edge device. A desktop quality advantage cannot serve the application when the request cannot finish on-device. We therefore added paired Jetson five-turn conversations at actual 16K and 32K.

Original Table 4 (V4, p. 8).

Method16K: completed / 8Delivered score32K: completed / 8Delivered score
BudgetKV8/86.458/87.96
Lexical control8/89.208/86.97
StreamingLLM8/87.648/86.74
Query-visible SnapKV6/87.080/80.00
Full KV0/80.000/80.00

Each length uses eight frozen SCBench-derived sources, two per task, with Qwen3-1.7B FP16 and B=1024B=1024. All methods receive the same prepared inputs and generation rules. Delivered score averages four task means and five turns, assigning zero to incomplete sessions. Sessions use greedy EOS with up to 64 tokens per turn and a 180-second end-to-end budget including setup. The initial query signal is unchanged; this is not a query-refresh modification.

At 32K, BudgetKV completes all eight five-turn sessions, with the highest delivered score: 7.96 versus lexical 6.97 and StreamingLLM 6.74, using at most 112 MiB persistent KV after each turn. SnapKV completes 6/8 at 16K and 0/8 at 32K; Full KV completes neither length. This new device cohort has its own input/metric protocol, so its absolute scores are separate from the earlier desktop averages.

The query must be available before the initial prefill, as in document QA and repository assistance. Later dialogue turns can change the question and reuse the retained cache; the initial selection does not anticipate their unknown information needs. The new experiment establishes that this fixed-signal design can deliver complete 32K five-turn sessions on the tested edge device.

Revised manuscript text. Section 4.4, p. 8, introduces the study as follows:

“We evaluate multi-turn dialogue on Jetson to test whether long-input delivery continues across successive questions. Each of eight frozen SCBench-derived conversations (Li et al., 2025) contains five turns and is tested with actual 16K and 32K initial inputs. Qwen3-1.7B uses FP16 weights and KV, with B=1024B=1024.”

The paragraph immediately after Table 4 states:

“At 32K, BudgetKV completes all eight five-turn conversations and achieves the highest delivered score among the tested policies, 7.96 versus 6.97 for its fully completing lexical control and 6.74 for StreamingLLM. Query-visible SnapKV completes six conversations at 16K and none at 32K; Full KV completes none at either length. BudgetKV maintains at most 112 MiB of persistent KV after each turn.”

Appendix I, p. 25, makes the cache reuse explicit:

“Each context is prefilled once; subsequent questions and generated answers reuse the cache with a fixed initial query signal.”

Section 5 (p. 10) and Appendix J.1 (p. 26) clarify the initial-query application contract; the latter states:

“BudgetKV assumes that the application receives a shared context and a task query before model prefill. The query may be serialized after the context in the model input, but its token IDs are available to the selector before the first context chunk.”

W2

Reviewer comment

The LongBench main table foregrounds query-hidden SnapKV as a secondary comparator, but this is an unnatural SnapKV configuration: in normal use the query sits inside the observation window and is visible to attention. Figure 9 shows query-visible SnapKV is stronger in all four model-budget cells.

Response

We are very sorry that foregrounding query-hidden SnapKV in the earlier presentation caused confusion about the fairness of the comparison. We should have made the normal query-visible setting and the purpose of the desktop experiment clear. Thank you for pointing this out. We now use normal query-visible SnapKV in both the desktop quality study and Jetson deployment experiments.

Why include desktop results? With enough memory, a common decoder and matched retained KV isolate selection quality. The target device separately tests full-request feasibility. The complete normal query-visible results are:

Original Table 25 (p. 21).

ModelBBBudgetKVLexical controlSnapKVEpiCache blockGate
Llama-3.2-1B51212.3411.5919.7316.69--
Llama-3.2-1B102419.8817.6220.3116.97--
Qwen3-1.7B51230.2627.6341.7230.3--
Qwen3-1.7B102436.9830.1246.2535.46--
Qwen3-4B51232.6427.3551.0523.6433.28
Qwen3-4B102442.7234.6552.6931.5538.4

The study uses 400 LongBench documents, five tasks, FP16 KV, an 8,192-token input cap and common greedy EOS decoding up to 64 outputs. All methods receive the same query. SnapKV includes it in its observation window; BudgetKV gives its IDs to the scorer. Median KV matches per cell; exact-byte SnapKV pairs preserve the ranking. Gate uses the released Fast KVzip selector with uniform-head gathering.

What changes on the target device. As shown in Figure 3 (p. 6), the complete paths have different feasibility as actual length grows:

Original Table 7 (p. 15).

MethodCompleted / 12: 8K16K32KDelivered score: 16K32K
BudgetKV12121255.4251.25
Lexical control12121234.5830.42
StreamingLLM12121216.6733.33
EpiCache block12121225.0025
Query-visible SnapKV126016.670
Active PyramidKV12868.3316.67
Expected Attention127000
Full KV60000

Every completion column has denominator 12. SnapKV completes 6/12 at 16K and 0/12 at 32K; BudgetKV completes 12/12 throughout, delivering 51.25 at 32K versus lexical 30.42 and StreamingLLM 33.33. Figure 4 (p. 7) additionally tests six natural 10,070–16,703-token questions under 1 GiB locked competing memory: BudgetKV completes 6/6 with 43.5 F1, and normal SnapKV 0/6. These results concern the tested device and conditions.

The desktop table measures quality when execution is possible; the Jetson studies measure useful long-input delivery under the actual memory constraint.

The complete 32K controlled-pressure table supplies another direct deployment comparison: Qwen3-1.7B FP16, B=1024B=1024, three sources, with unsuccessful requests scored zero.

Original Table 3 (p. 8).

MethodCompleted / 3: 0 MiB load1 GiB loadDelivered score: 1 GiB
BudgetKV3/33/358.33
Lexical control3/33/333.33
StreamingLLM3/33/333.33
EpiCache block3/33/333.33
Query-visible SnapKV0/30/30
Full KV0/30/30

With 1 GiB occupied, BudgetKV preserves 3/3 completions and a 25-point delivered-score gain over the fully completing lexical control.

Revised manuscript text. Appendix F.1, p. 21, now specifies:

“Table 25 gives every method the same task query. SnapKV includes it in its normal observation window; BudgetKV supplies its IDs to the scorer. Median prefill KV bytes match within each cell. Restricting SnapKV comparisons to exact-byte pairs preserves the ranking direction in all six cells.”

Sections 4.2–4.3/Figures 3–4 (pp. 6–7), Table 3 (p. 8) and Table 7 (p. 15) foreground Jetson completion and delivered quality. Historical Table 22 (p. 20) labels the earlier query-hidden condition.

W3

Reviewer comment

Key hyperparameters are frozen without sensitivity analysis, including the 0.7/0.3 block aggregation, lexical coefficients, ε/δ, q≤32, α=0.50, and sink/recent sizes. Only no-semantic and no-gate ablations are provided.

Response

Please accept our sincere apologies for the incomplete parameter rationale and component evidence, which made their effects difficult to assess. Thank you for identifying these gaps. We now provide direct rule and numerical tests beyond no-semantic/no-gate.

Parameter rationale. The 0.7-max/0.3-mean pooling combines strong local evidence with block-wide support. After normalization to [0,1], α=0.5\alpha=0.5 weights lexical/semantic signals equally; ε=0.15\varepsilon=0.15 permits a controlled lexical trade-off and δ=0.05\delta=0.05 requires a utility gain. The q=32q=32 cap bounds scoring work, prioritizing rarer prompt tokens. Sink/recent lengths scale with budget: 32/128 at B=512B=512, 64/256 at B=1024B=1024. Two swaps per call and sixteen per request limit repeated admissions. Q1 gives the lexical coefficients.

1. Direct component tests. Table 29 now reports all 14 configurations on 30 development documents, two models and two budgets: 1,680/1,680 completed runs. Every cell averages 15 2WikiMQA and 15 TriviaQA sources; entries are task-macro changes from the original setting at the nominal budget.

Original Table 29 (p. 24).

SettingLlama B=512B=512B=1024B=1024Qwen B=512B=512B=1024B=1024
Original+0.00+0.00+0.00+0.00+0.00+0.00+0.00+0.00
Semantic scoring disabled−0.45-0.45−4.73-4.73+4.97+4.97−1.94-1.94
Lexical guard removed+2.44+2.44+2.38+2.38+0.58+0.58+0.00+0.00
Combined-utility selector+3.81+3.81+2.42+2.42+1.15+1.15+4.29+4.29
Digit cue removed+5.24+5.24−0.46-0.46+8.84+8.84+2.95+2.95
Uppercase cue removed+0.32+0.32+2.30+2.30−2.46-2.46−3.76-3.76
Short-token cue removed+3.33+3.33+2.88+2.88+0.00+0.00+3.33+3.33
All non-query cues removed−1.34-1.34−0.48-0.48−0.94-0.94+3.23+3.23
Max/mean weights 0.5/0.5+1.11+1.11−0.06-0.06+0.00+0.00+3.33+3.33
Max-only pooling+8.64+8.64+2.94+2.94+6.85+6.85−0.71-0.71
Query cap q=16q=16+0.42+0.42+4.65+4.65+1.78+1.78−2.22-2.22
Query cap q=64q=64+4.88+4.88+2.38+2.38+2.06+2.06−5.40-5.40
Sink protection halved+0.00+0.00−10.14-10.14+0.00+0.00+0.00+0.00
Recent protection halved+2.82+2.82−0.37-0.37−0.37-0.37−2.22-2.22

2. Numerical perturbations. The complete Qwen study varies mixture, tolerances and admission cap while matching measured final KV bytes. Each task has 15 development documents.

Original Table 30 (p. 24).

Setting2WikiMQATriviaQA
α=0.25\alpha=0.25 / 0.750.75+7.46+7.46 / +6.14+6.14−1.67-1.67 / −6.41-6.41
ε=0\varepsilon=0 / 0.300.30−1.11-1.11 / +5.56+5.56−2.78-2.78 / 0.000.00
δ=0\delta=0 / 0.100.10−0.79-0.79 / 0.000.00−5.56-5.56 / +6.67+6.67
Remove lexical guard+5.56+5.56−5.56-5.56
Global swap cap from 16 to 80.000.000.000.00

Guard removal changes the tasks by +5.56/−5.56; reducing the global cap leaves both unchanged.

3. Second-backbone tests on Jetson. Ten settings on 30 Llama development sources, B=1024B=1024, complete 300/300 requests:

Original Table 31 (p. 25).

SettingScoreSettingScore
Original31.98Lexical control27.26
α=0.25\alpha=0.2533.05α=0.75\alpha=0.7534.92
ε=0\varepsilon=027.26ε=0.30\varepsilon=0.3034.37
δ=0\delta=034.37δ=0.10\delta=0.1035.56
Per-call cap 131.92Per-call cap 431.98

The original policy gains 4.72 points over lexical retention. With ε=0\varepsilon=0, all 30 outputs match lexical; changing the per-call cap from two to one or four changes the score by at most 0.06 points.

A separate 100-document simpler-selector study removes the exchange guard, margins and locks. Its score changes at B=512/1024B=512/1024 are −0.61/+3.22 on Llama and +1.48/+3.02 on Qwen; all 400 pairs match measured bytes and all four adjusted intervals include zero.

Revised manuscript text. Sections 3.2–3.3 (p. 4) specify the score scale and admission tests; Appendix H, Tables 29–31 (pp. 24–25), adds the studies above. Appendix J, Table 33 (p. 27), lists the full configuration. Appendix H.1, p. 25, states:

“With ε=0\varepsilon=0, all 30 generated outputs match the lexical condition, showing how the lexical guard regulates semantic admissions in this cohort. Changing the per-call cap from two to one or four leaves the macro score within 0.06 points of the original.”

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.