acceptodds
Under review as a conference paper at ICLR 2027

SWIFT: Second-order Weighted Importance For Tokens

11 views

Abstract

KV-cache eviction reduces the memory and bandwidth cost of long-context inference, but its effectiveness depends on identifying cached entries whose physical removal minimally changes model behavior. Existing methods typically address the physical removal perturbation, downstream sensitivity, and global budget allocation separately, making eviction scores difficult to compare across layers and KV heads. We introduce SWIFT, a task-free second-order KV-cache eviction method grounded in a self-fidelity objective. Using only the cached context, full-coverage observations define fidelity against detached outputs of the uncompressed model, while exact softmax-renormalized removal perturbations and matrix-free scalar probes estimate second-order output damage without materializing the Hessian. These globally comparable scores enable SWIFT to allocate a shared physical KV budget across layers, KV heads, and tokens. Experiments cover controlled interventions and comparisons across three 7–8B model families and four long-context benchmarks. SWIFT achieves Pearson and Spearman correlations of and with measured eviction damage and ranks first or joint-first in 22 of 24 primary model–benchmark–retention comparisons. At 20% retention, it provides physical KV compression, delivers a 9.7–14.1% steady-state decoding speedup, and realizes approximately 80% of the empirically attainable historical-KV latency reduction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache compression, token importance, second-order saliency, long-context inference, efficient LLM serving

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cP8cOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SWIFT, a task-free KV-cache eviction method that estimates the output corruption caused by removing individual KV pairs. SWIFT defines a self-fidelity objective against the detached output of the uncompressed model and combines exact softmax-renormalized eviction perturbations with a matrix-free second-order approximation, avoiding explicit Jacobian or Hessian construction. The resulting scores are comparable across layers, KV heads, and token positions, enabling allocation of a single global physical KV budget. Experiments show that the proposed importance score is highly correlated with measured singleton-eviction damage and that the resulting compressed caches preserve strong downstream accuracy while reducing physical KV memory and steady-state decoding latency.

Critical strengths

A principled formulation enabling global KV-budget allocation. The proposed method provides task-free and globally comparable importance scores, enabling a single physical KV budget to be allocated jointly across layers, KV heads, and token positions rather than relying on predetermined per-layer or per-head quotas. This capability is supported by a principled second-order formulation derived naturally from a self-fidelity objective based on output damage. By combining exact softmax-renormalized eviction perturbations with an efficient matrix-free estimator, SWIFT translates this formulation into a practical scoring procedure without explicitly constructing the Jacobian or Hessian.

Critical weaknesses
  1. Concerned about the practical efficiency of SWIFT, not sufficiently established by the reported latency analysis. Unlike methods that derive eviction scores from a single prefill, SWIFT appears to require ⌈N/512⌉\lceil N/512\rceil additional observation forwards, each attending to the full cached context and requiring a VJP. However, Figure 5 reports only 19–142ms19–142 ms of context-level pruning time, without a sufficiently precise accounting of which operations are included, while Supplementary Table 11 reports approximately 4.17–4.56s4.17–4.56 s for full-coverage selection on an 8K context. Moreover, the reported decoding speedups exclude selection, cache gathering, query appending, and graph setup. Without a consistent end-to-end latency breakdown and amortization analysis, it is difficult to determine when SWIFT provides a net practical benefit over less expensive eviction methods.

  2. The observation-based scoring protocol is insufficiently specified and validated despite being central to the method. SWIFT’s scores depend on synthetic repeat observations, yet the exact instruction prp_r, placement of the 8-token anchor, and use of model-specific chat templates are not provided. Since these choices directly affect attention patterns, gradients, and ultimately the global KV ranking, this omission limits reproducibility and leaves the robustness of the proposed task-free score unclear. More broadly, several definitions required to understand the scoring procedure and tensor aggregation are deferred to the supplementary material, making the main paper insufficiently self-contained.

Questions for authors

Q1. End-to-end selection cost

For a context of length NN, SWIFT appears to process ⌈N/512⌉\lceil N/512\rceil additional repeat observations after the initial dense prefill, with each observation attending to the full cached context and requiring a VJP. Does the 19–142ms19–142 ms pruning time in Figure 5 include all observation forwards, VJPs, candidate-score computation, max aggregation, global ranking, and physical KV gathering? Please also clarify how these results reconcile with the 4.17–4.56s4.17–4.56 s full-coverage selection times reported for an 8K Qwen3-8B context in Supplementary Table 11.

A latency breakdown relative to the original context prefill, reported as a function of context length, comparing to other baselines would materially affect my assessment.

Q2. Amortized practical benefit

Since the 9.7–14.19.7–14.1% decoding speedups exclude one-time scoring and cache construction, how many generated tokens or repeated queries are required for SWIFT to break even relative to the full-cache baseline and lower-cost eviction methods? Please report end-to-end latency including prefill, SWIFT scoring, physical gathering, query processing, and decoding for representative single-query and multi-query workloads.

Q3. Singleton scores under joint eviction

SWIFT ranks candidates using singleton-eviction scores but removes up to 8080% of the KV pairs simultaneously without recomputing interactions among removed candidates. How well do singleton scores predict damage under such large joint interventions? An analysis comparing the summed or ranked singleton estimates with measured joint-eviction damage at different retention ratios would clarify the validity of the global one-shot allocation.

Q4. Exact observation construction and robustness

What is the exact repeat instruction prp_r, including the placement of the preceding segment’s 8-token anchor and any model-specific chat templates? Is the same instruction used across Qwen, Mistral, and Llama?

3 comments
AuthorsOpenReview ↗

Response to Reviewer cP8c

Thank you for recognizing SWIFT's formulation. We have revised both the main paper and supplement, with changes marked in blue.

Q1. Complete selection cost

SWIFT reduces complete preparation time by 74–75% versus KVGrad. Figure 5's SWIFT/KVGrad milliseconds cover selection/gathering from precomputed scores, excluding observation forwards/VJPs. Original Table 11 (now Table 13) uses four prefixes, 64-token observations and averaging, unlike the final 512-token/max protocol.

Qwen3-8B/A100 80GB timings use nested prefixes, 20% retention, shared full-coverage observations and EvictCache, without CUDA Graphs. Both methods use two forwards and one VJP per observation, totaling 16/32/64 VJPs at 8K/16K/32K. SWIFT combines four vectors before differentiation. Three-trial medians:

HistoryMethodPrefill (s)Scoring (s)Preparation (s)Prep./prefill
8KSWIFT0.80920.98821.81226.96
8KKVGrad0.81085.80386.627106.91
16KSWIFT1.79073.74075.55442.22
16KKVGrad1.793283.477285.320159.15
32KSWIFT4.389280.075284.49664.82
32KKVGrad4.3961091.4381095.864249.28

Scoring includes all forwards/VJPs, scores and max aggregation. Preparation adds ranking, gathering and request setup. Ratios use unrounded medians. Section 3.4 and B.8 Tables 23–24 report costs and stages. All trials are retained, with ranges in B.8.

Q2. Single-query and multi-query benefit

One preparation serves 32 requests (two alternating queries, 16 tokens each). Full reuses its cache with native FlashAttention-2. Totals include preparation, cloning, query processing, decoding and transfer, excluding loading, input preparation, warm-up and validation:

HistoryFull, 1 / 32 requests (s)SWIFT, 1 / 32 (s)KVGrad, 1 / 32 (s)
8K1.306 / 14.31022.248 / 37.34087.057 / 102.381
16K2.286 / 16.26676.172 / 90.351285.855 / 300.736
32K4.982 / 22.967284.991 / 300.8411096.340 / 1111.953

SWIFT saves 64–73% versus KVGrad over 32 requests at the same KV budget. No Full break-even occurs at tested 1/2/4/8/16/32-request prefixes. Break-even with longer outputs or cheaper methods remains unmeasured.

The main benefit is quality with less KV memory. Figure 6 (Qwen3-8B, 24.7K, 20% retained) shows 3.42→0.71 GiB (~79% less) and 16.39→14.42 ms/token (~12% less) in steady decoding. Pruning reduces history attention and cache traffic, while other model computation remains. Memory savings apply once packed, while scoring/setup need enough later decoding to amortize.

Q3. Singleton scores under joint eviction

One Qwen3-8B/4K context uses four vectors, eight observations and KV-head scoring. Nine equal-budget sets comprise SWIFT, KVGrad, AV, two random and four nearby SWIFT sets. Summed deleted-pair scores rank measured joint hidden-output damage on the prespecified first observation:

RetentionNine-set SpearmanSWIFT + four nearby sets
20%0.88330.9000
50%0.98331.0000
80%0.93330.9000

Correlations remain 0.8833/0.9000 at 80% deletion, supporting joint-eviction ranking (Section 3.1, B.4 Table 7).

Q4. Exact observation construction and robustness

All three models encode these strings without added special tokens:

First: "\n\nRepeat the previous context exactly."
Later: "\n\nRepeat the part of the previous context exactly, starting with"

Token order: instruction → previous segment's last eight token IDs → suffix → segment. The first has no anchor. No added spaces or anchor decoding/re-encoding. Targets are current-segment final hidden positions.

SCBench Qwen3-8B extracts this suffix with thinking disabled, without wrapping each observation as a full chat message:

<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n

Mistral-7B-v0.3/Llama-3.1-8B have no chat template here and use a newline, as do independent RULER controls including Qwen (Section 2.3, A.4).

Qwen3-8B, 20% retention and four fixed later-segment windows give:

ConditionKV scoreRank SpearmanMask Jaccard
Full95.00——
Base, anchor 895.001.0001.000
Paraphrase95.000.9650.774
Anchor 495.000.9560.741
Anchor 1695.000.9510.721

KV scores cover two contexts/20 queries. Rank/mask agreement with Base excludes sinks and averages six contexts. All variants preserve 95% KV quality, with mean rank correlations above 0.95 (B.7 Table 19).

Section 2.3 clarifies target positions and observation order.

We hope the demonstrated memory savings, reliable joint pruning and observation robustness support a favorable reassessment.

I appreciate the authors' effort for the thorough rebuttal that addresses most of my concerns. As a result, I increased my rating for the submission.

AuthorsOpenReview ↗

Thank You for Your Thoughtful Review and Positive Reassessment

Thank you very much for your thoughtful reassessment and encouraging feedback! We are delighted that the clarifications and additional experiments addressed most of your concerns. Your constructive questions helped us strengthen the validation and improve the clarity of the paper. We sincerely appreciate your time, careful consideration, and engagement throughout the discussion.

Reviewer_qSMYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a method of removing KV entries considering the impact of victim candidate on the final hidden representation of the model. In order to estimate the impact in a low cost manner, the paper adopts a random projection of output vector to a scalar and its back-propagation. A shared physical KV budget is used when selecting victim candidates in terms of layer, head, and token. The experiments report that even 80% reduction of KV cache offers competitive quality to the original KV cache.

Critical strengths

A nice idea of evaluating the impact of KV eviction on the model output without explicitly calculating the Jacobian. Good quality of 80% compressed cases.

Critical weaknesses

The proposed work is similar to KVGrad in terms of backprop with a scalar objective to estimate the sensitivity though there is some difference like random vector probe. In addition to the existing iso-capacity comparison, it would be useful to compare the two methods in terms of forward/backward compute cost.

It would be nice to clarify whether selection time includes forward and VJP backprop compute time and the difference between the timings in Fig. 5 and Table 11 (in supp.).

Questions for authors

It would be useful to extend the first-order comparison in supplementary Section B.7 to the final second-order estimator. It is to compare the softmax-renormalized deletion perturbation with the non-renormalized AV approximation. Under identical observations, probe seeds, and physical KV budgets, we could evaluate the effects on damage correlation, saliency rankings, and downstream task scores.

It would be interesting to quantify the impact of random probe seeds on the saliency rankings and downstream task scores.

Additional ablations would be useful as follows.

  • Impact of globally estimated Jdh^2 vs locally estimated dh^2
  • Coverage effect: comparing uniformly/randomly sampled windows with early-only and late-only windows at matched observation budgets
  • One shot vs. incremental pruning (requiring new backprop)
3 comments
AuthorsOpenReview ↗

Response to Reviewer qSMY

Thank you for recognizing SWIFT's quality. We have revised both the main paper and supplement, with changes marked in blue.

1. KVGrad and complete cost

Like KVGrad, SWIFT backpropagates a scalar, but couples a stationary final-hidden fidelity quadratic with the renormalized deletion direction (Introduction). Four vectors combine before differentiation: one VJP per observation. Both use two forwards and one VJP per window, totaling 16/32/64 VJPs at 8K/16K/32K.

Qwen3-8B/A100 80GB, nested prefixes, 20% retention, shared 512-token observations and EvictCache yield three-trial median seconds:

HistoryPreparation, SWIFT / KVGrad32-request total, SWIFT / KVGrad
8K21.812 / 86.62737.340 / 102.381
16K75.554 / 285.32090.351 / 300.736
32K284.496 / 1095.864300.841 / 1111.953

Preparation includes prefill, all forwards/VJPs, scoring, max, ranking and gathering. Totals add cloning, queries, decoding and transfer (32 requests, two alternating queries, 16 tokens each). Loading, input preparation, warm-up and validation are excluded. No CUDA Graphs. SWIFT reduces preparation by 74–75% and total time by 64–73%. Call counts and measured runtime are reported separately. Section 3.4 and B.8 Tables 23–24 report these costs.

The main benefit is quality with less KV memory. Figure 6 (Qwen3-8B, 24.7K, 20% retained) shows 3.42→0.71 GiB (~79% less) and 16.39→14.42 ms/token (~12% less) in steady decoding. Pruning reduces history attention and cache traffic, while other model computation remains. Memory savings apply once packed, while scoring/setup need enough later decoding to amortize.

Figure 5's SWIFT/KVGrad timings exclude scoring. Original Table 11 (now 13) uses four prefixes, 64-token observations and averaging.

2. Deletion direction in the final estimator

The final second-order estimator changes only direction, fixing observations, random vectors and KV budgets. On three Qwen3-0.6B/512-token PG19 contexts, mean KL at 128 held-out continuation positions is:

RetentionRenormalizedAV
20%0.1239990.141161
50%0.0153740.017043
80%0.0015420.001578

Renormalization improves all three budget means and 7/9 context-budget pairs (B.7 Table 15). Twelve prespecified head-specific deletions on Qwen3-8B/4K give damage Spearman 0.8112 vs. 0.4895, FP64-verified on the same observation (B.4). On 26 RULER-4K examples, SWIFT–AV non-sink Spearman is 0.9808 and mask Jaccard 0.8049 (B.7).

3. Downstream versus local scoring

Local retains direction/projection, replacing downstream sensitivity with normalized local perturbation norm squared. This isolates downstream sensitivity. Qwen3-8B RULER-4K means across evaluation groups are:

MethodKV retainedMean score
Full100%87.50
SWIFT20%88.46
AV20%84.62
Local20%84.62
KVGrad20%83.65

SWIFT leads by about 3.8 points over AV/Local and 4.8 over KVGrad, supporting the scoring design (B.7 Table 16, Section 3.2).

4. Random probe seeds

With Qwen3-8B, four vectors, 26 fixed RULER-4K examples and 20% retention, seeds 42/43/44 score 88.46, with identical per-example task scores. Cross-seed non-sink Spearman is 0.930, and mask Jaccard 0.646.

Three-seed means for 1/2/4/8/16/32 vectors are 88.33/88.46/88.46/91.03/87.82/88.14. Four is a stable default, eight a strong alternative with the highest mean here. Four/eight exceed the 16/32 means here. All use one VJP per observation (B.7 Table 17).

5. Equal-budget observation windows

On six SCBench contexts, Qwen3-8B and 20% retention, compressed conditions use four complete 512-token later windows attending to the full cache. Uniform is evenly spaced, early/late select the first/last four, and random samples without replacement. Probe seeds bind to window IDs independently of sampling.

KV scores on two contexts and 20 queries are:

FullUniformEarly-onlyLate-onlyRandom 11Random 29Random 47
9595953040565

Uniform preserves Full's quality, ties early-only, and exceeds late-only and all random groups, isolating a window-location effect (B.7 Table 18).

6. One-shot versus incremental pruning

For 100%→50%→20% of original KV, stage two recomputes forwards/VJPs/scores on retained KV with current outputs as reference. Deleted entries cannot return.

On 26 RULER-4K examples, both score 88.46, tying per example. Mean preparation time, summed over synchronized stages, is 8.04/13.22 s, with 204/408 VJPs. One-shot saves 39.21% while matching task scores here (B.7 Table 20). Both fix observations, seeds, sinks and final budget.

We hope these results support a favorable reassessment.

Thank the authors for the rebuttal efforts. Thanks to the rebuttal, I think I can maintain the current positive evaluation.

AuthorsOpenReview ↗

Thank You for Your Encouraging Feedback and Support

Thank you very much for your encouraging feedback and continued support! We sincerely appreciate the time and care you devoted to reviewing our work and rebuttal. Your constructive suggestions helped us strengthen the experimental validation and improve the clarity of the paper. We are grateful for your thoughtful engagement throughout the discussion.

Reviewer_JqE5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SWIFT, a query-agnostic KV-cache eviction method that estimates token importance using second-order output sensitivity. The method combines an exact softmax-renormalized perturbation for single-token eviction with a stationary self-fidelity objective, allowing eviction damage to be approximated by a quadratic form. Matrix-free random probes are used to estimate this quantity without explicitly constructing the Jacobian or Hessian. The resulting importance scores are aggregated across context observations and used to allocate a shared physical KV budget across layers, KV heads, and token positions.

The authors evaluate SWIFT on three 7–8B language models and several long-context benchmarks, including PG19, LongBench, RULER, and SCBench. The results show improvements over multiple KV-cache compression baselines, particularly under aggressive retention budgets. The paper also reports physical KV-cache reduction and steady-state decoding speedups.

Overall, the paper addresses a relevant problem and presents promising empirical results. However, the technical novelty of the proposed scoring formulation, the validity of independently estimated eviction scores under aggressive joint pruning, and the practical end-to-end efficiency benefits are not sufficiently established.

Critical strengths

S1. Relevant problem formulation and principled importance estimation.

The paper identifies meaningful limitations of attention-based KV eviction, particularly the failure to account for softmax renormalization and downstream model sensitivity. Combining an exact single-token attention perturbation with an output-fidelity objective provides a reasonable formulation for estimating eviction importance.

S2. Competitive empirical results across multiple models and benchmarks.

The evaluation includes three model families and comparisons with a relatively broad set of KV-cache compression methods, including KVGrad, OBCache, and KVzip. SWIFT achieves competitive or leading results across language-model fidelity, long-context tasks, and multi-query cache reuse, particularly at 20% KV retention.

S3. Physical KV-cache compression and deployment evaluation.

Unlike approaches that evaluate only logical sparsity, the paper implements physical KV eviction and reports actual memory reduction and decoding latency. The multi-query evaluation is also relevant to the proposed query-agnostic compression setting.

Critical weaknesses

W1. Limited Methodological Novelty Relative to Existing Gradient- and Curvature-Based KV Eviction Methods

The proposed method shares substantial conceptual and methodological similarities with existing KV-cache eviction approaches. In particular, KVGrad already introduces query-agnostic importance estimation based on the downstream sensitivity of final hidden representations, using gradient-based scoring, context-reconstruction probes, and global KV selection. Similarly, OBCache formulates KV pruning through second-order perturbation analysis, while KVzip establishes the context-reconstruction framework for query-agnostic cache compression.

Against these closely related works, SWIFT's main methodological distinction appears to be the combination of exact softmax-renormalized single-token removal perturbations with a second-order self-fidelity objective estimated through random projections. While this combination is reasonable, the individual ingredients largely build upon established techniques.

More importantly, the paper does not sufficiently isolate the benefits of these modifications. It remains unclear whether the improvements arise primarily from the second-order sensitivity formulation, the more accurate local removal perturbation, or differences in observation and score aggregation. A controlled comparison with KVGrad under matched observation protocols, probe budgets, and computational costs would be particularly important for establishing the incremental methodological contribution of SWIFT.

W2. Insufficient Validation of the Importance Estimator Under Aggressive Joint Eviction

The proposed importance score is derived from the perturbation caused by removing an individual KV pair. However, the final compression policy independently ranks these scores and simultaneously removes a large fraction of cached entries. At 20% retention, interactions among the removed KV pairs, including changes in attention normalization and downstream sensitivity, may be substantial. The paper does not sufficiently justify why independently estimated single-pair importance scores remain reliable under such aggressive joint eviction.

Moreover, the saliency consistency experiment in Section 3.1 differs substantially from the actual deployment setting. The reported correlations are obtained on Qwen3-0.6B with a 512-token context, using 32 probes and interventions that remove a position across attention heads. In contrast, the main method uses a substantially smaller probe budget and performs head-specific global allocation on larger models and longer contexts.

Consequently, the extremely high correlations reported in Figure 2 do not directly establish the accuracy or stability of the importance estimates used in the main experiments. Additional validation under the actual scoring configuration, including sensitivity to probe count and the effects of joint eviction, would substantially strengthen the paper.

W3. Incomplete End-to-End Efficiency Evaluation and Limited Evidence of Scalability

Although SWIFT demonstrates physical KV-cache compression and decoding acceleration, the reported steady-state latency measurements exclude saliency computation, KV gathering, and other one-time setup costs. This distinction is particularly important because SWIFT requires additional observation passes and gradient-based importance estimation before the compressed cache can be used. While the paper reports relatively low context-level selection costs, Figure 5 also includes estimated rather than directly measured selection times for some comparisons. Consequently, the overall trade-off between compression overhead and subsequent decoding savings is not fully characterized. A more complete end-to-end latency evaluation, including the amortization of selection overhead across different numbers of queries, would help establish the practical efficiency benefits.

Furthermore, the downstream evaluation is relatively limited in its coverage of challenging long-context reasoning tasks. LongBench is evaluated on a subset of 160 examples, while RULER is restricted to 4K contexts. Although the physical deployment experiments include longer contexts, they primarily demonstrate decoding speedups rather than task performance under more demanding long-context settings. In particular, the paper does not evaluate more challenging benchmarks such as LongBench v2, which would provide stronger evidence of the method's ability to preserve information required for complex multi-document understanding and long-context reasoning.

Since SWIFT performs query-agnostic KV eviction before observing downstream questions, evaluating whether the compressed cache can preserve information relevant to diverse future queries is particularly important. Additional experiments on LongBench v2, with results broken down by context length and task category, would strengthen the evidence for the method's generalization capabilities.

Questions for authors

Q1. What is the incremental benefit of the second-order self-fidelity score over simpler first-order gradient-based or output-sensitivity-based alternatives, under matched observation and computational budgets? Could the authors provide an ablation isolating this contribution from exact attention perturbation modeling and global budget allocation?

Q2. How well do the estimated importance scores predict actual output damage when multiple KV pairs are removed simultaneously? In particular, do the high correlations reported in Section 3.1 persist under the practical four-probe, head-specific scoring configuration and aggressive retention budgets?

Q3. What is the end-to-end latency improvement when the complete KV selection and physical gathering overhead is included? How many subsequent queries or generated tokens are required to amortize this overhead, and how does this break-even point change with context length?

4 comments
AuthorsOpenReview ↗

Response to Reviewer JqE5

Thank you for recognizing SWIFT's formulation and quality. We have revised both the main paper and supplement, with changes marked in blue.

W1/Q1. Contribution and component benefits

KVGrad uses downstream gradients, OBCache local output damage, and KVzip reconstruction observations. SWIFT evaluates a stationary final-hidden self-fidelity quadratic along the exact softmax-renormalized deletion direction. The dense-point first derivative vanishes. Combined vectors estimate the leading quadratic with one VJP, retaining within-candidate cross terms (Introduction).

Only the deletion direction changes, fixing observations, vectors, KV budgets, aggregation and allocator. On three Qwen3-0.6B/512-token PG19 contexts, renormalization improves held-out KL at all three budgets and 7/9 context-budget pairs. At 20% retention, mean KL is 0.123999 versus 0.141161 for AV (B.7, Table 15).

B.7, Table 16 reports Qwen3-8B RULER-4K means across evaluation groups:

MethodKV retainedMean score
Full100%87.50
SWIFT20%88.46
AV20%84.62
Local20%84.62
KVGrad20%83.65

AV replaces the direction. Local retains direction/projection but replaces downstream sensitivity with normalized local perturbation norm squared. SWIFT leads by about 3.8 points over AV/Local and 4.8 over KVGrad (Section 3.2). Compute costs are measured below.

W2/Q2. Joint eviction and probe configuration

One Qwen3-8B/4K context uses four combined vectors, eight 512-token observations and allocation by KV head. Each budget compares nine sets: SWIFT, KVGrad, AV, two random, and four nearby SWIFT sets. Summed deleted-pair scores are compared with measured joint hidden-output damage on the prespecified first observation:

RetentionNine-set SpearmanSWIFT + four nearby sets
20%0.88330.9000
50%0.98331.0000
80%0.93330.9000

Scores rank joint damage at 80% deletion, including nearby sets (Section 3.1, B.4 Table 7). A 12-candidate singleton check gives Spearman 0.8112 in this configuration.

On 26 fixed RULER-4K examples, three seeds and 20% retention, means for 1/2/4/8/16/32 combined vectors are 88.33/88.46/88.46/91.03/87.82/88.14. Four scores 88.46 across seeds, supporting a stable default. Eight's highest mean (91.03) makes it a strong alternative here. Four/eight exceed the 16/32 means here. All use one VJP per observation (B.7 Table 17).

W3/Q3. Complete cost and amortization

Qwen3-8B/A100 80GB timings use nested context prefixes, 20% retention, shared 512-token observations and EvictCache, without CUDA Graphs. Three-trial median seconds:

HistoryPreparation, SWIFT / KVGrad32-request total, SWIFT / KVGrad
8K21.812 / 86.62737.340 / 102.381
16K75.554 / 285.32090.351 / 300.736
32K284.496 / 1095.864300.841 / 1111.953

Preparation includes prefill, all forwards/VJPs, scoring, aggregation, ranking and gathering. Totals add query processing, cache cloning, generation and output transfer. Two queries alternate, each generating 16 tokens. SWIFT reduces preparation by 74–75% and total cost by 64–73% versus KVGrad. Full also reuses its cache. At 8K/16K/32K, SWIFT does not break even against Full at tested 1/2/4/8/16/32-request prefixes.

The main benefit is quality with less KV memory. Figure 6 (Qwen3-8B, 24.7K, 20% retained) shows 3.42→0.71 GiB (~79% less) and 16.39→14.42 ms/token (~12% less) in steady decoding. Pruning reduces history attention and cache traffic, while other model computation remains. Memory savings apply once packed, while scoring/setup need enough later decoding to amortize.

Figure 5's SWIFT/KVGrad timings exclude scoring. Original Table 11 (now Table 13) uses a different 64-token development protocol. Section 3.4 and B.8 Tables 23–24 report complete costs and stages.

W3. Harder long-context tasks

We evaluate all 116 official-short LongBench v2 inputs with document ≥8K and complete prompt plus generation budget ≤32K, without truncation or correctness filtering. Selection sees no questions, options, or answers. Correct-answer counts are:

SubsetFullSWIFT, 20%KVGrad, 20%
All47/11648/11645/116
8K–16K11/2912/2910/29
16K–32K36/8736/8735/87
Single-document QA24/6128/6126/61
Multi-document QA14/3212/3211/32
Dialogue2/121/121/12
In-context learning5/75/75/7
Code2/32/32/3
Structured data0/10/10/1

SWIFT achieves 41.38%, versus Full's 40.52% and KVGrad's 38.79%, retaining only 20% KV. Section 3.3 and B.6 Table 12 report these results. We hope these results support a favorable reassessment.

Thank you for the detailed rebuttal and additional experiments. While I appreciate the authors' efforts, the rebuttal does not substantially change my overall assessment of the paper. Therefore, I will maintain my original score.

AuthorsOpenReview ↗

Follow-up on Remaining Concerns

Thank you for taking the time to review our rebuttal and additional experiments. We appreciate your consideration and would welcome any further guidance on which aspects you find insufficiently addressed.

We added validation under the practical Qwen3-8B, four-vector, head-specific configuration, obtaining a Spearman correlation of 0.8833 with measured joint-eviction damage at 80% deletion, and 0.9000 among SWIFT and nearby selections. On 116 complete LongBench v2 inputs, SWIFT retains only 20% of KV while achieving 41.38% accuracy, compared with 40.52% for Full and 38.79% for KVGrad.

If you have time to elaborate on the remaining concerns, we would be grateful for the opportunity to provide a focused clarification. Thank you again for your time and feedback.

AuthorsOpenReview ↗

Further Clarification: Effect of Deletion Direction on Joint Eviction Damage

We would like to further clarify one point connecting W1 and W2: does modeling the physical deletion direction more accurately lead to a cache that better preserves the model output?

On one Qwen3-8B/4K context, we compared SWIFT’s softmax-renormalized direction with the AV approximation while keeping downstream sensitivity fixed. Across 12 prespecified single-entry deletions, the Spearman correlation with measured damage was 0.8112 for SWIFT versus 0.4895 for AV (Appendix B.4, paragraph following Table 7).

Our earlier reply reported ranking correlations but did not show the damage incurred by the caches each variant actually selected. Using the same allocation rule and KV budget, we measured the final hidden-state change after joint deletion on the first reconstruction observation (lower is better):

KV retainedSWIFT directionAV approximation
20%0.0582690.065349
50%0.0092300.010372
80%0.0002160.000257

The renormalized direction thus did more than improve the singleton ranking: on this context, it selected caches with lower measured damage at all three budgets, including at 80% deletion. We hope this clarifies the practical value of the scoring design. Thank you again for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.