- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes SWIFT, a task-free KV-cache eviction method that estimates the output corruption caused by removing individual KV pairs. SWIFT defines a self-fidelity objective against the detached output of the uncompressed model and combines exact softmax-renormalized eviction perturbations with a matrix-free second-order approximation, avoiding explicit Jacobian or Hessian construction. The resulting scores are comparable across layers, KV heads, and token positions, enabling allocation of a single global physical KV budget. Experiments show that the proposed importance score is highly correlated with measured singleton-eviction damage and that the resulting compressed caches preserve strong downstream accuracy while reducing physical KV memory and steady-state decoding latency.
Critical strengths
A principled formulation enabling global KV-budget allocation. The proposed method provides task-free and globally comparable importance scores, enabling a single physical KV budget to be allocated jointly across layers, KV heads, and token positions rather than relying on predetermined per-layer or per-head quotas. This capability is supported by a principled second-order formulation derived naturally from a self-fidelity objective based on output damage. By combining exact softmax-renormalized eviction perturbations with an efficient matrix-free estimator, SWIFT translates this formulation into a practical scoring procedure without explicitly constructing the Jacobian or Hessian.
Critical weaknesses
-
Concerned about the practical efficiency of SWIFT, not sufficiently established by the reported latency analysis. Unlike methods that derive eviction scores from a single prefill, SWIFT appears to require additional observation forwards, each attending to the full cached context and requiring a VJP. However, Figure 5 reports only of context-level pruning time, without a sufficiently precise accounting of which operations are included, while Supplementary Table 11 reports approximately for full-coverage selection on an 8K context. Moreover, the reported decoding speedups exclude selection, cache gathering, query appending, and graph setup. Without a consistent end-to-end latency breakdown and amortization analysis, it is difficult to determine when SWIFT provides a net practical benefit over less expensive eviction methods.
-
The observation-based scoring protocol is insufficiently specified and validated despite being central to the method. SWIFT’s scores depend on synthetic repeat observations, yet the exact instruction , placement of the 8-token anchor, and use of model-specific chat templates are not provided. Since these choices directly affect attention patterns, gradients, and ultimately the global KV ranking, this omission limits reproducibility and leaves the robustness of the proposed task-free score unclear. More broadly, several definitions required to understand the scoring procedure and tensor aggregation are deferred to the supplementary material, making the main paper insufficiently self-contained.
Questions for authors
Q1. End-to-end selection cost
For a context of length , SWIFT appears to process additional repeat observations after the initial dense prefill, with each observation attending to the full cached context and requiring a VJP. Does the pruning time in Figure 5 include all observation forwards, VJPs, candidate-score computation, max aggregation, global ranking, and physical KV gathering? Please also clarify how these results reconcile with the full-coverage selection times reported for an 8K Qwen3-8B context in Supplementary Table 11.
A latency breakdown relative to the original context prefill, reported as a function of context length, comparing to other baselines would materially affect my assessment.
Q2. Amortized practical benefit
Since the decoding speedups exclude one-time scoring and cache construction, how many generated tokens or repeated queries are required for SWIFT to break even relative to the full-cache baseline and lower-cost eviction methods? Please report end-to-end latency including prefill, SWIFT scoring, physical gathering, query processing, and decoding for representative single-query and multi-query workloads.
Q3. Singleton scores under joint eviction
SWIFT ranks candidates using singleton-eviction scores but removes up to of the KV pairs simultaneously without recomputing interactions among removed candidates. How well do singleton scores predict damage under such large joint interventions? An analysis comparing the summed or ranked singleton estimates with measured joint-eviction damage at different retention ratios would clarify the validity of the global one-shot allocation.
Q4. Exact observation construction and robustness
What is the exact repeat instruction , including the placement of the preceding segment’s 8-token anchor and any model-specific chat templates? Is the same instruction used across Qwen, Mistral, and Llama?
3 comments
I appreciate the authors' effort for the thorough rebuttal that addresses most of my concerns. As a result, I increased my rating for the submission.
Thank You for Your Thoughtful Review and Positive Reassessment
Thank you very much for your thoughtful reassessment and encouraging feedback! We are delighted that the clarifications and additional experiments addressed most of your concerns. Your constructive questions helped us strengthen the validation and improve the clarity of the paper. We sincerely appreciate your time, careful consideration, and engagement throughout the discussion.
Response to Reviewer cP8c
Thank you for recognizing SWIFT's formulation. We have revised both the main paper and supplement, with changes marked in blue.
Q1. Complete selection cost
SWIFT reduces complete preparation time by 74–75% versus KVGrad. Figure 5's SWIFT/KVGrad milliseconds cover selection/gathering from precomputed scores, excluding observation forwards/VJPs. Original Table 11 (now Table 13) uses four prefixes, 64-token observations and averaging, unlike the final 512-token/max protocol.
Qwen3-8B/A100 80GB timings use nested prefixes, 20% retention, shared full-coverage observations and EvictCache, without CUDA Graphs. Both methods use two forwards and one VJP per observation, totaling 16/32/64 VJPs at 8K/16K/32K. SWIFT combines four vectors before differentiation. Three-trial medians:
Scoring includes all forwards/VJPs, scores and max aggregation. Preparation adds ranking, gathering and request setup. Ratios use unrounded medians. Section 3.4 and B.8 Tables 23–24 report costs and stages. All trials are retained, with ranges in B.8.
Q2. Single-query and multi-query benefit
One preparation serves 32 requests (two alternating queries, 16 tokens each). Full reuses its cache with native FlashAttention-2. Totals include preparation, cloning, query processing, decoding and transfer, excluding loading, input preparation, warm-up and validation:
SWIFT saves 64–73% versus KVGrad over 32 requests at the same KV budget. No Full break-even occurs at tested 1/2/4/8/16/32-request prefixes. Break-even with longer outputs or cheaper methods remains unmeasured.
The main benefit is quality with less KV memory. Figure 6 (Qwen3-8B, 24.7K, 20% retained) shows 3.42→0.71 GiB (~79% less) and 16.39→14.42 ms/token (~12% less) in steady decoding. Pruning reduces history attention and cache traffic, while other model computation remains. Memory savings apply once packed, while scoring/setup need enough later decoding to amortize.
Q3. Singleton scores under joint eviction
One Qwen3-8B/4K context uses four vectors, eight observations and KV-head scoring. Nine equal-budget sets comprise SWIFT, KVGrad, AV, two random and four nearby SWIFT sets. Summed deleted-pair scores rank measured joint hidden-output damage on the prespecified first observation:
Correlations remain 0.8833/0.9000 at 80% deletion, supporting joint-eviction ranking (Section 3.1, B.4 Table 7).
Q4. Exact observation construction and robustness
All three models encode these strings without added special tokens:
Token order: instruction → previous segment's last eight token IDs → suffix → segment. The first has no anchor. No added spaces or anchor decoding/re-encoding. Targets are current-segment final hidden positions.
SCBench Qwen3-8B extracts this suffix with thinking disabled, without wrapping each observation as a full chat message:
Mistral-7B-v0.3/Llama-3.1-8B have no chat template here and use a newline, as do independent RULER controls including Qwen (Section 2.3, A.4).
Qwen3-8B, 20% retention and four fixed later-segment windows give:
KV scores cover two contexts/20 queries. Rank/mask agreement with Base excludes sinks and averages six contexts. All variants preserve 95% KV quality, with mean rank correlations above 0.95 (B.7 Table 19).
Section 2.3 clarifies target positions and observation order.
We hope the demonstrated memory savings, reliable joint pruning and observation robustness support a favorable reassessment.