- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Lifetime, a training-free, query-aware, prefill-only KV-cache eviction method. After prefill, it computes the attention of the last query tokens over the old prefix, partitions it into contiguous bins, and obtains an "ordered attention trajectory" for each KV head. It then uses one of two branches—periodic (the phase/spacing of repeated active bins) or rising (extrapolation from the slope of the last two bins)—to estimate a "next important use" distance proxy , discounts the peak-attention rank by , linearly combines it with mean-attention rank and value-norm rank, adds 4 sink positions and 64 recent protected positions, selects top- independently per layer–head, physically gathers a short cache, and decodes directly. The authors evaluate on 16 LongBench tasks and 7 RULER-16K tasks with Qwen3-4B and Llama-3.1-8B-Instruct, and report the best scores on the LongBench average and on RULER-256.
Critical strengths
- Intuitive motivation. Mean attention conflates tokens whose attention is fading with tokens whose attention is rising; this observation is reasonable, and replacing a single mean with an ordered trajectory is a natural direction.
- Complete engineering implementation. The method physically gathers a regular short cache, uses a fixed per-head budget, and avoids decode-time rescoring. The efficiency table reports prefill, peak memory, and decode throughput, and the paper documents paired runs.
- Clear writing and figures. The method figure, algorithm box, and equations are well organized; the Limitations section also acknowledges that the proxy is a heuristic and that the model scale is limited.
Critical weaknesses
W1. The central assumption is unverified: the query-token axis is not decoding time
The paper's core narrative is that the "temporal order" of attention within the observation window can estimate the distance to each token's "next important use," thereby approximating Belady's MIN policy. But this "time" is actually the positional order of the last query tokens. Figure 1's x-axis explicitly says "Prompt-tail queries → old tokens (early → late)." These 16 rows of attention come from 16 semantically different query tokens, not from the evolution of a single query over 16 decoding steps. Interpreting as a decode-time next-use distance is therefore an unproven analogy: Belady's MIN requires a future access sequence, whereas the paper has only the order of tokens within the query, which is not the same object.
More importantly, the paper performs no direct validation:
- For each old token, record how it is actually attended to during decoding;
- Compute the correlation/AUC between and the number of steps until the token is first attended to above a threshold;
- Compare against simple baselines: last-active recency (), active count , mean attention, peak attention, and a binary flag for whether the token is active in the last bin.
Without this experiment, it is impossible to tell whether provides any information beyond simpler features such as "recently active," "repeatedly active," or "high peak attention." Downstream benchmark scores cannot substitute for this validation, because they are jointly affected by protection, the value cue, pooling, and the budget. Figure 1 is only a hypothetical toy example, not empirical evidence.
W2. Coupling between the protection set and the budget: 5% is not really 5%
The paper explicitly states that "Protected positions count toward ." With and , the budget that actually participates in scoring is only
When , the protection set alone exhausts the budget: at , , and at , . LongBench contains short tasks of roughly 1–2K tokens (e.g., code completion); at 5% retention, may be only 50–100, and when no old token is scored at all, so the method degenerates to "sink + recent." Even at , 10% retention gives , leaving only about 52 positions for scoring after subtracting 68; at 5% the number is negative. The short-task results at 5% therefore cannot serve as evidence that the scorer is effective.
In addition, the temporal score applies only to the first positions; positions through , although part of the old prefix, are covered by the protected score. This further narrows the scorer's effective range, and the paper does not mention it.
The protection window is not matched to the baselines. Appendix A.2 says SnapKV uses a 32-query observation window but does not state how many recent tokens it retains; Lifetime explicitly protects 64 recent tokens. If SnapKV retains these 32 observation tokens in its standard implementation, then on RULER-256 Lifetime devotes
of its budget to fixed protection, whereas SnapKV devotes about . This is one of the central fairness issues in the experiments.
W3. Ablation coverage and Experimental fairness
Ablation coverage is insufficient. Table 6 covers only Qwen3-4B, LongBench, and 5% retention; it includes neither RULER-256 nor Llama nor 10%. The paper's headline result is on RULER-256, yet component contributions are measured only in a completely different setting. The authors should add:
- w/o Periodic, w/o Rising, w/o Value, w/o Smoothing, and Short Recent on RULER-256;
- at least one ablation on Llama;
- at least one ablation at 10%.
The ablations are not clearly explained. The asymmetry between "w/o Periodic" and "w/o Rising" (7.85 vs. 1.76) suggests that the usage ratio of the two branches may be highly skewed; the paper reports neither branch usage nor the branch composition of the retained tokens. The β sweep is "derived from auxiliary runs rather than a direct fixed-m rerun," so it is not a controlled experiment; sensitivity to θ and ε is entirely missing; and the meaning of "same per-token permutation" in Shuffled-Time is ambiguous (do all tokens share one permutation, or does each token get its own?). As written, these ablations are not reproducible.
Tuning on the test set. were selected on LongBench Qwen3-4B at 5%/10% using Figures 3–4 and Table 6, and the same benchmark is then used to report the main results. The paper's statement of "one fixed configuration ... without any per-benchmark or per-model retuning" is true in the sense that no further tuning is done per setting, but the configuration itself was selected on the test benchmark, with no held-out dev set. This matters given that some margins are only 0.07–0.09. I recommend selecting the configuration on a disjoint dev set (e.g., a subset of tasks or a different seed) and reporting on the remaining tasks, or at least stating which data were used for configuration selection and whether they overlap with the main results.
W4. the delta relative to SnapKV / ReST-KV / Expected Attention
Each individual component—observation window, peak/mean attention, local pooling, sink+recent protection, and value norm—is already known. The new component is the periodic/rising proxy. The problems are:
- The paper does not compare against simpler temporal features such as "last-active recency," "active count," or "whether the token is active in the last bin"; "w/o Periodic" in Table 6 also conflates recurrence with phase. It is therefore impossible to tell whether phase/slope extrapolation is necessary at all. If a recurrence flag explains most of the gain, the paper's novelty is substantially lower than claimed.
- The value-norm cue overlaps with Expected Attention: Appendix A.2 explicitly says that Expected Attention uses "covariance and value-norm terms." The paper presents the value cue as one of its contributions without acknowledging this overlap and without ablating against Expected Attention's value term. This cue is not new.
- ReST-KV's "spatial-temporal smoothing" is very close to the paper's temporal trajectory; the related work mentions it in a single sentence and does not explain the essential difference between "smoothing" and "phase/slope extrapolation." The related work also discusses no prior work that uses attention slope/trend within an observation window, so the reader cannot assess the delta.
- The Belady analogy is rhetorical: Belady's MIN requires a future access sequence, whereas the paper uses the order of tokens within the query, not future accesses. "Reuse-distance prediction for hardware caches" is also defined over memory-access sequences, not query tokens. Without the predictive-validity experiment in W1, the analogy does not hold.
- The method has eight hyperparameters () plus , and its score is a hand-tuned linear combination. "Training-free" is not the same as "parameter-free," and the configuration was selected on the test benchmark. A more accurate characterization is "an effective engineering heuristic" rather than "a principled next-use estimator."
Questions for authors
- Predictive validity. Please record the actual attention during decoding on a set of prompts, compute the correlation/AUC between and "the number of future steps until the token is first attended above a threshold," and compare against last-active recency, active count, mean attention, and peak attention. Does really provide information beyond these simple features? This is the key to the paper's central claim.
- Branch usage. Please report the proportion of tokens assigned to the periodic and rising branches, the proportion of retained tokens, and the proportion with . Please add four controls—" and ," "," " globally," and " globally"—to separate the contributions of recurrence and phase/slope.
- The case in Equation (3). When , for any ; is this intentional? Please report the proportion of tokens with and its effect on top- selection. If it is not intentional, please fix the formula (e.g., penalize missed periods) and rerun.
- Budget and protection set. Please report , , and for each task; when , which positions does tie-breaking actually keep? Please add three baselines—"SnapKV + 64 recent + 4 sinks," "sink+recent only," and "random selection"—and sweep on RULER-256.
- Baseline reproduction. Please reproduce the published numbers of Expected Attention and CapKV using the official code/settings.
5 comments
Response to Reviewer qJAk (2/4)
Table R10. RULER-16K, 256 entries per KV head.
| Method | Qwen Avg. | Qwen VT | Llama Avg. | Llama VT |
|---|---|---|---|---|
| SnapKV (original) | 47.60 | 25.40 | 66.09 | 46.80 |
| SnapKV + 4 sinks + 64 recent | 55.30 | 43.00 | 68.20 | 57.40 |
| Sinks + recent only | 10.12 | 5.00 | 12.57 | 4.00 |
| 68 protected + random selection | 14.36 | 7.40 | 16.92 | 6.80 |
| Lifetime pipeline, mean-only (m = 1, β = 0) | 66.48 | 73.60 | 71.83 | 72.40 |
| Lifetime pipeline, d = 0 | 69.07 | 88.00 | 73.62 | 82.40 |
| Lifetime | 70.56 | 97.20 | 74.87 | 88.60 |
- Matched protection improves SnapKV by +7.70 / +2.11, so protection is part of the original gap. SnapKV with matched protection remains 15.26 / 6.67 below Lifetime.
- Protection alone does not explain the results: filling the remaining slots with recent or random positions gives 10–17 points.
- With protection and every other component identical, Lifetime exceeds mean-only aggregation by +4.08 / +3.04, of which the distance term accounts for +1.49 / +1.25 (VT +9.20 / +6.20).
The revision adds these rows to Section 4.3, attributes the RULER-256 margin over SnapKV to protection, the scoring pipeline, and the distance term separately, and lists every baseline's protected positions in Appendix A.2.
Recent-window size. Table R11 sweeps with the budget fixed at 256.
Table R11. Recent-window sweep on RULER-256 (4 sinks throughout).
| Recent positions R | Protected share of B | Scored slots | Qwen Avg. | Llama Avg. |
|---|---|---|---|---|
| 16 | 7.8% | 236 | 69.88 | 74.25 |
| 32 | 14.1% | 220 | 70.20 | 74.57 |
| 64 (default) | 26.6% | 188 | 70.56 | 74.87 |
| 128 | 51.6% | 124 | 69.85 | 74.12 |
Averages vary by at most 0.71 (Qwen) and 0.75 (Llama) across this range. At , whose protected share is close to SnapKV's, Lifetime scores 70.20 / 74.57, versus 47.60 / 66.09 for SnapKV.
W3 / Q2. Ablations, branch usage, and configuration selection
Coverage. Table R12 extends the ablations to RULER-256 on both models and to LongBench at 10%, changing one component at a time.
Table R12. Component ablations.
| Variant | RULER-256 Qwen Avg. (VT) | RULER-256 Llama Avg. (VT) | LongBench Qwen 5% | LongBench Qwen 10% |
|---|---|---|---|---|
| Lifetime | 70.56 (97.20) | 74.87 (88.60) | 45.13 | 47.90 |
| w/o Periodic | 65.76 (75.60) | 72.41 (74.00) | 37.28 | 46.88 |
| w/o Rising | 69.81 (94.40) | 74.31 (86.00) | 43.37 | 47.57 |
| w/o Value | 69.76 (94.80) | 74.15 (85.20) | 44.02 | 47.61 |
| w/o Smoothing | 69.31 (92.80) | 73.98 (83.80) | 43.07 | 47.42 |
| Short recent (R = 16) | 69.88 (94.80) | 74.25 (86.60) | 44.91 | 47.74 |
Every variant is below Lifetime in every setting.
Branch usage and the Periodic/Rising asymmetry. On RULER-256 (Table R3), the periodic branch covers 8.4% / 7.9% of candidates but 55.2% / 52.6% of the retained scored entries, and holds for 4.0% / 3.7% of candidates and 34.7% / 32.9% of retained entries. The w/o Periodic variant sends the periodic-branch tokens through the rising branch, which assigns to any token whose attention did not increase between the last two bins. Repeatedly active tokens then lose most of their peak weight. This is why removing the periodic branch costs far more than removing the rising branch, on RULER-256 (−4.80 vs. −0.75 on Qwen) as on LongBench (−7.85 vs. −1.76).
Separating recurrence from phase and slope. Table R13 changes only the distance in ; all other components are fixed.
Table R13. Controls that change only d (RULER-256).
| Distance d | Qwen Avg. | Qwen VT | Llama Avg. | Llama VT |
|---|---|---|---|---|
| Rising branch for all tokens (w/o Periodic) | 65.76 | 75.60 | 72.41 | 74.00 |
| Last-bin active flag | 68.39 | 85.20 | 73.36 | 81.00 |
| d = 0 for all tokens | 69.07 | 88.00 | 73.62 | 82.40 |
| Active count, d = T − c | 69.33 | 89.00 | 73.90 | 83.20 |
| Recency, d = δ | 69.73 | 90.80 | 74.25 | 84.20 |
| Phase if c ≥ 2, d = T otherwise (w/o Rising) | 69.81 | 94.40 | 74.31 | 86.00 |
| Phase if c ≥ 2, slope otherwise (Lifetime) | 70.56 | 97.20 | 74.87 | 88.60 |
Response to Reviewer qJAk (3/4)
- Recurrence alone explains little: scoring tokens by how often they were active () adds +0.26 / +0.28 over (VT +1.00 / +0.80).
- Each branch alone falls short. Phase alone is on par with recency on average (69.81 vs. 69.73; 74.31 vs. 74.25) and higher on VT (+3.60 / +1.80). Slope alone falls below .
- Together, the two branches exceed every simple alternative, including the EMA score (69.89 / 74.28, Table R2), by at least +0.59 on average and +4.30 on VT.
The gain therefore requires both the phase estimate for recurring tokens and the slope estimate for the others.
Sensitivity to θ, ε, and β.
Table R14. Single-parameter sweeps (Qwen3-4B, RULER-256 average; other parameters fixed).
| Parameter | Values | Score |
|---|---|---|
| θ | 0.90 / 0.95 / 0.97 | 70.11 / 70.56 / 70.18 |
| ε | 10⁻⁷ / 10⁻⁶ / 10⁻⁵ | 70.50 / 70.56 / 70.22 |
Every setting stays above the strongest temporal alternative (EMA, 69.89). The default was missing from Section 3.4 and is now listed. For β, the controlled endpoints are in Table R12: β = 0 at fixed m = 0.35 (w/o Value) loses 1.11 / 0.29 on LongBench Qwen 5% / 10% and 0.80 / 0.72 on RULER-256. The revision replaces Figure 3(b) with a direct sweep at fixed and makes every sensitivity plot a single-parameter sweep.
Shuffled-Time. Within each layer–KV-head pair, all tokens share one permutation of the 8 bins, applied to both the raw and the rank-normalized trajectories. Each bin moves intact, so within-bin ranks and all order-invariant statistics are preserved and only the temporal order changes. Results average two permutation seeds.
Configuration selection. The configuration was selected on Qwen3-4B LongBench, so the Qwen3-4B LongBench rows of Table 1 overlap with the selection data. No Llama-3.1-8B result and no RULER result was used for selection. Given this overlap, the revision describes the LongBench results as parity with the strongest baselines and rests the evidence for temporal scoring on the held-out settings, with all other components matched:
- Llama-3.1-8B LongBench: the distance term adds +0.28 (10%) and +0.34 (5%) over .
- RULER-256 on both models: +1.49 / +1.25 over , and VT +9.20 / +6.20.
The revision states the selection data and rewrites the sentence "without any per-benchmark or per-model retuning" accordingly.
W4. Delta relative to SnapKV, ReST-KV, and Expected Attention
Simpler temporal features. Tables R1–R2 and R13 compare Lifetime with recency, active count, the last-bin flag, and EMA in an otherwise identical pipeline, and separate recurrence from phase and slope. Lifetime is above all of them on LongBench and RULER-256, by at least +0.59 on the RULER-256 average and +4.30 on VT, and recurrence alone accounts for little of the gain.
Value cue. You are right that Expected Attention uses value norms, and CriticalKV also accounts for value states. The revision credits this prior work and removes the value cue from our contributions. Our estimates of the temporal term hold the value cue fixed (row (b) in Tables R1–R2), so they do not depend on it.
ReST-KV's temporal smoothing. ReST-KV smooths scores with an exponential moving average (coefficient 0.3 in our runs). This weights later evidence more but returns one scalar per token, so tokens with different trajectories can receive the same score. Lifetime instead reads two properties of the trajectory, the spacing of repeated activations and the slope at the boundary, and uses them to discount the peak rank. The EMA control isolates this difference: with ReST-KV's coefficient and every other component identical, Lifetime is higher by 0.15–0.22 on LongBench and 0.67 / 0.59 on RULER-256 (VT +5.00 / +4.30). The revised related work explains this distinction and compares Lifetime with other aggregations of observation-window attention (mean, EMA smoothing, and worst-case aggregation).
Belady's MIN and reuse distance. We agree that Belady's MIN requires a future access sequence and that reuse-distance prediction is defined over memory accesses, not query tokens. The revision keeps both as motivation only, removes the sentence placing Lifetime "closer to reuse-distance prediction," and supports the proxy with the direct test in W1 / Q1.
Hyperparameters and characterization. We agree that training-free is not parameter-free and that the score is a hand-set linear combination. The revision describes Lifetime as a training-free heuristic with one fixed configuration and lists all nine hyperparameters, including ε. Sensitivity is reported for m and β (Figure 3 and the direct β sweep), W and T (Table R6), θ and ε (Table R14), and R (Table R11). k = 5 matches SnapKV's pooling kernel, and S = 4 matches the four sinks used by StreamingLLM and CapKV.
Response to Reviewer qJAk (1/4)
Thank you for the detailed review and for specifying the analyses that would test each claim. We answer each point below and repeat the key numbers from the General Response where they apply.
W1 / Q1. Does d predict decoding attention beyond simple features?
You are right that prompt-tail positions are not decoding steps, and that downstream scores cannot replace a direct test, since protection, the value cue, pooling, and the budget also affect them. We ran the test you proposed (setup in G3). During Full-KV decoding, a token counts as accessed when it enters the top 5% of old-prefix attention in a layer–KV-head pair, and the label is whether the number of steps until its first access is at most 8. Table R4 compares 1/(1+d) with the five baselines you listed:
So d alone does not predict better than static importance, and Lifetime does not rank by it alone: d discounts the peak rank, which is then blended with the mean rank. The test shows that the spacing and slope features behind d carry information about near-term decoding attention beyond recency, count, last-bin activity, mean, and peak.
On the eviction side, the matched controls change only d and hold protection, window, pooling, value cue, and budget fixed (Tables R1–R2 and R13 below). Lifetime exceeds d=0 by +1.49 / +1.25 on RULER-256 (VT +9.20 / +6.20), and every simple temporal alternative by at least +0.59 on average (VT +4.30).
The revision presents d as an order-aware proxy for near-term relevance rather than a count of decoding steps, keeps Belady's MIN as motivation only, adds this test to the paper, and labels Figure 1 as a schematic.
W2 / Q4. Protection, budget accounting, and matched baselines
Scored budget and range. You are right on both points. Protected positions come out of B, so Bscored=max(0,B−68). The 64 recent positions also cover the last 48 old-prefix positions, so only the N−68 unprotected positions compete for the scored slots.
Table R9. Scored budget under relative retention.
The scorer is inactive (Bscored=0) for N<1,380 at 5% and N<690 at 10%; Lifetime then keeps only sinks and recent positions. These instances make up 3.8% / 3.5% (Qwen / Llama) of LongBench at 5% and 0.6% / 0.5% at 10%, and we agree they give no evidence for the scorer. On RULER, B=256 for every instance, so Bscored=188 throughout; the RULER-256 comparisons below therefore test the scorer at the same nonzero scored budget on every instance. The revision reports N, B, and Bscored for every task and states the 48-position overlap in Section 3.4.
When the protected set does not fit. This occurs for N<1,360 (5%) or N<680 (10%). In the submitted runs, the backend's top-k tie handling chose among the equal protected scores, so the kept positions were implementation-defined. The revision fixes the rule: keep the 4 sinks first, then the most recent positions in reverse order until B is reached. All matched controls use this rule.
Protection matched to SnapKV. In our KVPress implementation, SnapKV keeps its 32-query observation window and no sinks: 32 of 256 entries (12.5%), versus 68 (26.6%) for Lifetime. Table R10 adds the three baselines you requested; every row keeps exactly 256 entries per KV head.