acceptodds
Under review as a conference paper at ICLR 2027

Lifetime: KV Cache Eviction from Attention Trajectories for Long-Context Language Models

Abstract

Long-context language models store a key–value (KV) cache whose size grows linearly with the sequence length and quickly dominates inference memory and bandwidth. Training-free prefill eviction reduces this cost, but observation-window scorers typically collapse prompt-tail attention into a single static statistic, discarding its temporal order and conflating tokens whose attention is fading with tokens whose attention is recurring or rising. We introduce , a query-aware, prefill-only method that keeps a temporally binned, ordered attention trajectory for every KV head. Using only evidence available once prefill is complete, estimates a proxy for the proximity of each token's next important use, weights its peak attention rank by this proxy, stabilizes the result with mean-attention rank, and adds a lightweight value-magnitude cue. Every layer–head pair then retains the same number of independently selected entries, which are physically gathered into a regular shortened cache and decoded without rescoring or further eviction. On 16 LongBench tasks, attains the highest average among compressed methods at both and retention on Qwen3-4B ( and ) and Llama-3.1-8B-Instruct ( and ). On RULER-16K with 256 retained entries per layer–KV-head pair, it exceeds the strongest alternative by and points on the two models, while adding less than prefill latency over SnapKV. These results show that prompt-tail temporal structure is a practical, training-free signal for aggressive one-shot KV-cache compression.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache compression, KV cache eviction, long-context inference, large language models, training-free, attention trajectory, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qJAkOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Lifetime, a training-free, query-aware, prefill-only KV-cache eviction method. After prefill, it computes the attention of the last W=16W=16 query tokens over the old prefix, partitions it into T=8T=8 contiguous bins, and obtains an "ordered attention trajectory" for each KV head. It then uses one of two branches—periodic (the phase/spacing of repeated active bins) or rising (extrapolation from the slope of the last two bins)—to estimate a "next important use" distance proxy dh,id_{h,i}, discounts the peak-attention rank Ph,iP_{h,i} by 1/(1+dh,i)1/(1+d_{h,i}), linearly combines it with mean-attention rank and value-norm rank, adds 4 sink positions and 64 recent protected positions, selects top-BB independently per layer–head, physically gathers a short cache, and decodes directly. The authors evaluate on 16 LongBench tasks and 7 RULER-16K tasks with Qwen3-4B and Llama-3.1-8B-Instruct, and report the best scores on the LongBench average and on RULER-256.

Critical strengths
  • Intuitive motivation. Mean attention conflates tokens whose attention is fading with tokens whose attention is rising; this observation is reasonable, and replacing a single mean with an ordered trajectory is a natural direction.
  • Complete engineering implementation. The method physically gathers a regular short cache, uses a fixed per-head budget, and avoids decode-time rescoring. The efficiency table reports prefill, peak memory, and decode throughput, and the paper documents paired runs.
  • Clear writing and figures. The method figure, algorithm box, and equations are well organized; the Limitations section also acknowledges that the proxy is a heuristic and that the model scale is limited.
Critical weaknesses

W1. The central assumption is unverified: the query-token axis is not decoding time

The paper's core narrative is that the "temporal order" of attention within the observation window can estimate the distance to each token's "next important use," thereby approximating Belady's MIN policy. But this "time" is actually the positional order of the last W=16W=16 query tokens. Figure 1's x-axis explicitly says "Prompt-tail queries → old tokens (early → late)." These 16 rows of attention come from 16 semantically different query tokens, not from the evolution of a single query over 16 decoding steps. Interpreting dh,id_{h,i} as a decode-time next-use distance is therefore an unproven analogy: Belady's MIN requires a future access sequence, whereas the paper has only the order of tokens within the query, which is not the same object.

More importantly, the paper performs no direct validation:

  • For each old token, record how it is actually attended to during decoding;
  • Compute the correlation/AUC between dh,id_{h,i} and the number of steps until the token is first attended to above a threshold;
  • Compare against simple baselines: last-active recency (T−1−ℓh,iT-1-\ell_{h,i}), active count ch,ic_{h,i}, mean attention, peak attention, and a binary flag for whether the token is active in the last bin.

Without this experiment, it is impossible to tell whether dh,id_{h,i} provides any information beyond simpler features such as "recently active," "repeatedly active," or "high peak attention." Downstream benchmark scores cannot substitute for this validation, because they are jointly affected by protection, the value cue, pooling, and the budget. Figure 1 is only a hypothetical toy example, not empirical evidence.

W2. Coupling between the protection set and the budget: 5% is not really 5%

The paper explicitly states that "Protected positions count toward BB." With S=4S=4 and R=64R=64, the budget that actually participates in scoring is only

B−(S+R)=B−68.B-(S+R)=B-68.

When N<68/ρN<68/\rho, the protection set alone exhausts the budget: at ρ=0.05\rho=0.05, N<1,360N<1{,}360, and at ρ=0.10\rho=0.10, N<680N<680. LongBench contains short tasks of roughly 1–2K tokens (e.g., code completion); at 5% retention, BB may be only 50–100, and when B<68B<68 no old token is scored at all, so the method degenerates to "sink + recent." Even at N≈1,200N\approx1{,}200, 10% retention gives B≈120B\approx120, leaving only about 52 positions for scoring after subtracting 68; at 5% the number is negative. The short-task results at 5% therefore cannot serve as evidence that the scorer is effective.

In addition, the temporal score applies only to the first N−RN-R positions; positions N−64N-64 through N−17N-17, although part of the old prefix, are covered by the protected score. This further narrows the scorer's effective range, and the paper does not mention it.

The protection window is not matched to the baselines. Appendix A.2 says SnapKV uses a 32-query observation window but does not state how many recent tokens it retains; Lifetime explicitly protects 64 recent tokens. If SnapKV retains these 32 observation tokens in its standard implementation, then on RULER-256 Lifetime devotes

68/256=26.6%68/256=26.6\%

of its budget to fixed protection, whereas SnapKV devotes about 32/256=12.5%32/256=12.5\%. This is one of the central fairness issues in the experiments.

W3. Ablation coverage and Experimental fairness

Ablation coverage is insufficient. Table 6 covers only Qwen3-4B, LongBench, and 5% retention; it includes neither RULER-256 nor Llama nor 10%. The paper's headline result is on RULER-256, yet component contributions are measured only in a completely different setting. The authors should add:

  • w/o Periodic, w/o Rising, w/o Value, w/o Smoothing, and Short Recent on RULER-256;
  • at least one ablation on Llama;
  • at least one ablation at 10%.

The ablations are not clearly explained. The asymmetry between "w/o Periodic" and "w/o Rising" (7.85 vs. 1.76) suggests that the usage ratio of the two branches may be highly skewed; the paper reports neither branch usage nor the branch composition of the retained tokens. The β sweep is "derived from auxiliary runs rather than a direct fixed-m rerun," so it is not a controlled experiment; sensitivity to θ and ε is entirely missing; and the meaning of "same per-token permutation" in Shuffled-Time is ambiguous (do all tokens share one permutation, or does each token get its own?). As written, these ablations are not reproducible.

Tuning on the test set. W,T,θ,m,β,k,S,RW,T,\theta,m,\beta,k,S,R were selected on LongBench Qwen3-4B at 5%/10% using Figures 3–4 and Table 6, and the same benchmark is then used to report the main results. The paper's statement of "one fixed configuration ... without any per-benchmark or per-model retuning" is true in the sense that no further tuning is done per setting, but the configuration itself was selected on the test benchmark, with no held-out dev set. This matters given that some margins are only 0.07–0.09. I recommend selecting the configuration on a disjoint dev set (e.g., a subset of tasks or a different seed) and reporting on the remaining tasks, or at least stating which data were used for configuration selection and whether they overlap with the main results.

W4. the delta relative to SnapKV / ReST-KV / Expected Attention

Each individual component—observation window, peak/mean attention, local pooling, sink+recent protection, and value norm—is already known. The new component is the periodic/rising proxy. The problems are:

  • The paper does not compare against simpler temporal features such as "last-active recency," "active count," or "whether the token is active in the last bin"; "w/o Periodic" in Table 6 also conflates recurrence with phase. It is therefore impossible to tell whether phase/slope extrapolation is necessary at all. If a recurrence flag explains most of the gain, the paper's novelty is substantially lower than claimed.
  • The value-norm cue overlaps with Expected Attention: Appendix A.2 explicitly says that Expected Attention uses "covariance and value-norm terms." The paper presents the value cue as one of its contributions without acknowledging this overlap and without ablating against Expected Attention's value term. This cue is not new.
  • ReST-KV's "spatial-temporal smoothing" is very close to the paper's temporal trajectory; the related work mentions it in a single sentence and does not explain the essential difference between "smoothing" and "phase/slope extrapolation." The related work also discusses no prior work that uses attention slope/trend within an observation window, so the reader cannot assess the delta.
  • The Belady analogy is rhetorical: Belady's MIN requires a future access sequence, whereas the paper uses the order of tokens within the query, not future accesses. "Reuse-distance prediction for hardware caches" is also defined over memory-access sequences, not query tokens. Without the predictive-validity experiment in W1, the analogy does not hold.
  • The method has eight hyperparameters (W,T,θ,m,β,k,S,RW,T,\theta,m,\beta,k,S,R) plus ε\varepsilon, and its score is a hand-tuned linear combination. "Training-free" is not the same as "parameter-free," and the configuration was selected on the test benchmark. A more accurate characterization is "an effective engineering heuristic" rather than "a principled next-use estimator."
Questions for authors
  1. Predictive validity. Please record the actual attention during decoding on a set of prompts, compute the correlation/AUC between dh,id_{h,i} and "the number of future steps until the token is first attended above a threshold," and compare against last-active recency, active count, mean attention, and peak attention. Does dd really provide information beyond these simple features? This is the key to the paper's central claim.
  2. Branch usage. Please report the proportion of tokens assigned to the periodic and rising branches, the proportion of retained tokens, and the proportion with ch,i≥2c_{h,i}\ge2. Please add four controls—"c≥2c\ge2 and d=0d=0," "d=T−1−ℓd=T-1-\ell," "d=0d=0 globally," and "d=Td=T globally"—to separate the contributions of recurrence and phase/slope.
  3. The p=1p=1 case in Equation (3). When p=1p=1, d=0d=0 for any δ\delta; is this intentional? Please report the proportion of tokens with p=1p=1 and its effect on top-BB selection. If it is not intentional, please fix the formula (e.g., penalize missed periods) and rerun.
  4. Budget and protection set. Please report NN, BB, and B−(S+R)B-(S+R) for each task; when B<S+RB<S+R, which positions does tie-breaking actually keep? Please add three baselines—"SnapKV + 64 recent + 4 sinks," "sink+recent only," and "random selection"—and sweep R∈{16,32,64,128}R\in\{16,32,64,128\} on RULER-256.
  5. Baseline reproduction. Please reproduce the published numbers of Expected Attention and CapKV using the official code/settings.
5 comments
AuthorsOpenReview ↗

Response to Reviewer qJAk (1/4)

Thank you for the detailed review and for specifying the analyses that would test each claim. We answer each point below and repeat the key numbers from the General Response where they apply.

W1 / Q1. Does dd predict decoding attention beyond simple features?

You are right that prompt-tail positions are not decoding steps, and that downstream scores cannot replace a direct test, since protection, the value cue, pooling, and the budget also affect them. We ran the test you proposed (setup in G3). During Full-KV decoding, a token counts as accessed when it enters the top 5% of old-prefix attention in a layer–KV-head pair, and the label is whether the number of steps until its first access is at most 8. Table R4 compares 1/(1+d)1/(1+d) with the five baselines you listed:

  • As a single feature, 1/(1+d)1/(1+d) (AUROC 0.709 / 0.703) is above recency (0.687 / 0.681) and the last-bin flag (0.675 / 0.670), on par with active count (0.701 / 0.704), and below mean and peak rank (0.725–0.741).
  • Added to a predictor that already uses all five baselines and the value-norm rank, it raises AUROC from 0.755 to 0.766 (Qwen) and from 0.761 to 0.774 (Llama).

So dd alone does not predict better than static importance, and Lifetime does not rank by it alone: dd discounts the peak rank, which is then blended with the mean rank. The test shows that the spacing and slope features behind dd carry information about near-term decoding attention beyond recency, count, last-bin activity, mean, and peak.

On the eviction side, the matched controls change only dd and hold protection, window, pooling, value cue, and budget fixed (Tables R1–R2 and R13 below). Lifetime exceeds d=0d=0 by +1.49 / +1.25 on RULER-256 (VT +9.20 / +6.20), and every simple temporal alternative by at least +0.59 on average (VT +4.30).

The revision presents dd as an order-aware proxy for near-term relevance rather than a count of decoding steps, keeps Belady's MIN as motivation only, adds this test to the paper, and labels Figure 1 as a schematic.

W2 / Q4. Protection, budget accounting, and matched baselines

Scored budget and range. You are right on both points. Protected positions come out of BB, so Bscored=max⁡(0,B−68)B_{\text{scored}}=\max(0, B-68). The 64 recent positions also cover the last 48 old-prefix positions, so only the N−68N-68 unprotected positions compete for the scored slots.

Table R9. Scored budget under relative retention.

Prompt length NScored candidates (N − 68)B / B_scored at 5%B / B_scored at 10%
1,02495651 / 0102 / 34
2,0481,980102 / 34204 / 136
4,0964,028204 / 136409 / 341
8,1928,124409 / 341819 / 751

The scorer is inactive (Bscored=0B_{\text{scored}}=0) for N<1,380N < 1{,}380 at 5% and N<690N < 690 at 10%; Lifetime then keeps only sinks and recent positions. These instances make up 3.8% / 3.5% (Qwen / Llama) of LongBench at 5% and 0.6% / 0.5% at 10%, and we agree they give no evidence for the scorer. On RULER, B=256B=256 for every instance, so Bscored=188B_{\text{scored}}=188 throughout; the RULER-256 comparisons below therefore test the scorer at the same nonzero scored budget on every instance. The revision reports NN, BB, and BscoredB_{\text{scored}} for every task and states the 48-position overlap in Section 3.4.

When the protected set does not fit. This occurs for N<1,360N < 1{,}360 (5%) or N<680N < 680 (10%). In the submitted runs, the backend's top-k tie handling chose among the equal protected scores, so the kept positions were implementation-defined. The revision fixes the rule: keep the 4 sinks first, then the most recent positions in reverse order until BB is reached. All matched controls use this rule.

Protection matched to SnapKV. In our KVPress implementation, SnapKV keeps its 32-query observation window and no sinks: 32 of 256 entries (12.5%), versus 68 (26.6%) for Lifetime. Table R10 adds the three baselines you requested; every row keeps exactly 256 entries per KV head.

AuthorsOpenReview ↗

Response to Reviewer qJAk (2/4)

Table R10. RULER-16K, 256 entries per KV head.

MethodQwen Avg.Qwen VTLlama Avg.Llama VT
SnapKV (original)47.6025.4066.0946.80
SnapKV + 4 sinks + 64 recent55.3043.0068.2057.40
Sinks + recent only10.125.0012.574.00
68 protected + random selection14.367.4016.926.80
Lifetime pipeline, mean-only (m = 1, β = 0)66.4873.6071.8372.40
Lifetime pipeline, d = 069.0788.0073.6282.40
Lifetime70.5697.2074.8788.60
  • Matched protection improves SnapKV by +7.70 / +2.11, so protection is part of the original gap. SnapKV with matched protection remains 15.26 / 6.67 below Lifetime.
  • Protection alone does not explain the results: filling the remaining slots with recent or random positions gives 10–17 points.
  • With protection and every other component identical, Lifetime exceeds mean-only aggregation by +4.08 / +3.04, of which the distance term accounts for +1.49 / +1.25 (VT +9.20 / +6.20).

The revision adds these rows to Section 4.3, attributes the RULER-256 margin over SnapKV to protection, the scoring pipeline, and the distance term separately, and lists every baseline's protected positions in Appendix A.2.

Recent-window size. Table R11 sweeps RR with the budget fixed at 256.

Table R11. Recent-window sweep on RULER-256 (4 sinks throughout).

Recent positions RProtected share of BScored slotsQwen Avg.Llama Avg.
167.8%23669.8874.25
3214.1%22070.2074.57
64 (default)26.6%18870.5674.87
12851.6%12469.8574.12

Averages vary by at most 0.71 (Qwen) and 0.75 (Llama) across this range. At R=32R=32, whose protected share is close to SnapKV's, Lifetime scores 70.20 / 74.57, versus 47.60 / 66.09 for SnapKV.

W3 / Q2. Ablations, branch usage, and configuration selection

Coverage. Table R12 extends the ablations to RULER-256 on both models and to LongBench at 10%, changing one component at a time.

Table R12. Component ablations.

VariantRULER-256 Qwen Avg. (VT)RULER-256 Llama Avg. (VT)LongBench Qwen 5%LongBench Qwen 10%
Lifetime70.56 (97.20)74.87 (88.60)45.1347.90
w/o Periodic65.76 (75.60)72.41 (74.00)37.2846.88
w/o Rising69.81 (94.40)74.31 (86.00)43.3747.57
w/o Value69.76 (94.80)74.15 (85.20)44.0247.61
w/o Smoothing69.31 (92.80)73.98 (83.80)43.0747.42
Short recent (R = 16)69.88 (94.80)74.25 (86.60)44.9147.74

Every variant is below Lifetime in every setting.

Branch usage and the Periodic/Rising asymmetry. On RULER-256 (Table R3), the periodic branch covers 8.4% / 7.9% of candidates but 55.2% / 52.6% of the retained scored entries, and d=0d=0 holds for 4.0% / 3.7% of candidates and 34.7% / 32.9% of retained entries. The w/o Periodic variant sends the periodic-branch tokens through the rising branch, which assigns d=Td=T to any token whose attention did not increase between the last two bins. Repeatedly active tokens then lose most of their peak weight. This is why removing the periodic branch costs far more than removing the rising branch, on RULER-256 (−4.80 vs. −0.75 on Qwen) as on LongBench (−7.85 vs. −1.76).

Separating recurrence from phase and slope. Table R13 changes only the distance dd in P/(1+d)P/(1+d); all other components are fixed.

Table R13. Controls that change only d (RULER-256).

Distance dQwen Avg.Qwen VTLlama Avg.Llama VT
Rising branch for all tokens (w/o Periodic)65.7675.6072.4174.00
Last-bin active flag68.3985.2073.3681.00
d = 0 for all tokens69.0788.0073.6282.40
Active count, d = T − c69.3389.0073.9083.20
Recency, d = δ69.7390.8074.2584.20
Phase if c ≥ 2, d = T otherwise (w/o Rising)69.8194.4074.3186.00
Phase if c ≥ 2, slope otherwise (Lifetime)70.5697.2074.8788.60
AuthorsOpenReview ↗

Response to Reviewer qJAk (3/4)

  • Recurrence alone explains little: scoring tokens by how often they were active (d=T−cd=T-c) adds +0.26 / +0.28 over d=0d=0 (VT +1.00 / +0.80).
  • Each branch alone falls short. Phase alone is on par with recency on average (69.81 vs. 69.73; 74.31 vs. 74.25) and higher on VT (+3.60 / +1.80). Slope alone falls below d=0d=0.
  • Together, the two branches exceed every simple alternative, including the EMA score (69.89 / 74.28, Table R2), by at least +0.59 on average and +4.30 on VT.

The gain therefore requires both the phase estimate for recurring tokens and the slope estimate for the others.

Sensitivity to θ, ε, and β.

Table R14. Single-parameter sweeps (Qwen3-4B, RULER-256 average; other parameters fixed).

ParameterValuesScore
θ0.90 / 0.95 / 0.9770.11 / 70.56 / 70.18
ε10⁻⁷ / 10⁻⁶ / 10⁻⁵70.50 / 70.56 / 70.22

Every setting stays above the strongest temporal alternative (EMA, 69.89). The default ϵ=10−6\epsilon=10^{-6} was missing from Section 3.4 and is now listed. For β, the controlled endpoints are in Table R12: β = 0 at fixed m = 0.35 (w/o Value) loses 1.11 / 0.29 on LongBench Qwen 5% / 10% and 0.80 / 0.72 on RULER-256. The revision replaces Figure 3(b) with a direct sweep at fixed mm and makes every sensitivity plot a single-parameter sweep.

Shuffled-Time. Within each layer–KV-head pair, all tokens share one permutation of the 8 bins, applied to both the raw and the rank-normalized trajectories. Each bin moves intact, so within-bin ranks and all order-invariant statistics are preserved and only the temporal order changes. Results average two permutation seeds.

Configuration selection. The configuration was selected on Qwen3-4B LongBench, so the Qwen3-4B LongBench rows of Table 1 overlap with the selection data. No Llama-3.1-8B result and no RULER result was used for selection. Given this overlap, the revision describes the LongBench results as parity with the strongest baselines and rests the evidence for temporal scoring on the held-out settings, with all other components matched:

  • Llama-3.1-8B LongBench: the distance term adds +0.28 (10%) and +0.34 (5%) over d=0d=0.
  • RULER-256 on both models: +1.49 / +1.25 over d=0d=0, and VT +9.20 / +6.20.

The revision states the selection data and rewrites the sentence "without any per-benchmark or per-model retuning" accordingly.

W4. Delta relative to SnapKV, ReST-KV, and Expected Attention

Simpler temporal features. Tables R1–R2 and R13 compare Lifetime with recency, active count, the last-bin flag, and EMA in an otherwise identical pipeline, and separate recurrence from phase and slope. Lifetime is above all of them on LongBench and RULER-256, by at least +0.59 on the RULER-256 average and +4.30 on VT, and recurrence alone accounts for little of the gain.

Value cue. You are right that Expected Attention uses value norms, and CriticalKV also accounts for value states. The revision credits this prior work and removes the value cue from our contributions. Our estimates of the temporal term hold the value cue fixed (row (b) in Tables R1–R2), so they do not depend on it.

ReST-KV's temporal smoothing. ReST-KV smooths scores with an exponential moving average (coefficient 0.3 in our runs). This weights later evidence more but returns one scalar per token, so tokens with different trajectories can receive the same score. Lifetime instead reads two properties of the trajectory, the spacing of repeated activations and the slope at the boundary, and uses them to discount the peak rank. The EMA control isolates this difference: with ReST-KV's coefficient and every other component identical, Lifetime is higher by 0.15–0.22 on LongBench and 0.67 / 0.59 on RULER-256 (VT +5.00 / +4.30). The revised related work explains this distinction and compares Lifetime with other aggregations of observation-window attention (mean, EMA smoothing, and worst-case aggregation).

Belady's MIN and reuse distance. We agree that Belady's MIN requires a future access sequence and that reuse-distance prediction is defined over memory accesses, not query tokens. The revision keeps both as motivation only, removes the sentence placing Lifetime "closer to reuse-distance prediction," and supports the proxy with the direct test in W1 / Q1.

Hyperparameters and characterization. We agree that training-free is not parameter-free and that the score is a hand-set linear combination. The revision describes Lifetime as a training-free heuristic with one fixed configuration and lists all nine hyperparameters, including ε. Sensitivity is reported for m and β (Figure 3 and the direct β sweep), W and T (Table R6), θ and ε (Table R14), and R (Table R11). k = 5 matches SnapKV's pooling kernel, and S = 4 matches the four sinks used by StreamingLLM and CapKV.

Reviewer_SHAyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies one-shot KV-cache eviction after prefill. The authors argue that averaging prompt-tail attention, as SnapKV does, hides whether a token's attention is fading or rising. Lifetime splits the last 16 queries into 8 bins, estimates from this trajectory a distance d to each token's next important use, and scores tokens by mean-attention rank plus peak rank. It reports the best compressed-method average on LongBench at 10% and 5% retention on two models, and on RULER-16K at 256 entries, with under 1% extra prefill time over SnapKV.

Critical strengths
  • The problem is clear, and the method is easy to apply.
Critical weaknesses
  • It is unclear whether next-use prediction explains the gains. Two controls are missing: m = 1, β = 0 and d = 0 for all tokens, with everything else unchanged. The four temporal ablations score at or below SnapKV, but do not establish an advantage over these simpler controls. Eq. 3 also gives d = 0 whenever p = 1, regardless of how long the token has been inactive, so even an early burst of attention can imply imminent reuse.
  • The LongBench gains over the next-best method are small (0.07 to 0.54 points). The same stated configuration differs by 0.97 points at 10% retention between §4.5 and Table 1. Both 5% leads also depend on TREC. These results need an explanation of the discrepancy and evidence that the gains hold across sampling seeds.
Questions for authors
  • With all other settings unchanged, how do two separate variants perform on LongBench and RULER, especially VT: (a) m = 1, β = 0; and (b) d = 0 for all tokens?
  • What explains the §4.5/Table 1 discrepancy? Do the gains persist across sampling seeds?
  • Which modulo convention does Eq. 3 use, and what fraction of tokens has p = 1 and therefore receives d = 0? I would consider raising my score if Lifetime clearly outperforms both controls and the behavior of Eq. 3 is satisfactorily explained or corrected.
2 comments
AuthorsOpenReview ↗

Response to Reviewer SHAy (1/2)

Thank you for proposing two precise controls and for identifying both the Eq. (3) boundary case and the §4.5 inconsistency. We address each point below.

W1 / Q1. Do the gains hold against (a) m=1,β=0m=1, \beta=0 and (b) d=0d=0, with everything else unchanged?

Yes. Your controls are rows (a) and (b) of Tables R1–R2 (General Response, G1). Both share Lifetime's window, binning, protection, pooling, and budget.

Table R5. Margin of Lifetime over your two controls.

LongBench Qwen 10% / 5%LongBench Llama 10% / 5%RULER-256 Qwen Avg. (VT)RULER-256 Llama Avg. (VT)
Lifetime − (a)+0.77 / +0.75+0.55 / +0.64+4.08 (+23.60)+3.04 (+16.20)
Lifetime − (b)+0.38 / +0.29+0.28 / +0.34+1.49 (+9.20)+1.25 (+6.20)

Control (b) is the most direct test: it keeps mean rank, peak rank, value rank, and pooling, and removes only the distance discount. Lifetime is above it in every comparison. The margin is small on LongBench and largest on VT, the task that requires following assignments revisited at irregular intervals.

With the corrected Eq. (3) (W1 / Q3 below), the margins over (b) are:

  • +0.25 to +0.32 on LongBench;
  • +1.37 / +1.16 on RULER-256;
  • +8.40 / +5.60 on VT.

This also explains why the four temporal ablations in Table 6 fall below SnapKV. They corrupt the temporal term rather than remove it:

  • Shuffled and Reversed feed it a wrong order.
  • w/o Periodic and w/o Rising give one branch wrong distances.

In both cases, the corrupted term discounts the peaks of relevant tokens. Your controls remove the term instead. On LongBench Qwen 5%, the resulting order is w/o Periodic 37.28 < Reversed 39.27 < Shuffled 42.84 < w/o Rising 43.37 < (a) 44.38 < (b) 44.84 < Lifetime 45.13.

W1 / Q3. Modulo convention and the p=1p=1 case of Eq. (3)

We use the nonnegative remainder x mod p=x−p⌊x/p⌋x \bmod p = x - p\lfloor x/p\rfloor. You are right that this gives d=0d=0 for p=1p=1 regardless of how long the token has been inactive. More generally, the phase term never penalizes missed cycles. This was unintended.

The fractions you asked for are in Table R3 (G2):

  • p=1p=1 covers 2.6% (Qwen) and 2.3% (Llama) of candidates.
  • The affected case (p=1p=1, δ>0\delta>0) covers 0.7% of candidates but 6.1% / 5.5% of retained scored entries.

We now add one period per missed cycle, which reduces to d=min⁡(T,δ)d=\min(T,\delta) for p=1p=1 (G2). This has three effects:

  • the affected share of retained entries drops to 1.9% / 1.8%;
  • averages change by at most 0.12 points, and VT by at most 0.80;
  • Lifetime stays above both of your controls everywhere.

The revision adopts the corrected Eq. (3).

AuthorsOpenReview ↗

Response to Reviewer SHAy (2/2)

W2 / Q2. The §4.5 vs. Table 1 discrepancy

Thank you for catching this. The §4.5 scan was run on an earlier cached instance manifest, whereas Table 1 uses the final seed-42 manifest. We re-ran every structural configuration on the Table 1 manifest:

Table R6. Structural scan on the Table 1 manifest (Qwen3-4B, LongBench).

Configuration10%5%
W = 16, T = 446.8943.51
W = 16, T = 8 (submitted)47.9045.13
W = 16, T = 1647.1143.96
W = 32, T = 1647.7244.56

The submitted configuration now matches Table 1 exactly. It remains the best structural setting at both budgets, by 0.18 to 1.62 points. §4.5 will use these numbers, and the manifest identifiers will be released with the code.

W2 / Q2. Small LongBench margins, the TREC dependence, and sampling seeds

We agree that these margins are too small to establish a LongBench advantage.

  • Instance sensitivity. The same configuration scores 0.23 (5%) and 0.97 (10%) points apart on the two manifests above. This is comparable to our margins over the strongest baseline (0.07–0.54). We have not re-sampled LongBench with additional seeds.
  • TREC dependence. We confirm it: without TREC, Lifetime is 0.13 (Qwen) and 0.28 (Llama) points below ReST-KV at 5%.

The revision will therefore describe the LongBench results as on par with the strongest baselines. The claim about the temporal term will rest on two other sources:

  • the matched controls, where all variants run on identical instances (Tables R1–R2);
  • RULER-256, where the proximity term adds +1.49 / +1.25 on average and +9.20 / +6.20 on VT.

In summary, Lifetime is above both of your controls in every LongBench and RULER-256 comparison, with the largest margins on VT. Eq. (3) is corrected, with essentially unchanged results.


Reviewer_2x7LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Lifetime, a training-free, query-aware KV cache eviction method applied at the prefill stage. The authors argue that SnapKV-style methods average attention over the observation window and thereby discard temporal information. Lifetime instead keeps a binned attention trajectory for each KV head, estimates each token's "next-use distance" from it, and combines this estimate with peak rank, mean rank, and value norm to select the top-B entries. On Qwen3-4B and Llama-3.1-8B, it achieves the best LongBench average and leads RULER-16K under low budgets, with less than 1% additional prefill overhead.

Critical strengths

Strengths  The motivation is intuitive and the problem is clearly defined.  The method is well designed and lightweight, which makes it easy to deploy.

Critical weaknesses

Weaknesses  The improvements are marginal. On some benchmarks and subtasks, the gains fall within the noise range.  The estimator relies on sparse statistics, and the heuristic rules are complex and tightly coupled.  The evaluation scope is limited. The largest model has 8B parameters and the longest context is about 32K tokens, with no results at 64K–128K. Query-agnostic, multi-turn, long-generation, or reasoning settings are not covered, and the efficiency evaluation is also limited.  The writing needs further polishing.

Questions for authors

Questions  Can the authors directly validate the relationship between the proxy d and the actual attention observed during decoding?  Why can the positional order of the prompt-tail queries be treated as "time" and used to predict the next use of tokens during generation?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2x7L (1/2)

Thank you for your review and for the two direct questions. The first one led to the decoding-attention diagnostic below.

W1. Marginal improvements, some within noise

We agree for LongBench. The gains are concentrated in one regime:

Table R7. Margin of Lifetime over the best baseline in each row (paper Tables 1, 3, 4).

Qwen3-4BLlama-3.1-8B
LongBench avg., 10%+0.54+0.09
LongBench avg., 5%+0.07+0.36
RULER-256 avg.+9.97+5.16
RULER-256 VT+54.00+26.20
RULER-256 avg. without VT (vs. ReST-KV)+0.46−0.11

The LongBench margins are comparable to the 0.23–0.97-point shift we observed when the same configuration was run on a different instance manifest. Outside VT, the RULER-256 results are on par with ReST-KV.

The revision will state this directly: Lifetime matches the strongest baselines on general long-context tasks and is stronger on multi-step tracking under tight budgets. Within Lifetime's pipeline, the proximity term adds +9.20 / +6.20 on VT over an otherwise identical scorer with d=0d=0 (Table R2).

W2. Sparse statistics and complex, coupled heuristics

Each rule now has a measured contribution, and the rules interact less than our description suggested.

  • One estimate per token. The periodic rule applies when a token is active in at least two bins, and the rising rule applies otherwise. Each token therefore receives exactly one estimate. It enters the score only through P/(1+d)P/(1+d), which is blended linearly with the mean rank (weight mm) and the value rank (weight β\beta).
  • Sparse events select a small, important set. The periodic branch fires on 8.4% / 7.9% of candidates. However, these tokens form 55.2% / 52.6% of the retained scored entries (Table R3). The mean-rank term keeps the score informative for all other tokens.
  • Each rule contributes. On RULER-256, the average drops when each component is removed (Table R12):
Removed componentQwenLlama
Periodic branch−4.80−2.46
Rising branch−0.75−0.56
Value cue−0.80−0.72
Smoothing−1.25−0.89
  • Settings are stable. The threshold θ\theta controls how sparse the active events are. On Qwen RULER-256:

    • θ∈{0.90,0.95,0.97}\theta \in \{0.90, 0.95, 0.97\} gives 70.11 / 70.56 / 70.18;
    • ϵ∈{10−7,10−6,10−5}\epsilon \in \{10^{-7}, 10^{-6}, 10^{-5}\} gives 70.50 / 70.56 / 70.22.

    Every setting stays above the strongest temporal alternative (EMA, 69.89).

AuthorsOpenReview ↗

Response to Reviewer 2x7L (2/2)

W3. Evaluation scope

Longer contexts. We evaluated RULER at 64K and 128K on Llama-3.1-8B with 1,024 entries per head (64× and 128× compression), using the unchanged configuration:

Table R8. RULER at 64K and 128K, Llama-3.1-8B, budget 1,024.

Method64K128K
Full KV85.1080.30
SnapKV60.4047.70
ReST-KV70.2057.60
Lifetime70.7057.80

Lifetime stays on par with ReST-KV (+0.50, +0.20) and well above SnapKV (+10.30, +10.10).

Efficiency. The <1% prefill overhead over SnapKV is +47.3 ms at batch size 1 and +140.1 ms at batch size 4. It is measured on Qwen3-4B with ~30K-token prompts and the SDPA backend (Table 5). Decode throughput and peak memory match SnapKV's, because both produce a physically compacted cache of the same shape. The revision states these settings next to the claim.

Other settings. Several settings lie outside what a query-aware, one-shot prefill method addresses:

  • Query-agnostic and multi-turn use require scores that do not depend on the current query, or that are revised at each turn.
  • Long generations can drift away from what the prompt tail emphasized.
  • We have not evaluated models beyond 8B.

The revision will state this scope in the introduction and the Limitations.

W4. Writing

We will revise for clarity, including these specific changes:

  • dd is introduced as a relative relevance proxy, with Belady's MIN as motivation only.
  • Eq. (3) states its modulo convention and uses the corrected form.
  • Section 3.4 gives the scored budget Bscored=max⁡(0,B−68)B_{\text{scored}}=\max(0, B-68), the overlap between the recent region and the old prefix, and the tie rule.
  • §4.5 uses the re-run scan.
  • All ablation and sweep definitions are made explicit.

Q1. Can dd be validated against the attention observed during decoding?

Yes; we ran this test (G3, Table R4). We recorded attention during Full-KV decoding and measured how well each prefill feature predicts whether a token enters the top 5% of attention within the first 8 decoding steps.

As a single feature, 1/(1+d)1/(1+d) reaches AUROC 0.709 (Qwen) and 0.703 (Llama). This is:

  • above recency (0.687 / 0.681) and the last-bin flag (0.675 / 0.670);
  • on par with active count (0.701 / 0.704);
  • below mean and peak rank (0.725–0.741).

Added to a predictor that already uses all of these features, it raises AUROC from 0.755 to 0.766 (Qwen) and from 0.761 to 0.774 (Llama). This matches the role dd plays in Lifetime: it refines static importance rather than replacing it.

Q2. Why can the order of prompt-tail queries be treated as time?

You are right that the prompt-tail positions are not decoding steps, and the revision will say so. Our use of their order rests on one observation, which we test rather than assume.

The window holds the last 16 positions before generation, and decoding continues the same causal sequence. The binned trajectory therefore shows how attention to each token evolves as the sequence approaches the generation boundary, for example while the end of the user query is processed. A token whose attention recurs or rises toward the boundary is more likely to be needed in the first decoding steps than one whose attention has faded.

Two results support this:

  • 1/(1+d)1/(1+d) adds predictive information about decoding attention beyond the simple features (Q1).
  • On LongBench Qwen 5%, Lifetime exceeds its shuffled-order version by 2.29 points. Meanwhile, the shuffled and reversed versions fall below the d=0d=0 control (42.84 and 39.27 vs. 44.84). The same statistics in the wrong order mislead the score.

We therefore present dd as a relative proxy for near-term relevance, not as a count of decoding steps.


Comments on the paper

1 comment

Dear Reviewers,

The authors have posted responses. As the discussion period is ending soon, could you please take a look and reply shortly, or at least acknowledge that you have read it?

Thank you!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.