- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TokenButler, a learned, query-aware token selection framework for accelerating long-context LLM decoding. Unlike KV-cache eviction methods that permanently discard tokens or page-based methods that retrieve coarse groups of tokens, TokenButler preserves the full KV cache and dynamically selects individual tokens to participate in attention computation.
The key component is a lightweight importance predictor trained by distilling the attention distributions of a frozen backbone LLM. Specifically, the method predicts low-dimensional importance queries from intermediate hidden states and combines them with learned projections of cached keys to estimate token importance in subsequent transformer layers. The framework further incorporates sink tokens, local attention windows, deferred key projection, prediction intervals, and neighbor fetching to reduce inference overhead.
The paper evaluates TokenButler on a synthetic co-reference retrieval benchmark, RULER, LongBench, LongMemEval, and AIME24. It reports competitive accuracy under constrained KV-access budgets, up to 1.6× on-GPU decoding speedup, and up to 7.6× latency reduction under CPU-offloaded KV-cache settings. The authors also provide a preliminary integration with SGLang.
Overall, this work addresses an important practical problem in efficient LLM inference and presents a reasonably comprehensive system. However, the incremental nature of several components and insufficiently isolated experimental comparisons make it difficult to determine how much of the observed improvement is attributable to the proposed learned importance prediction mechanism itself.
Critical strengths
S1. A practically important problem with a well-motivated design.
The paper addresses the memory-bandwidth bottleneck of long-context decoding, where accessing the entire KV cache becomes increasingly expensive. Preserving the full cache while dynamically selecting a small subset of tokens is a reasonable design choice, particularly for tasks where previously unimportant information may become relevant later.
S2. A lightweight and potentially reusable importance prediction mechanism.
TokenButler combines hidden-state-based query prediction with low-dimensional projections of the actual cached keys. This design allows it to estimate token importance without computing full-dimensional query-key scores. The predictor is trained while keeping the backbone frozen and introduces only approximately 0.3–0.9% additional parameters. Training on 1K-token contexts while evaluating up to 64K is an interesting empirical result.
S3. Encouraging results across several evaluation settings.
The evaluation covers multiple backbone models and benchmarks, including RULER, LongBench, LongMemEval, and AIME24. At an 8× KV-access reduction on LongBench, TokenButler obtains average scores of 44.43 and 45.00 on Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct-1M, respectively, compared with dense baselines of 43.27 and 44.79. These results indicate that substantial reductions in KV access can be achieved without large quality degradation in the evaluated settings.
S4. Attention to system-level efficiency.
The paper goes beyond an isolated importance predictor and considers implementation issues such as deferred key projection, contiguous KV buffers, batched token selection, and prediction intervals. The reported 1.6× on-GPU speedup at 128K and preliminary SGLang integration provide useful evidence of practical applicability.
S5. A targeted evaluation of long-range token reactivation.
The synthetic co-reference benchmark illustrates a meaningful limitation of permanent eviction and coarse page-based retrieval. Although the benchmark has limitations discussed below, it provides a clear diagnostic setting for studying tokens whose relevance changes across decoding steps.
Critical weaknesses
W1. Limited conceptual novelty relative to existing token-selection and learned sparse-attention approaches.
The main components of TokenButler are closely related to existing approaches: query-aware token-level KV selection has been studied in TokenSelect; lightweight learned indexers have been explored in DeepSeek Sparse Attention and IndexMem; and reusing token selections across decoding steps is closely related to the Selection Cache mechanism in TokenSelect.
While TokenButler combines these ideas in a useful way, it remains unclear which component represents the central algorithmic novelty beyond replacing full-dimensional importance scoring with a learned low-dimensional approximation. In particular, the paper would benefit from a more direct distinction between its contribution and the selection reuse mechanism in TokenSelect.
W2. Missing key ablations to isolate the benefit of learned importance prediction.
The paper does not sufficiently establish whether learning the importance predictor is necessary for the observed improvements. A particularly important missing baseline is a non-learned low-dimensional token selector using random projections, PCA-based projections, or other lightweight approximations of query-key similarity under the same dimensionality, token budget, and kernel implementation.
Such comparisons would distinguish the benefit of learned attention distillation from the computational advantages of low-dimensional scoring alone. Additional ablations separating query prediction, learned key projection, and their interaction would make the central contribution more convincing.
W3. Insufficiently controlled end-to-end efficiency comparisons.
The paper reports promising decoding speedups, but the evidence is distributed across different models, hardware configurations, token budgets, and serving implementations. In particular, the comparison against TokenSelect in Appendix J uses a modified TokenButler variant that removes head-specific predictions to fit the compared retrieval implementation.
Furthermore, the 7.6× result is obtained in a CPU-offloading regime where reducing host-to-device KV transfers can naturally offer large benefits. This is a useful system result, but it should not be interpreted as an intrinsic 7.6× advantage of the learned importance predictor over other sparse selection methods.
A matched end-to-end comparison against TokenSelect and Quest using the same backbone, hardware, KV residency, selection budget, and optimized sparse-attention backend would considerably strengthen the efficiency claims.
W4. The proposed prediction interval mechanism is not sufficiently isolated from neighbor fetching.
Prediction intervals and neighbor fetching are introduced together to amortize selection overhead and maintain accuracy. However, neighbor fetching expands each selected token with an additional position, potentially improving coverage simply by accessing more tokens.
The paper should provide a controlled factorial ablation comparing per-step prediction, periodic prediction, neighbor fetching, and their combination under exactly matched total KV-access budgets.
Additionally, the observation that periodically refreshing token selections yields small accuracy losses is interesting, but the paper does not sufficiently characterize when temporal reuse breaks down, especially under abrupt shifts in token importance.
W5. Some evaluation settings and generalization claims remain insufficiently validated.
The synthetic co-reference benchmark is explicitly designed to expose weaknesses of eviction-based and page-based methods. Its extremely large performance gaps are therefore less informative about average performance on realistic workloads. The main text should avoid treating these gaps as evidence of general superiority.
Moreover, the reported superiority on standard benchmarks is not uniform. On Llama-3.1-8B-Instruct, TokenButler achieves 89.93 on RULER, only slightly above SnapKV at 89.68 and below dense attention at 91.89. On Qwen2.5-7B-Instruct-1M, its LongBench average of 45.00 is only 0.21 points above dense attention.
The paper would also benefit from evaluations across more aggressive sparsity ratios, multiple training seeds, longer decode trajectories, and additional reasoning workloads. These are particularly relevant because the authors acknowledge a substantial decline in importance recall at very small token budgets.
Questions for authors
Q1. How much of the improvement comes from learning rather than dimensionality reduction?
Could the authors compare TokenButler with random projection, PCA-based projection, and other training-free low-dimensional query-key approximations, using the same scoring dimension (d'=16), sparse-attention implementation, and KV-access budget? This would help isolate the contribution of attention distillation.
Q2. How does TokenButler compare with a carefully optimized TokenSelect implementation?
Could the authors provide end-to-end latency and throughput comparisons under identical hardware, model, context length, token budget, and KV residency conditions? It would be particularly helpful to distinguish selection overhead from sparse-attention execution and KV-gather costs.
Q3. What is the independent contribution of prediction intervals and neighbor fetching?
Could the authors provide a 2×2 ablation with and without prediction intervals and neighbor fetching, while controlling for the total number of retrieved tokens? How does accuracy change when attention shifts abruptly to distant, previously unselected tokens?
Q4. What makes token importance predictable across layers and context lengths?
The predictor is trained on sequences of length 1K but evaluated at 64K. Could the authors provide attention top-k recall measurements, rank correlations, or other diagnostics across layer distances, attention heads, context lengths, and decoding positions? In particular, how does prediction quality change under distribution shifts or on contexts substantially longer than those seen during training?
Q5. How robust are the results under more aggressive sparsity and longer generation?
Could the authors report accuracy-efficiency Pareto curves at multiple KV-access budgets, particularly below 10%, and provide additional evaluations on longer reasoning trajectories? Reporting variance across training and evaluation seeds would also help establish whether the relatively small gains over strong baselines are statistically reliable.
Thank you for the thorough review. Full results are in App. J–N of the revision.
W1 (novelty; Selection Cache). (1) Importance is predictable ahead of the layers that use it: a module with <1% of the parameters, trained in <10 GPU-hours for the 8B model with the LLM frozen, predicts which tokens the next four layers need; DSA retrains the model, and LookaheadKV and IndexMem evict permanently. (2) Cross-layer look-ahead: TokenSelect and the Lightning Indexer score the cache with each layer's real query at every layer and step. TokenButler scores a layer group at once against 16-d key sketches (20.1 vs 35.3 ms/token of selection and gather at 128K, Table 14). (3) Scheduled reuse: the Selection Cache (now in Sec. 2 as the closest prior mechanism) still computes and compares the query at every step and layer; our interval does no work on non-refresh steps (1.33× / 1.69× dense speed at 64K / 128K vs 0.79× / 1.02×, Table 14). The two are complementary.
W2 / Q1 (is learning necessary?).
Key sketches at the same d'=16, budget and kernels (RULER 64K, per-step, 8,832 positions per layer).
A random projection with the real query is not enough (86.64). The learned key sketch is at dense level with the real query and 3.3 points above the random projection with a predicted one. A PCA sketch (Loki-style) also needs each layer's real query, so it scores inside every layer: 0.77× dense speed at this budget, and 0.90× at the budget where it reaches TokenButler's accuracy (4K: 89.97), against 1.25× for TokenButler with an interval of 16 (90.01; Table 7; all rules per task in Table 13). Predicting the query up to four layers ahead is what the learned predictor adds: one scoring pass serves a group of layers before they run, and with an interval of 16 decoding reaches 1.33× at this budget.
W3 / Q2 (controlled end-to-end efficiency).
Decode latency in ms/token (speed vs dense), K/V on GPU, 8,832 positions per layer.
TokenSelect is our port, optimized so as not to penalize it. With an interval TokenButler is the fastest method at both lengths, within 4–9% of a zero-selection-cost lower bound. Fig. 7 used TokenSelect's codebase with a per-layer variant of TokenButler; Table 14 is the converse, with the full per-head method. We agree that the CPU-offloading speedup is a system result against dense; Sec. 4.3 now says so.
W4 / Q3 (interval vs neighbor fetching; abrupt shifts).
Interval × neighbor fetching at the same 8,832 positions per layer (RULER 64K; B is halved with neighbors).
At the same positions per layer, i=16 decodes 1.43–1.45× faster than per-step prediction. Neighbor fetching recovers accuracy exactly when selections are stale (i=4: 88.03 → 88.44; i=16: 87.30 → 88.29, with N-MQ 82.6 → 90.9) and is not needed at i=1. Sec. 3.4 now presents it as a remedy for stale selections, and this factorial is now in the main text (Table 8). A fixed schedule can be stale after an abrupt change of focus; a forced refresh at known boundaries or a similarity trigger are compatible remedies.
W5 (benchmark framing). Sec. 4.1 now introduces the co-reference benchmark as a targeted diagnostic of token reactivation and states that RULER, LongBench and LongMemEval serve as the general-purpose evaluation. On realistic workloads the claim is dense-level quality at a fraction of the KV access: LongBench 44.43 vs 43.27 at 8× lower KV access, LongMemEval 26.6 vs 27.4. Lower budgets and seeds are in Q5, 128K in Table 7.
Q4 (what makes importance predictable). Much of the attention mass is structural: up to 64K, 81–87% lies on the sink and recent window. The heaviest remaining tokens are recoverable from an earlier hidden state: the predicted query finds 93% of the top-16 tokens at 16K and 53% at 64K. Predictability degrades with length (training used 1K-token sequences), now stated as a limitation (Sec. 5); diagnostics are in App. L.
Q5 (Pareto below 10% budget; seeds). RULER 64K, per-step prediction:
Three training seeds give 89.97 / 89.96 / 90.07 (App. N).