acceptodds
Under review as a conference paper at ICLR 2027

TokenButler: Token Importance is Predictable

Abstract

Large Language Models (LLMs) rely on the Key-Value (KV) Cache to store token history, enabling efficient decoding of tokens. As the KV-Cache grows, reading it at every decoding step becomes a major memory-bandwidth bottleneck. However, only a small subset of tokens contribute meaningfully to each decoding step, which is an opportunity to alleviate this bottleneck. A key challenge in finding these *critical tokens* is that they are dynamic, and heavily input query-dependent. Existing methods evict tokens permanently, risking quality; retrieve coarse pages, failing on dense, context-rich tasks; or score every token in the full head dimension, at prohibitive cost. Additionally, many existing KV-Cache sparsity methods rely on inaccurate proxies for token importance. To address these limitations, we introduce **TokenButler**, a high-granularity, query-aware predictor that learns to identify these critical tokens. TokenButler predicts low-dimensional *importance queries* at a fixed depth stride, and combines them with a learned projection of the *real KV-cache keys* to score tokens cheaply, enabling dynamic per-token selection under a fixed budget while preserving the full KV cache. We train TokenButler by distilling the model's masked causal attention distributions, optimizing a lightweight predictor with minimal parameter overhead. On a novel synthetic small-context co-referential retrieval task, designed as a targeted stress test of eviction- and page-based methods, TokenButler achieves near-oracle accuracy where existing methods fail. On general-purpose long-context benchmarks (RULER, LongBench, LongMemEval), TokenButler achieves competitive or superior performance, up to on-GPU speedup using our proposed *prediction interval with neighbor fetching* that amortizes predictor cost while maintaining accuracy within , and up to reduction in latency compared to Dense Attention with CPU offloading, enabling efficient decoding for long contexts in limited GPU memory systems. Code [is available](https://anonymous.4open.science/r/TokenButler-0B6F).

Primary area:
generative models
Keywords:
token sparsity, token pruning, sparsity, pruning, language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_29TbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TokenButler, a learned, query-aware token selection framework for accelerating long-context LLM decoding. Unlike KV-cache eviction methods that permanently discard tokens or page-based methods that retrieve coarse groups of tokens, TokenButler preserves the full KV cache and dynamically selects individual tokens to participate in attention computation.

The key component is a lightweight importance predictor trained by distilling the attention distributions of a frozen backbone LLM. Specifically, the method predicts low-dimensional importance queries from intermediate hidden states and combines them with learned projections of cached keys to estimate token importance in subsequent transformer layers. The framework further incorporates sink tokens, local attention windows, deferred key projection, prediction intervals, and neighbor fetching to reduce inference overhead.

The paper evaluates TokenButler on a synthetic co-reference retrieval benchmark, RULER, LongBench, LongMemEval, and AIME24. It reports competitive accuracy under constrained KV-access budgets, up to 1.6× on-GPU decoding speedup, and up to 7.6× latency reduction under CPU-offloaded KV-cache settings. The authors also provide a preliminary integration with SGLang.

Overall, this work addresses an important practical problem in efficient LLM inference and presents a reasonably comprehensive system. However, the incremental nature of several components and insufficiently isolated experimental comparisons make it difficult to determine how much of the observed improvement is attributable to the proposed learned importance prediction mechanism itself.

Critical strengths

S1. A practically important problem with a well-motivated design.

The paper addresses the memory-bandwidth bottleneck of long-context decoding, where accessing the entire KV cache becomes increasingly expensive. Preserving the full cache while dynamically selecting a small subset of tokens is a reasonable design choice, particularly for tasks where previously unimportant information may become relevant later.

S2. A lightweight and potentially reusable importance prediction mechanism.

TokenButler combines hidden-state-based query prediction with low-dimensional projections of the actual cached keys. This design allows it to estimate token importance without computing full-dimensional query-key scores. The predictor is trained while keeping the backbone frozen and introduces only approximately 0.3–0.9% additional parameters. Training on 1K-token contexts while evaluating up to 64K is an interesting empirical result.

S3. Encouraging results across several evaluation settings.

The evaluation covers multiple backbone models and benchmarks, including RULER, LongBench, LongMemEval, and AIME24. At an 8× KV-access reduction on LongBench, TokenButler obtains average scores of 44.43 and 45.00 on Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct-1M, respectively, compared with dense baselines of 43.27 and 44.79. These results indicate that substantial reductions in KV access can be achieved without large quality degradation in the evaluated settings.

S4. Attention to system-level efficiency.

The paper goes beyond an isolated importance predictor and considers implementation issues such as deferred key projection, contiguous KV buffers, batched token selection, and prediction intervals. The reported 1.6× on-GPU speedup at 128K and preliminary SGLang integration provide useful evidence of practical applicability.

S5. A targeted evaluation of long-range token reactivation.

The synthetic co-reference benchmark illustrates a meaningful limitation of permanent eviction and coarse page-based retrieval. Although the benchmark has limitations discussed below, it provides a clear diagnostic setting for studying tokens whose relevance changes across decoding steps.

Critical weaknesses

W1. Limited conceptual novelty relative to existing token-selection and learned sparse-attention approaches.

The main components of TokenButler are closely related to existing approaches: query-aware token-level KV selection has been studied in TokenSelect; lightweight learned indexers have been explored in DeepSeek Sparse Attention and IndexMem; and reusing token selections across decoding steps is closely related to the Selection Cache mechanism in TokenSelect.

While TokenButler combines these ideas in a useful way, it remains unclear which component represents the central algorithmic novelty beyond replacing full-dimensional importance scoring with a learned low-dimensional approximation. In particular, the paper would benefit from a more direct distinction between its contribution and the selection reuse mechanism in TokenSelect.

W2. Missing key ablations to isolate the benefit of learned importance prediction.

The paper does not sufficiently establish whether learning the importance predictor is necessary for the observed improvements. A particularly important missing baseline is a non-learned low-dimensional token selector using random projections, PCA-based projections, or other lightweight approximations of query-key similarity under the same dimensionality, token budget, and kernel implementation.

Such comparisons would distinguish the benefit of learned attention distillation from the computational advantages of low-dimensional scoring alone. Additional ablations separating query prediction, learned key projection, and their interaction would make the central contribution more convincing.

W3. Insufficiently controlled end-to-end efficiency comparisons.

The paper reports promising decoding speedups, but the evidence is distributed across different models, hardware configurations, token budgets, and serving implementations. In particular, the comparison against TokenSelect in Appendix J uses a modified TokenButler variant that removes head-specific predictions to fit the compared retrieval implementation.

Furthermore, the 7.6× result is obtained in a CPU-offloading regime where reducing host-to-device KV transfers can naturally offer large benefits. This is a useful system result, but it should not be interpreted as an intrinsic 7.6× advantage of the learned importance predictor over other sparse selection methods.

A matched end-to-end comparison against TokenSelect and Quest using the same backbone, hardware, KV residency, selection budget, and optimized sparse-attention backend would considerably strengthen the efficiency claims.

W4. The proposed prediction interval mechanism is not sufficiently isolated from neighbor fetching.

Prediction intervals and neighbor fetching are introduced together to amortize selection overhead and maintain accuracy. However, neighbor fetching expands each selected token with an additional position, potentially improving coverage simply by accessing more tokens.

The paper should provide a controlled factorial ablation comparing per-step prediction, periodic prediction, neighbor fetching, and their combination under exactly matched total KV-access budgets.

Additionally, the observation that periodically refreshing token selections yields small accuracy losses is interesting, but the paper does not sufficiently characterize when temporal reuse breaks down, especially under abrupt shifts in token importance.

W5. Some evaluation settings and generalization claims remain insufficiently validated.

The synthetic co-reference benchmark is explicitly designed to expose weaknesses of eviction-based and page-based methods. Its extremely large performance gaps are therefore less informative about average performance on realistic workloads. The main text should avoid treating these gaps as evidence of general superiority.

Moreover, the reported superiority on standard benchmarks is not uniform. On Llama-3.1-8B-Instruct, TokenButler achieves 89.93 on RULER, only slightly above SnapKV at 89.68 and below dense attention at 91.89. On Qwen2.5-7B-Instruct-1M, its LongBench average of 45.00 is only 0.21 points above dense attention.

The paper would also benefit from evaluations across more aggressive sparsity ratios, multiple training seeds, longer decode trajectories, and additional reasoning workloads. These are particularly relevant because the authors acknowledge a substantial decline in importance recall at very small token budgets.

Questions for authors

Q1. How much of the improvement comes from learning rather than dimensionality reduction?

Could the authors compare TokenButler with random projection, PCA-based projection, and other training-free low-dimensional query-key approximations, using the same scoring dimension (d'=16), sparse-attention implementation, and KV-access budget? This would help isolate the contribution of attention distillation.

Q2. How does TokenButler compare with a carefully optimized TokenSelect implementation?

Could the authors provide end-to-end latency and throughput comparisons under identical hardware, model, context length, token budget, and KV residency conditions? It would be particularly helpful to distinguish selection overhead from sparse-attention execution and KV-gather costs.

Q3. What is the independent contribution of prediction intervals and neighbor fetching?

Could the authors provide a 2×2 ablation with and without prediction intervals and neighbor fetching, while controlling for the total number of retrieved tokens? How does accuracy change when attention shifts abruptly to distant, previously unselected tokens?

Q4. What makes token importance predictable across layers and context lengths?

The predictor is trained on sequences of length 1K but evaluated at 64K. Could the authors provide attention top-k recall measurements, rank correlations, or other diagnostics across layer distances, attention heads, context lengths, and decoding positions? In particular, how does prediction quality change under distribution shifts or on contexts substantially longer than those seen during training?

Q5. How robust are the results under more aggressive sparsity and longer generation?

Could the authors report accuracy-efficiency Pareto curves at multiple KV-access budgets, particularly below 10%, and provide additional evaluations on longer reasoning trajectories? Reporting variance across training and evaluation seeds would also help establish whether the relatively small gains over strong baselines are statistically reliable.

1 comment

Thank you for the thorough review. Full results are in App. J–N of the revision.

W1 (novelty; Selection Cache). (1) Importance is predictable ahead of the layers that use it: a module with <1% of the parameters, trained in <10 GPU-hours for the 8B model with the LLM frozen, predicts which tokens the next four layers need; DSA retrains the model, and LookaheadKV and IndexMem evict permanently. (2) Cross-layer look-ahead: TokenSelect and the Lightning Indexer score the cache with each layer's real query at every layer and step. TokenButler scores a layer group at once against 16-d key sketches (20.1 vs 35.3 ms/token of selection and gather at 128K, Table 14). (3) Scheduled reuse: the Selection Cache (now in Sec. 2 as the closest prior mechanism) still computes and compares the query at every step and layer; our interval does no work on non-refresh steps (1.33× / 1.69× dense speed at 64K / 128K vs 0.79× / 1.02×, Table 14). The two are complementary.

W2 / Q1 (is learning necessary?).

Key sketches at the same d'=16, budget and kernels (RULER 64K, per-step, 8,832 positions per layer).

Selection ruleQueryRULER
Dense (no selection)–92.02
Random projectionreal86.64
TokenButler's learned key sketchreal92.09
TokenButlerpredicted up to 4 layers ahead89.97

A random projection with the real query is not enough (86.64). The learned key sketch is at dense level with the real query and 3.3 points above the random projection with a predicted one. A PCA sketch (Loki-style) also needs each layer's real query, so it scores inside every layer: 0.77× dense speed at this budget, and 0.90× at the budget where it reaches TokenButler's accuracy (4K: 89.97), against 1.25× for TokenButler with an interval of 16 (90.01; Table 7; all rules per task in Table 13). Predicting the query up to four layers ahead is what the learned predictor adds: one scoring pass serves a group of layers before they run, and with an interval of 16 decoding reaches 1.33× at this budget.

W3 / Q2 (controlled end-to-end efficiency).

Decode latency in ms/token (speed vs dense), K/V on GPU, 8,832 positions per layer.

Method64K128Kselection + gather, 128K
Dense37.149.5–
TokenSelect49.6 (0.75×)63.2 (0.78×)35.3 ms
TokenSelect + Selection Cache47.0 (0.79×)48.6 (1.02×)20.7 ms
TokenButler, i=140.3 (0.92×)48.6 (1.02×)20.1 ms
TokenButler, i=1627.8 (1.33×)29.4 (1.69×)1.3 ms

TokenSelect is our port, optimized so as not to penalize it. With an interval TokenButler is the fastest method at both lengths, within 4–9% of a zero-selection-cost lower bound. Fig. 7 used TokenSelect's codebase with a per-layer variant of TokenButler; Table 14 is the converse, with the full per-head method. We agree that the CPU-offloading speedup is a system result against dense; Sec. 4.3 now says so.

W4 / Q3 (interval vs neighbor fetching; abrupt shifts).

Interval × neighbor fetching at the same 8,832 positions per layer (RULER 64K; B is halved with neighbors).

IntervalWithout neighborsWith neighbors
i=189.9789.15
i=488.0388.44
i=1687.3088.29

At the same positions per layer, i=16 decodes 1.43–1.45× faster than per-step prediction. Neighbor fetching recovers accuracy exactly when selections are stale (i=4: 88.03 → 88.44; i=16: 87.30 → 88.29, with N-MQ 82.6 → 90.9) and is not needed at i=1. Sec. 3.4 now presents it as a remedy for stale selections, and this factorial is now in the main text (Table 8). A fixed schedule can be stale after an abrupt change of focus; a forced refresh at known boundaries or a similarity trigger are compatible remedies.

W5 (benchmark framing). Sec. 4.1 now introduces the co-reference benchmark as a targeted diagnostic of token reactivation and states that RULER, LongBench and LongMemEval serve as the general-purpose evaluation. On realistic workloads the claim is dense-level quality at a fraction of the KV access: LongBench 44.43 vs 43.27 at 8× lower KV access, LongMemEval 26.6 vs 27.4. Lower budgets and seeds are in Q5, 128K in Table 7.

Q4 (what makes importance predictable). Much of the attention mass is structural: up to 64K, 81–87% lies on the sink and recent window. The heaviest remaining tokens are recoverable from an earlier hidden state: the predicted query finds 93% of the top-16 tokens at 16K and 53% at 64K. Predictability degrades with length (training used 1K-token sequences), now stated as a limitation (Sec. 5); diagnostics are in App. L.

Q5 (Pareto below 10% budget; seeds). RULER 64K, per-step prediction:

Positions per layer (% of 64K)1,664 (2.5%)2,688 (4.1%)4,736 (7.2%)8,832 (13.5%)17,024 (26%)
TokenButler (dense 92.02)83.1286.0288.6189.9791.31

Three training seeds give 89.97 / 89.96 / 90.07 (App. N).

Reviewer_zihBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TokenButler, a lightweight predictor trained through attention distillation to dynamically select important tokens while retaining the full KV cache. The paper further introduces prediction intervals with neighbor fetching to amortize inference overhead. Experiments on a synthetic co-reference task, RULER, LongBench, and AIME24 demonstrate competitive task performance and substantial decoding speedups.

Critical strengths
  1. TokenButler preserves the full KV cache, allowing previously unimportant tokens to become accessible when needed, unlike eviction-based methods such as H2O and SnapKV.

  2. The experimental evaluation is broad, covering a synthetic co-reference task, RULER, LongBench, and AIME24, with comparisons against several categories of KV-cache sparsity and compression methods.

Critical weaknesses
  1. The conceptual novelty appears limited relative to existing approaches such as TokenSelect, the Lightning Indexer, and LookaheadKV, which already explore query-dependent or learned token selection.

  2. The paper lacks a systematic analysis linking the fidelity of teacher-attention approximation, important-token recall, and downstream task accuracy.

  3. The paper reports improvements in latency and throughput but does not quantify peak GPU or host memory usage, particularly in experiments involving CPU offloading.

Questions for authors
  1. Can attention distillation, while improving a model's alignment with the attention distribution of a teacher model, simultaneously preserve its ability to perform downstream tasks?

  2. Does the fixed interaction dimension d'=16 remain effective for models substantially larger than the evaluated 1B–8B backbones?

  3. Could you report accuracy alongside latency at 128K–1M contexts under matched token budgets and prediction intervals? The current accuracy analysis is at 64K, while speedups are measured at longer contexts.

  4. Section 3.4 states that "neighbor fetching" expands the B initially selected positions into 2B unique positions, while Table 8 specifies a "sparse budget" of 8K. Does this budget refer to the initially selected positions, or to the total number of positions accessed after the neighbor-fetching expansion?

2 comments

Thank you for your careful review and thoughtful questions. We updated the manuscript based on your feedback and hope the new version does not fail to impress you.

W1 (novelty). TokenButler selects top tokens before a group of layers runs, from a predicted query, with the LLM frozen and nothing evicted. TokenSelect and the Lightning Indexer score with each layer's real query at every layer; DSA retrains the model in the sparse regime while TokenButler does not require retraining; LookaheadKV also learns its selector on a frozen backbone, but it predicts once, during prefill, which prompt tokens the response will attend to and prunes the rest. Table 7 shows what our look-ahead buys: at the same accuracy TokenButler decodes 1.4× faster than TokenSelect (90.01 at 1.25× vs 90.05 at 0.86×). Sec. 2 now states this positioning.

W2 (fidelity, recall, accuracy). We recorded the dense model's exact attention at each sparse layer and decode step, together with the selector's choice (per-step prediction, B=8K):

ContextTop-16 tokens foundAttention mass captured, TokenButler / random subsetTotal attention mass kept
16K0.930.88 / 0.540.98
32K0.770.70 / 0.260.95
64K0.530.46 / 0.130.89
128K0.370.22 / 0.060.66

Up to 64K the sparse layers keep 89–98% of the attention mass and RULER stays within about 2 points of dense (89.97 vs 92.02 at 64K). At 128K the kept mass drops to 0.66 and accuracy follows (80.23 vs 84.85), mainly on N-MK2. The predictor was trained on 1K-token sequences; training on longer ones is the direct remedy (App. L, Sec. 5). Per-layer and per-head breakdowns are now in App. L.

W3 (memory). With K/V on GPU at 128K, TokenButler adds 2.1 GiB (prefill peak 38.5 vs 36.4 GiB). With K/V on the host (32 / 64 / 128 GiB at 256K / 512K / 1M), GPU peak is 20.3 / 23.5 / 29.9 GiB, so a 1M-token context is served from one 48 GB GPU, and decoding is 5.7× / 6.6× / 7.5× faster than dense (App. M, Table 18).

Q1 (does distillation preserve downstream ability?). Yes. The LLM's weights are never updated: distillation trains only the external predictor, which chooses which cached tokens each layer reads. With all tokens selected the output is the dense model's, so quality depends only on the selected set (LongBench 44.43 vs 43.27 dense; LongMemEval 26.6 vs 27.4, Table 3).

Q2 (d'=16 for larger models). d' is the size of a per-head sketch, and the head dimension stays at 128 in larger models, which add heads and layers. A d'=16 predictor trained with the unchanged recipe on Qwen2.5-14B-Instruct reaches 92.77 vs 92.94 for dense on RULER 16K, a smaller gap than on Llama-3.1-8B (0.17 vs 0.73; App. A.2).

Q3 (accuracy alongside latency). Table 7 reports both for each configuration: 90.01 at 1.25× dense speed at 64K (dense 92.02) and 81.19 at 1.44× at 128K (dense 84.85). Contexts of 256K–1M exceed the model's trained window, so we report latency and memory there and limit accuracy claims to ≤128K.

Q4 (budget definition). The 8K budget counts predictor-selected positions, before neighbor fetching. Each sparse layer also reads 128 sink and 512 recent positions, and neighbor fetching adds one neighbor per selected position, so the i=1 row of the interval ablation (Table 6) reads 8,832 positions per layer and the i>1 rows up to 17,024. This is now stated in Sec. 3.4 and the captions. At exactly matched positions (8,832), i=16 with neighbor fetching reaches 88.29 vs 89.97 for per-step prediction, while decoding 1.43× faster.

Thank you for the detailed response and additional experiments. We still have some concerns about the novelty, as compared with other methods, the main improvement is to predict tokens for future layers and share the selection results. However, the authors address most of our questions and provide enough experimental evidence, so we increase the score from 2 to 3.

Reviewer_277NOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TokenButler trains a lightweight predictor to select relevant KV-cache tokens for each decoding query. It distills attention distributions from a frozen language model, scores tokens through low-dimensional query and key projections, and retains the full KV cache so previously skipped tokens can be selected later. A prediction interval and neighbor fetching reduce the cost of repeated selection. The paper evaluates quality on several long-context benchmarks and reports decoding speedups on GPU and with CPU offloading.

Critical strengths
  • The method can be added to a frozen model with less than 1% additional parameters and modest predictor training time.

  • The evaluation covers retrieval, long-context understanding, memory, and reasoning tasks, alongside latency measurements.

  • The paper measures predictor overhead and proposes a practical way to amortize it, instead of treating token selection as free.

Critical weaknesses
  • The TokenSelect comparisons use different model sizes and context lengths for accuracy and speed. A fairer comparison would make the efficiency advantage clearer.

  • The main quality results are at 64K context, while the highlighted speedups are at 128K and beyond. Quality results at those longer lengths would clarify the tradeoff.

Questions for authors

Could you compare TokenButler and TokenSelect on the same model, hardware, context length, and KV-access budget, reporting both accuracy and decoding latency?

1 comment

Thank you for the positive assessment and the two clear requests. Both are answered with new experiments (Table 7 and App. J of the revision).

W1 / Q1: TokenSelect on the same model, hardware, context length and budget. Fig. 7 compares throughput inside TokenSelect's codebase. We now add the comparison you ask for: Llama-3.1-8B-Instruct, one A6000, same context length and dense prefill, accuracy and decode latency together (Table 7, App. J). It replaces our earlier accuracy comparison with TokenSelect.

  • At the same accuracy, TokenButler is 1.4× faster. At its default 2K budget TokenSelect scores 90.05 at 0.86× dense speed; TokenButler reaches 90.01 at 1.25×.
  • More accurate and faster. With an interval of 4, TokenButler scores 90.26 at 1.12× dense speed, against 90.05 at 0.86× for TokenSelect: higher accuracy and 1.3× faster decoding (Table 13, App. J).
  • Same budget (8K). TokenSelect decodes at 0.75× / 0.78× dense speed at 64K / 128K (0.79× / 1.02× with its Selection Cache), because it scores every cached key with the full-dimensional query in every layer at every step. TokenButler with an interval of 16 runs at 1.33× / 1.69×.

TokenSelect's released code applies plain RoPE, whereas Llama-3.1 uses frequency-scaled RoPE, so the comparison uses our implementation of its selection in our backend, optimized as in its released code. Both methods get the same dense prefill, so the comparison isolates decoding, which is the phase TokenButler targets.

W2: quality at 64K, speedups at 128K and beyond. Accuracy and speed are now reported for the same configurations (Table 7): at 64K, i=16 with a 16K budget reaches 90.01 (dense 92.02) at 1.25×; at 128K, i=16 with a 32K budget reaches 81.19 (dense 84.85) at 1.44×. At 128K the largest gaps are on N-MQ and N-MK2; the wider gap is now stated as a limitation (Sec. 5). LongMemEval, with 105K–114K-token prompts, is within 1 point of dense (26.6 vs 27.4). The interval ablation is now in the main text with latency and positions per layer (Table 6).

Beyond 128K the context exceeds the model's trained window, and dense decoding takes 3.1 s/token at 256K and 6.3 s/token at 512K on our hardware, so we do not report RULER there. The revision presents the 256K–1M results as latency and memory measurements (App. M) and limits accuracy claims to ≤128K. The predictor was trained on 1K-token sequences only; we expect training on longer sequences to improve accuracy at long contexts and leave this to future work (Sec. 5).

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback. All new experiments use matched settings: Llama-3.1-8B-Instruct (unless stated), one RTX A6000, one inference backend. Everything below is in the uploaded revision which we believe makes the submission much stronger.

1. Accuracy and speed at the same operating point (277N W2, zihB Q3). At 64K, TokenButler with a prediction interval of 16 and a 16K budget keeps 98% of dense accuracy (90.01 vs 92.02) while decoding 1.25× faster. At 128K, with a 32K budget, it keeps 96% (81.19 vs 84.85) at 1.44× (Table 7).

2. Matched comparison with TokenSelect and a PCA sketch (277N W1/Q1, 29Tb W3/Q2). Fig. 7 compares with TokenSelect inside TokenSelect's code; we now add the converse, TokenSelect inside our backend. At the same accuracy, TokenButler decodes 1.4× faster than TokenSelect (90.01 at 1.25× vs 90.05 at 0.86× dense speed); TokenSelect and the PCA sketch stay slower than dense at 64K (Table 7, App. J).

RULER average and decode speed vs dense (i = prediction interval; pos. = KV positions read per layer).

ContextMethodpos.RULERspeed
64KDenseall92.021.00×
64KTokenSelect, default 2K budget2,68890.050.86×
64KPCA sketch (Loki-style), 4K budget4,73689.970.90×
64KTokenButler, i=16, B=16K17,02490.011.25×
128KDenseall84.851.00×
128KTokenButler, i=16, B=32K33,40881.191.44×

3. What learning adds (29Tb W2/Q1). At the same d'=16 and budget, a random projection scored with the real query reaches 86.64. TokenButler reaches 89.97 with a query predicted up to four layers ahead, and its learned key sketch is at dense level when given the real query (92.09 vs 92.02; App. J.1).

4. Prediction interval vs neighbor fetching at matched positions (29Tb W4/Q3, zihB Q4). At identical positions per layer, refreshing every 16 steps with neighbor fetching instead of every step costs 1.7 points (88.29 vs 89.97) and decodes 1.43× faster (Table 8, App. K).

5. The same d'=16 on a larger backbone (zihB Q2). With the unchanged recipe on Qwen2.5-14B-Instruct, TokenButler reaches 92.77 vs 92.94 for dense on RULER 16K (App. A.2).

6. From attention fidelity to accuracy (zihB W2, 29Tb Q4). Up to 64K the sparse layers keep 89–98% of the attention mass, and at 64K the selected set captures 3.6× the mass of a random subset of the same size (App. L).

7. Memory and CPU offloading (zihB W3). TokenButler adds 2.1 GiB of GPU memory at 128K. With K/V in host memory, a 1M-token context is served from one 48 GB GPU, decoding 5.7–7.5× faster than dense at 256K–1M (App. M).

8. Robustness (29Tb W5/Q5). Three training seeds give 89.97 / 89.96 / 90.07 on RULER 64K, and accuracy degrades gracefully with the budget (88.61 at 7.2% of the context, 86.02 at 4.1%; App. N).

Other revisions. The interval ablation is now in the main text with latency and positions per layer (Table 6). The budget definition is in Sec. 3.4, Sec. 2 sharpens the positioning against prior work, and Sec. 5 adds a long-context limitation. We also took the opportunity to improve the manuscript and make experimental details more explicit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.