acceptodds
Under review as a conference paper at ICLR 2027

TokenGate: Fast and Memory-Efficient Prefill with Trainable Token Pruning

Abstract

Expert human readers process large inputs (e.g., code repositories and long documents) by skimming to closely read only what matters. On the other hand, language models process all input tokens through each model layer, resulting in significant processing costs for long-context inputs. We introduce TokenGate, a small, drop-in component that extends existing trained models to enable end-to-end learnable token pruning over long-context inputs for fast and memory-efficient prefill. To enable end-to-end learning despite the non-differentiable top- operator for token pruning, we introduce token-gated attention, a mechanism inspired by Mixture-of-Experts that re-weights attention logits using token importance scores. Across diverse tasks, TokenGate achieves - faster time-to-first token (TTFT) and - lower peak memory at no accuracy degradation for prefill in transformers. Compared to the best-performing prior methods for token-pruning, TokenGate gives up to faster prefill and lower peak memory. TokenGate natively extends both standard transformers (e.g., Qwen3) and hybrid-attention models (e.g., Qwen3.5) with simple lightweight adaptation comparable to standard parameter-efficient fine-tuning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
token pruning, prefill, long-context, inference optimization

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_KdDnOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper propose a learned lora-style approach to prefill token pruning. The main idea is to train an MLP that attempts to score the important of each token in prefill, which is then used to prune tokens. To compensate the downstream affects of the pruned tokens, the method adds adapters to recover from the distribution shift the pruning introduces. The paper trains unique adapters for task-specific domains and focuses on qwen3-1.7b as the main model target. Compared to prior strong (but training free) baselines (e.g, FastKV) the methods shows improvement across a pareto frontier.

Critical strengths

The paper lays out the problem space and prior art quite well. Figure 1 makes the method immediately clear to anyone with knowledge of prior work. The idea itself is simple (making it easier to defend) and clearly improves over prior strong baselines. The authors provided TTFT speedup on common hardware setups (A100 at batchsize 1) making the claimed improvements seem concrete.

Critical weaknesses

The paper trains separate adapters for each of 5 specific evaluated tasks. While this can make sense in some settings (e.g., where we know only one type of document will be routed to the serving engine), it does decrease the usefulness of the approach. It is unclear why this method could not be trained in a mixed-task setting in-order to see how well it generalizes across domains.

While the authors acknowledge the method requires training relative to prior methods, it still needs to be mentioned as a weakness. Since prior methods (e.g., FastKV) are training free, it arguably change the category of the method. This is a bit related to the prior point, because the training component in a specific domains sort of stacks these effects.

Finally, while the analysis is well done, it is on mostly on a single small model (Qwen3 1.7B). It is unclear how large of a gap would exist at 10x or 100x this scale between TokenGate and prior art.

Questions for authors

Can you report any experiments you did for a mixed-task training regime? Reporting a pareto curve of a single model across all domains compared relative to FastKV would make it more clear how much the approach could work in a general setting like FastKV.

Similarly, could you provide any indication that this method could be extended to say a ~10B parameter model like the Llama 3 8B model FastKV was evaluated on? I believe the training would not be too difficult on the A100 setup, but a full re-evaluation of all tasks would take a lot more time.

How does TokenGate's token selection differ from FastKV? Another framing is what is the % overlap in selected tokens? If the overlap is high, it implies to me the benefit would come mainly from the adapter layers vs improved token selection.

4 comments
AuthorsOpenReview ↗

Response to Reviewer KdDn

We thank the reviewer for their valuable, detailed review, which we believe helped us to strengthen the paper. We are thrilled that the reviewer finds the paper to be clearly written, and highlights that our method is simple and defensible, providing superior performance over strong baselines, which was one of our primary contributions. Following the reviewer's suggestions, we ran two new experiments, which led us to two new findings: TokenGate’s performance extends to larger models e.g., Qwen3-8b, and exhibits strong multi-task performance. We also clarify how our ablations separate the contribution of token selection from that of the adapter. We respond to each point inline below.

The paper trains separate adapters for each of 5 specific evaluated tasks. While this can make sense in some settings…it does decrease the usefulness of the approach...

Can you report any experiments you did for a mixed-task training regime? ...

We agree this is an important question and we believe training a general TokenGate model that generalizes across downstream tasks is an exciting opportunity for future work. To study this, we provide a new, controlled multi-task experiment, which demonstrates that a single trained TokenGate model can transfer across tasks. We trained a single multi-task TokenGate model on a mixture of two disparate tasks, Biodex and NIAH. The two tasks call for very different pruning patterns (Appendix E, Table 4). As Table 1 shows, the multi-task model closely tracks the single-task models on both tasks, and on both tasks, it continues to outperform FastKV's accuracy–TTFT frontier. This shows that one gate can learn task-appropriate importance across domains without per-task specialization, which supports the feasibility of training a general TokenGate model on broader task mixtures.

While the authors acknowledge the method requires training relative to prior methods, it still needs to be mentioned as a weakness… Thank you for this point and we will state this explicitly in a new Limitations section, copied below:

Limitations. Unlike training-free token pruning methods such as FastKV, TokenGate requires a training stage to learn the scoring network jointly with a LoRA adapter. Its cost is comparable to standard LoRA fine-tuning, and a single checkpoint serves all inference-time token budgets and longer contexts. Our experiments train on individual tasks or small task mixtures. Training TokenGate on a broad instruction-tuning mixture and evaluating zero-shot transfer to unseen tasks is an important direction for future work.

Finally, while the analysis is well done, it is on mostly on a single small model (Qwen3 1.7B). It is unclear how large of a gap would exist at 10x or 100x this scale between TokenGate and prior art. Similarly, could you provide any indication that this method could be extended to say a ~10B parameter model like the Llama 3 8B model FastKV was evaluated on? ... We thank the reviewer for this suggestion, and we have added new evaluation training TokenGate on Qwen3-8B, which has led us to a new finding: TokenGate’s performance generalizes to larger models (Table 2). We find the TokenGate model trained with the Qwen3-8B backbone matches the fine-tuned base model’s full-context accuracy at 2.2× lower TTFT. It also matches FastKV's best accuracy at 2.0× lower TTFT. We will add these results to the manuscript.

How does TokenGate's token selection differ from FastKV? Another framing is what is the % overlap in selected tokens? If the overlap is high, it implies to me the benefit would come mainly from the adapter layers vs improved token selection.

This is a great question, and our ablation in Section 5.2 (Figure 5, 2WikiMultihopQA) separates the two effects. FastKV ranks tokens by the attention they receive from the last few queries. At layer 1, this attention-based heuristic carries little signal, so on the same LoRA backbone FastKV's selector performs close to random-K selection at every budget. In contrast, TokenGate's learned scorer substantially outperforms both. For example, at 2k kept tokens, TokenGate reaches 0.70 F1, compared to 0.37 for FastKV and 0.29 for random-K. Holding the selector fixed, we also observe a co-adaptation effect: with the same random-K selection, the TokenGate backbone outperforms the LoRA backbone at every budget (0.37 vs. 0.29 F1 at 2k tokens). The gains therefore come primarily from improved token selection, with a complementary gain from co-adaptation, and not from the adapter alone. We note that all baselines, including FastKV and token-free pruning methods, also run on a backbone fine-tuned with the same LoRA configuration (Section 5).

Thank you for the new experiments and the Limitations section. I will keep my score.

The new multi-task results are encouraging but only cover 2 of the 5 tasks. Since a single model on all 5 tasks probably takes ~12 hours based on the training times in your paper, I suggest including it in the final version.

In your new table 1, it is surprising that the multi-task TTFT is actually lower than the single task ones. Also, the speedups seem wrongly computed (87/29 is 3.0 not 3.3). This would be worth double-checking.

Also, it seems like on a larger model the gap between TokenGate and FastKV shrinks substantially (from 1.9x to ~1.2x). This was my original concern.

Finally, I am going to push back a bit on your selector ablation for FastKV. No one would run FastKV at layer 1, so of course the tokens selected at that layer is weak. We could take FastKV's tokens selected at layer 13 and the replay them pruning at layer 1, which would separate this question a bit. But, still I understand why you felt to frame it this way.

AuthorsOpenReview ↗

Response to Reviewr KdDN

We sincerely thank the reviewer for their response and helpful feedback on our updates. We provide clarifications below:

In your new table 1, it is surprising that the multi-task TTFT is actually lower than the single task ones. Also, the speedups seem wrongly computed (87/29 is 3.0 not 3.3). This would be worth double-checking.

Thank you for this point and we sincerely apologize for two typos in the table, which have been resolved. The table now shows the corrected speedups, and the experiment demonstrate that the multi-task TTFT speedup is comparable or higher to the TTFT speedup of the single-task TokenGate models compared to the full-context baseline. We are also currently training a multi-task.

Also, it seems like on a larger model the gap between TokenGate and FastKV shrinks substantially (from 1.9x to ~1.2x). This was my original concern.

Thank you for raising this, and to clarify, our experiments, summarized in Table 2, demonstrate that TokenGate performance is 1.84x higher TTFT at full-context accuracy compared to FastKV, maintaining strong pareto-optimal performance across token budgets, with a significant margin in the high-accuracy regime.

The new multi-task results are encouraging but only cover 2 of the 5 tasks. Since a single model on all 5 tasks probably takes ~12 hours based on the training times in your paper, I suggest including it in the final version.

Thank you for this feedback and we agree. To clarify, we are currently running the TokenGate model and base model fine-tuned on all tasks, which we will plan to add to the paper. We note that for both convergence across tasks takes significantly longer (several days on our available machines) compared to the single task setup, which is why we have first provided a smaller scale, controlled multi-task experiment. We will update our response here and the manuscript with the new multi-task experiments once they have completed as well.

Finally, I am going to push back a bit on your selector ablation for FastKV. No one would run FastKV at layer 1, so of course the tokens selected at that layer is weak. We could take FastKV's tokens selected at layer 13 and the replay them pruning at layer 1, which would separate this question a bit. But, still I understand why you felt to frame it this way.

Thank you for this feedback, and we are happy to provide a new ablation. Our original setup aimed to answer the question, How does TokenGate's token selection differ from FastKV? We found that TokenGate enables more aggressive pruning at earlier layers due to its learnable gate module as opposed to FastKV's attention-based heuristic. We agree that the framing of taking FastKV's tokens at layer 13 and applying them at layer 1 would help develop our analysis further. We will add this and share additional results here.

Thanks for you reply. I will raise my score to 4. Overall, I find the work solid and interesting.

Reviewer_7GQ2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TokenGate, a method for accelerating LLM prefilling. It uses the first few layers of a Transformer as an indexer to predict a score for each token, and then retains only the positions of tokens with higher predicted scores in the subsequent layers. The authors incorporate these scores into the attention logits, enabling end-to-end training of the entire indexer and model. The authors claim that this method achieves both acceleration and memory savings, and performs well on a range of downstream tasks.

Critical strengths

Accelerating LLMs' prefilling is an important research problem in the community, and TokenGate attempts to address this issue while performing well on the benchmarks provided by the authors. The paper is well written and easy to follow.

Critical weaknesses

TokenGate's novelty and technical challenges are limited. While reading the paper, I did not come across any technical innovations that particularly excited me. In my view, the paper is essentially an "A + B" approach that combines FastKV with sparse attention indexer techniques, and this combination does not appear to present any significant independent challenges. Compared to FastKV, this paper is highly incremental. TokenGate's motivation for mid-layer token pruning largely overlaps with that of FastKV, while its training approach, which achieves differentiable pruning by incorporating token scores into attention logits, has similarities to techniques already explored in sparse attention research.

One major contribution claimed by the authors, to which they devote considerable space, is their token-gated attention, designed to address the challenge of training non-differentiable top-k operators end-to-end. This problem also exists in sparse attention, and researchers have proposed numerous solutions, including the use of straight-through estimators [1] or, as in DeepSeek Sparse Attention [2], optimizing the KL divergence with the attention distribution. It's unclear what advantages this design offers, especially compared to the numerous existing sparse attention methods.

[1] Bengio et al. "Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation." 2013. [2] DeepSeek-AI. "DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models." 2025.

Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7GQ2 (Part 1)

We sincerely thank the reviewer for their detailed feedback, which we believe helped us to strengthen the paper. We are encouraged that the reviewer finds our paper to address an important research problem and highlights the significant performance improvements our method demonstrates across diverse downstream tasks. The reviewer raises a concern about limited novelty, relative to FastKV and sparse attention. We ran a new experimental ablation comparing our token-gated training to STE, and we also clarify the key differences to prior work. We will add this discussion and the suggested references to the manuscript. We respond to points inline below.

TokenGate's novelty and technical challenges are limited. ... The paper is essentially an "A + B" approach that combines FastKV with sparse attention indexer techniques...

We agree that TokenGate shares FastKV's motivation, and we view FastKV as an important baseline; it is among the strongest prior methods in our main results. TokenGate advances prior methods by making token selection learnable end-to-end with the standard task loss, unlike FastKV which cannot train its selector end to end, and unlike sparse attention methods, like DSA, that require auxiliary losses. We designed each component of TokenGate to make it a simple, drop-in addition that makes minimal changes to the existing training and serving stack, while still delivering large performance gains. Our design allows TokenGate to outperform FastKV and all other baselines, with up to 6.4× lower TTFT and 2× lower peak memory at matched accuracy (Section 5.1). Specifically, our design features:

(a) A learnable scoring network trained with the task loss. In contrast to TokenGate, FastKV scores tokens with an attention heuristic, and attention scores carry little signal at early layers, so FastKV must prune at mid-depth (layer 13 of 28), whereas our design, a trainable MLP scoring network scores tokens effectively at layer 1 (Figure 5). In contrast to sparse attention methods, which apply top-k decisions at each layer, our learnable gate entails a significantly different design, applying a single pruning decision for all subsequent layers. makes a single pruning decisions which applies for all subsequent layers. Sparse attention methods also cannot supervise their selector with the task loss. For example, DSA trains its indexer with an auxiliary KL loss to imitate dense attention, and its sparse attention still processes every token through every layer.

(b) Token-gated attention with e2e adaptation for pretrained models. In sparse attention methods (DSA, NSA, MoBA, SeerAttention), scores only select keys and never enter the attention computation. Our gate follows MoE routing instead (Section 4.2). The closest related design, MoSA (Piękos et al., 2025), scales each head's output by a router score, but it is trained from scratch and does not reduce prefill latency. The challenge we address is making a gate in the attention logits work on a pretrained model. Two choices make this work, which we ablate in Appendix D, Figure 7: head-wise gains, initialized to each layer's logit scale and temperature scaling, which corrects the sharper softmax caused by pruning.

(c) Gate learning supervised directly by the task loss, with no auxiliary loss. Token-gated attention gives the scoring network a gradient through the top-K selection, so TokenGate trains with the standard next-token loss alone (Eq. 10). This is in contrast, sparse attention methods, like DSA, which trains its indexer with an auxiliary KL loss to imitate dense attention and prior learned token pruning methods, like LTP and DynamicViT, which add an auxiliary loss term that fix the token budget at training time, requiring re-training per inference-time budget.

(d) Kernels that fit the existing serving stack. Token-gated attention runs in a patched FlashAttention-2 kernel at 1.0–1.1× the cost of standard attention, and the pruned forward is captured as a CUDA graph. During decoding, the gate term is folded into an additive attention mask, so decoding uses standard attention (Appendix A).

AuthorsOpenReview ↗

Response to Reviewer 7GQ2 (Part 2)

One major contribution claimed by the authors, to which they devote considerable space, is their token-gated attention, designed to address the challenge of training non-differentiable top-k operators end-to-end. This problem also exists in sparse attention, and researchers have proposed numerous solutions, including the use of straight-through estimators [1] or, as in DeepSeek Sparse Attention [2], optimizing the KL divergence with the attention distribution. It's unclear what advantages this design offers, especially compared to the numerous existing sparse attention methods. [1] Bengio et al. "Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation." 2013. [2] DeepSeek-AI. "DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models." 2025.

Thank you for these helpful pointers. We have added a new ablation and will expand our Related Work with the following discussion to clarify this important point on token-gated attention and its comparison to existing methods.

(a) The task loss trains the gate directly, with no teacher or auxiliary loss. DSA trains its indexer with a KL loss toward the dense model's attention distribution, so the indexer learns to imitate attention mass. That is the same signal FastKV uses, and our ablation shows it is weak at the early layers where pruning saves the most compute (Figure 5). In TokenGate, the gate score enters the attention logits of every layer above the gate (Eq. 8), so the next-token loss trains the scoring network directly, leading to superior performance.

(b) Token-gated attention outperforms straight-through-estimator (STE) training. We considered STE-based training, and we ran a new ablation comparing it directly with token-gated attention. We trained an STE variant of TokenGate on BioDEX (Qwen3-1.7B, gate at layer 1) that is identical to TokenGate except for the training signal. Its forward pass performs exact top-K token dropping, with no gate term in the attention logits, while its backward pass propagates the gradient to the gate scores through that zeroed term. Both models share the same architecture, so TTFT at each budget is identical. As Table 3 shows, token-gated attention outperforms STE at every budget from 512 kept tokens up, by up to 0.06 F1 (0.331 vs. 0.271 at 2,048 tokens), and also at full context (0.341 vs. 0.307).

Intuitively, an STE's forward and backward passes disagree: the forward pass applies a hard top-K mask, while the backward pass uses a surrogate gradient for a computation that never happened. In token-gated attention, the gate value is part of the forward computation at both training and inference (Eq. 8), so the scoring network receives the exact gradient (almost everywhere) of the computation actually performed.

Table 3: Token-gated attention outperforms STE training (BioDEX, Qwen3-1.7B, gate at layer 1, A100, batch size 1). Both models share the same architecture and training recipe; only the gradient path to the gate differs.

Kept tokensTTFT (ms)STE (F1)TokenGate (F1)
512190.2590.269
1,024270.2650.294
2,048440.2710.331
4,096660.3100.334
8,192810.3060.342
Full context880.3070.341
AuthorsOpenReview ↗

Official Comment by Authors

Thanks once again for your careful review and constructive suggestions, which have helped us strengthen the paper! As the discussion period draws to a close, we would appreciate hearing whether you have any remaining questions or concerns. We would be happy to use the remaining time to address them and provide any further clarification.

Reviewer_URdPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TokenGate, a lightweight, drop-in module that brings end-to-end trainable token pruning to existing pre-trained language models for fast and memory-efficient long-context prefill. This can be placed at an early layer, it scores tokens and forwards only the top-K subset to upper layers, using a proposed token-gated attention mechanism to inject importance scores into upper-layer attention logits so the module can be trained end-to-end with standard next-token prediction loss despite the non-differentiable top-K selection. Jointly trained with LoRA across sampled token budgets, TokenGate generalizes to different pruning ratios with minimal training overhead and substantially reduces prefill latency and peak memory without accuracy degradation on both standard and hybrid-attention architectures.

Critical strengths
  1. The proposed token-gated attention bypasses the non-differentiability of top-K pruning by injecting importance scores into upper-layer attention logits. This allows the model to train and infer using only standard next-token prediction loss.

  2. Combined with LoRA, TokenGate achieves total training times comparable to standard PEFT. Moreover, a single checkpoint seamlessly generalizes across different inference-time token budgets and extrapolates to longer context lengths.

  3. The authors implement a custom kernel and CUDA graph capture to translate token sparsity into real-world efficiency. Across diverse benchmarks, TokenGate outperforms baseline in accuracy, TTFT and peak prefill memory.

Critical weaknesses
  1. In the experiments, a separate TokenGate + LoRA model is trained on each individual dataset. It remains unclear whether a single instruction-tuned TokenGate model trained on a general mixture can transfer zero-shot to unseen tasks.

  2. As shown in the paper's own analysis, the gate tends to retain function words while dropping capitalized names and locations. Although ROUGE-L remains competitive, pruning can induce entity hallucinations when tasks require synthesizing dense details across the entire context.

  3. Empirical validation is restricted to small models, leaving the question whether aggressive pruning maintains accuracy on larger, more capable models.

Questions for authors

n/a

2 comments
AuthorsOpenReview ↗

Response to Reviewer URdP

We sincerely thank the reviewer for their feedback, which we believed helped us strengthen the paper. We are encouraged that the reviewer recognizes TokenGate’s superior performance on accuracy, TTFT and peak prefill memory compared to strong baselines like FastKV and SlimInfer, which is one of our primary contributions. The reviewer’s suggested experiments on evaluating whether TokenGate’s performance generalizes to larger models and across tasks with a single trained model were very insightful and led us to two new findings, which we have added to the paper. Please find responses to each point inline below.

  1. In the experiments, a separate TokenGate + LoRA model is trained on each individual dataset….

We agree this is an important question and we believe training a general TokenGate foundation model that generalizes zero-shot across downstream tasks is an exciting opportunity for future work. While it requires a large-scale training run beyond the scope of the current paper, which focuses on model adaptation in the fine-tuning setting, we provide a new, controlled multi-task experiment, which demonstrates that a single trained TokenGate model can transfer across tasks. We trained a single multi-task TokenGate model on a mixture of two disparate tasks, Biodex and NIAH. The two tasks call for very different pruning patterns (Appendix E, Table 4). As Table 1 shows, the multi-task model closely tracks the single-task models on both tasks, and on both tasks, it continues to outperform FastKV's accuracy–TTFT frontier. This shows that one gate can learn task-appropriate importance across domains without per-task specialization, which supports the feasibility of training a general TokenGate model on broader task mixtures.

We also note that TokenGate already generalizes zero-shot beyond its training distribution across context lengths. A model trained on 16k-token NIAH extrapolates to 32k, 64k, and 128k contexts, with 11.3–13.0× TTFT speedups at no accuracy degradation (Appendix D, Table 3).

  1. As shown in the paper's own analysis, the gate tends to retain function words while dropping capitalized names and locations. Although ROUGE-L remains competitive, pruning can induce entity hallucinations…

Thank you for raising this, and we clarify the following points, which we will also update the manuscript with to make our analysis clearer.

(a) The token budget is chosen at inference time, and our appendix visualizations use a very small token budget. Our appendix failure analysis uses very small budgets for visualization (e.g., 13% of the context for MultiNews) and does not reflect recommended operating points. In practice, TokenGate matches full-context accuracy with 1.4–11.3× faster TTFT, and is 1.2–6.4× faster than the next-best baseline (Section 5.1). Since a single model generalizes across budgets, entity-dense tasks can simply use a larger K without retraining.

(a) Entity retention depends on the task. The gate learns to keep what each task needs and does not systematically bias against retaining names and locations. For summarization (MultiNews), the visualization in Figure 9 shows that the gate retains words distributed across the text, likely re-distributing information in the trained-backbone across tokens for summarization. On the other hand, for NIAH, the answer sits in a few specific spans, and the gate tends to retain localized spans of tokens with entities, names and locations related to the query.

(b) Dropped tokens still propagate information. Layers 0–1 attend over the full context before pruning, so information from pruned tokens is mixed into the kept tokens. During decoding, new tokens still attend to the pruned tokens' KV entries in these pre-gate layers. End-to-end training also lets the model redistribute information: with the same random selection, the TokenGate model outperforms the fine-tuned model at every budget (Figure 5).

  1. Empirical validation is restricted to small models, leaving the question whether aggressive pruning maintains accuracy on larger, more capable models.

We thank the reviewer for this suggestion, and we have added new evaluation training TokenGate on Qwen3-8B, which has led us to a new finding: TokenGate’s performance generalizes to larger models (Table 2). We find the TokenGate model trained with the Qwen3-8B backbone matches the fine-tuned base model’s full-context accuracy at 2.2× lower TTFT. It also matches FastKV's best accuracy at 2.0× lower TTFT. We will add these results to the manuscript.

AuthorsOpenReview ↗

Official Comment by Authors

Thanks once again for your careful review and constructive suggestions, which have helped us strengthen the paper! As the discussion period draws to a close, we would appreciate hearing whether you have any remaining questions or concerns. We would be happy to use the remaining time to address them and provide any further clarification.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely thank all three reviewers for their constructive feedback, which helped us strengthen the paper. We are deeply encouraged that the reviewers find our work to address an important research problem in accelerating LLM prefill (@7GQ2), with a simple, defensible design (@KdDn) that clearly improves over strong prior baselines (@7GQ2, @URdP, @KdDn). We are also glad the reviewers find our paper well written and easy to follow (@7GQ2, @KdDn) and appreciate our method design, including token-gated attention, which bypasses the non-differentiability of top-K pruning and trains with standard next-token prediction loss, and our kernel and CUDA-graph implementation that translates token sparsity into concrete TTFT and memory gains on common hardware (@URdP, @KdDn).

Following the reviewers' suggestions, we provide results from 4 new experiments: (1) training a single multi-task TokenGate model (suggested by URdP, KdDn), (2) scaling TokenGate to a larger model, Qwen3-8B (suggested by URdP, KdDn), (3) benchmarking FastKV, the strongest prior baseline, on the same larger model (suggested by KdDn), and (4) an ablation comparing token-gated attention with straight-through-estimator (STE) training (suggested by 7GQ2). The results are summarized in Tables 1 and 2 below and Table 3 in our response, which we will add to the paper.

We find the multi-task model closely tracks the single-task models. It reaches 95% of full-context accuracy at 3.2× lower TTFT on BioDEX and 10.8× on NIAH, close to the single-task models (3.0× and 10.6×) and well ahead of FastKV (1.6× and 1.9×) (Table 1). This shows that one gate can learn task-appropriate pruning across domains without per-task specialization. On Qwen3-8B, TokenGate's gains hold at scale, matching the fine-tuned full-context accuracy at 2.2× lower TTFT and reaches FastKV's best accuracy at 2.0× lower TTFT on the Biodex dataset (Table 2). Finally, our trained model with token-gated attention outperforms STE training at each inference token budget, by up to 0.06 F1 at matched TTFT (Table 3).

We address specific questions in our individual responses to each reviewer below, including how TokenGate differs from FastKV and sparse attention (7GQ2), how our ablation separates token selection from co-adaptation (KdDn), clarification of qualitative analysis (URdP) and a new Limitations section (KdDn).

We believe these new results and clarifications reinforce our central finding: TokenGate’s end-to-end trainable token pruning lets existing models prefill substantially faster and with less memory at no accuracy degradation, outperforming strong baselines.

Table 1: A single multi-task TokenGate model closely tracks per-task models (Qwen3-1.7B, gate at layer 1, A100, batch size 1). One TokenGate model is trained on a two disparate tasks, BioDEX and NIAH, and evaluated on each task. We report the TTFT (ms) each method needs to reach 95% of full-context accuracy, where full-context accuracy is that of the task's LoRA fine-tuned (SFT) model, as in our paper. Speedups over full-context prefill are in parentheses. FastKV runs on the per-task SFT models.

MethodBioDEX: TTFT to 95% acc.NIAH: TTFT to 95% acc.
SFT model, full context87 (1.0×)381 (1.0×)
FastKV (layer 13)54 (1.6×)202 (1.9×)
TokenGate, single-task29 (3.0×)36 (10.6×)
TokenGate, multi-task27 (3.2×)35 (10.9×)

Table 2: TokenGate scales to Qwen3-8B (BioDEX, gate at layer 1, A100, batch size 1). We report the TTFT (ms) each method needs to reach (a) full-context accuracy and (b) 95% of full-context accuracy, where full-context accuracy is that of the LoRA fine-tuned (SFT) model (0.328 F1). Speedups over full-context prefill are in parentheses. As in our paper, FastKV runs on the SFT model at its default relative depth (46%, layer 17).

Method(a) TTFT to full-context acc.(b) TTFT to 95% acc.
SFT model, full context373 (1.0×)373 (1.0×)
FastKV (layer 17)318 (1.2×)208 (1.8×)
TokenGate173 (2.2×)173 (2.2×)

Table 3: Token-gated attention outperforms STE training (BioDEX, Qwen3-1.7B, gate at layer 1, A100, batch size 1). Both models share the same architecture and training recipe; only the gradient path to the gate differs.

Kept tokensTTFT (ms)STE (F1)TokenGate (F1)
512190.2590.269
1,024270.2650.294
2,048440.2710.331
4,096660.3100.334
8,192810.3060.342
Full context880.3070.341
AuthorsOpenReview ↗

Summary of Discussion: New Experiments, Analysis and Clarifications

We extend our deepest gratitude to the reviewers for their productive feedback and discussion. We are encouraged that the reviewers find our work to address an important problem in accelerating LLM prefill, with a simple, defensible design that clearly improves over strong prior baselines. We were also encouraged that reviewer who engaged during the discussion period confirmed that our responses addressed their concerns, increasing their score and recommending clear acceptance of our paper.

In response to reviewer feedback and over the discussion period, we conducted 6 new experiments: two multi-task training runs, scaling TokenGate and FastKV to Qwen3-8B, a new ablation separating token selection from pruning depth, and an ablation against straight-through-estimator (STE) training. We also updated our paper with additional analyses and clarifications, summarized below.

1. A single multi-task TokenGate model generalizes across tasks. (a) Controlled two-task run. A single TokenGate model trained on a mixture of BioDEX and NIAH reaches 95% of full-context accuracy at 3.2× lower TTFT on BioDEX and 10.9× on NIAH. That is close to the per-task models (3.0× and 10.6×) and well ahead of FastKV (Table 1). (b) Extended multi-task run. We then trained a single TokenGate model on a on all all five datasets of our paper's tasks. TokenGate once again offers Pareto-optimal performance, achieving 95% of full-context accuracy at 1.3–10.9× lower TTFT across the five tasks compared to standard prefill and up to 2× lower TTFT than FastKV.

2. TokenGate's gains hold at larger scale. On Qwen3-8B, TokenGate matches the fine-tuned model's full-context accuracy on BioDEX at 2.2× lower TTFT (173 vs. 373 ms), and reaches FastKV's best accuracy at 2.0× lower TTFT (Table 2).

3. A learned gate enables aggressive early pruning, where attention-based heuristics break down. We ran a new ablation on 2WikiMQA that separates which tokens are selected from where pruning happens. We find that FastKV degrades accuracy at earlier layers due to its attention-based heuristic which performs poorly, whereas our end-to-end learned gate module exceeds.

4. Token-gated attention outperforms STE training. An STE variant of TokenGate, identical in architecture and TTFT, underperforms token-gated attention at every budget from 512 kept tokens up, by up to 0.06 F1. TokenGate also wins at full context (0.341 vs. 0.307) (Table 3). This supports our design: the gate value is part of the forward computation, so the scoring network receives the exact gradient of the computation actually performed.

5. New analysis.

  • Token selection vs. adaptation. Our existing ablation (Section 5.2, Figure 5) shows that the gains come primarily from improved token selection, with a complementary co-adaptation gain. With the same random selection, the TokenGate backbone outperforms the LoRA backbone at every budget.

6. Clarifications and Paper Updates.

  • We clarified TokenGate's differences from FastKV and sparse attention
  • We also clarified our qualitative analysis in the appendix.
  • We added a limitations section

We believe these new results and clarifications strongly reinforce TokenGate's strength in enabling end-to-end trainable token for faster and more memory-efficient prefill, outperforming all strong baselines across tasks, model scales, and multi-task training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.