- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper propose a learned lora-style approach to prefill token pruning. The main idea is to train an MLP that attempts to score the important of each token in prefill, which is then used to prune tokens. To compensate the downstream affects of the pruned tokens, the method adds adapters to recover from the distribution shift the pruning introduces. The paper trains unique adapters for task-specific domains and focuses on qwen3-1.7b as the main model target. Compared to prior strong (but training free) baselines (e.g, FastKV) the methods shows improvement across a pareto frontier.
Critical strengths
The paper lays out the problem space and prior art quite well. Figure 1 makes the method immediately clear to anyone with knowledge of prior work. The idea itself is simple (making it easier to defend) and clearly improves over prior strong baselines. The authors provided TTFT speedup on common hardware setups (A100 at batchsize 1) making the claimed improvements seem concrete.
Critical weaknesses
The paper trains separate adapters for each of 5 specific evaluated tasks. While this can make sense in some settings (e.g., where we know only one type of document will be routed to the serving engine), it does decrease the usefulness of the approach. It is unclear why this method could not be trained in a mixed-task setting in-order to see how well it generalizes across domains.
While the authors acknowledge the method requires training relative to prior methods, it still needs to be mentioned as a weakness. Since prior methods (e.g., FastKV) are training free, it arguably change the category of the method. This is a bit related to the prior point, because the training component in a specific domains sort of stacks these effects.
Finally, while the analysis is well done, it is on mostly on a single small model (Qwen3 1.7B). It is unclear how large of a gap would exist at 10x or 100x this scale between TokenGate and prior art.
Questions for authors
Can you report any experiments you did for a mixed-task training regime? Reporting a pareto curve of a single model across all domains compared relative to FastKV would make it more clear how much the approach could work in a general setting like FastKV.
Similarly, could you provide any indication that this method could be extended to say a ~10B parameter model like the Llama 3 8B model FastKV was evaluated on? I believe the training would not be too difficult on the A100 setup, but a full re-evaluation of all tasks would take a lot more time.
How does TokenGate's token selection differ from FastKV? Another framing is what is the % overlap in selected tokens? If the overlap is high, it implies to me the benefit would come mainly from the adapter layers vs improved token selection.
4 comments
Thank you for the new experiments and the Limitations section. I will keep my score.
The new multi-task results are encouraging but only cover 2 of the 5 tasks. Since a single model on all 5 tasks probably takes ~12 hours based on the training times in your paper, I suggest including it in the final version.
In your new table 1, it is surprising that the multi-task TTFT is actually lower than the single task ones. Also, the speedups seem wrongly computed (87/29 is 3.0 not 3.3). This would be worth double-checking.
Also, it seems like on a larger model the gap between TokenGate and FastKV shrinks substantially (from 1.9x to ~1.2x). This was my original concern.
Finally, I am going to push back a bit on your selector ablation for FastKV. No one would run FastKV at layer 1, so of course the tokens selected at that layer is weak. We could take FastKV's tokens selected at layer 13 and the replay them pruning at layer 1, which would separate this question a bit. But, still I understand why you felt to frame it this way.
Response to Reviewr KdDN
We sincerely thank the reviewer for their response and helpful feedback on our updates. We provide clarifications below:
In your new table 1, it is surprising that the multi-task TTFT is actually lower than the single task ones. Also, the speedups seem wrongly computed (87/29 is 3.0 not 3.3). This would be worth double-checking.
Thank you for this point and we sincerely apologize for two typos in the table, which have been resolved. The table now shows the corrected speedups, and the experiment demonstrate that the multi-task TTFT speedup is comparable or higher to the TTFT speedup of the single-task TokenGate models compared to the full-context baseline. We are also currently training a multi-task.
Also, it seems like on a larger model the gap between TokenGate and FastKV shrinks substantially (from 1.9x to ~1.2x). This was my original concern.
Thank you for raising this, and to clarify, our experiments, summarized in Table 2, demonstrate that TokenGate performance is 1.84x higher TTFT at full-context accuracy compared to FastKV, maintaining strong pareto-optimal performance across token budgets, with a significant margin in the high-accuracy regime.
The new multi-task results are encouraging but only cover 2 of the 5 tasks. Since a single model on all 5 tasks probably takes ~12 hours based on the training times in your paper, I suggest including it in the final version.
Thank you for this feedback and we agree. To clarify, we are currently running the TokenGate model and base model fine-tuned on all tasks, which we will plan to add to the paper. We note that for both convergence across tasks takes significantly longer (several days on our available machines) compared to the single task setup, which is why we have first provided a smaller scale, controlled multi-task experiment. We will update our response here and the manuscript with the new multi-task experiments once they have completed as well.
Finally, I am going to push back a bit on your selector ablation for FastKV. No one would run FastKV at layer 1, so of course the tokens selected at that layer is weak. We could take FastKV's tokens selected at layer 13 and the replay them pruning at layer 1, which would separate this question a bit. But, still I understand why you felt to frame it this way.
Thank you for this feedback, and we are happy to provide a new ablation. Our original setup aimed to answer the question, How does TokenGate's token selection differ from FastKV? We found that TokenGate enables more aggressive pruning at earlier layers due to its learnable gate module as opposed to FastKV's attention-based heuristic. We agree that the framing of taking FastKV's tokens at layer 13 and applying them at layer 1 would help develop our analysis further. We will add this and share additional results here.
Thanks for you reply. I will raise my score to 4. Overall, I find the work solid and interesting.
Response to Reviewer KdDn
We thank the reviewer for their valuable, detailed review, which we believe helped us to strengthen the paper. We are thrilled that the reviewer finds the paper to be clearly written, and highlights that our method is simple and defensible, providing superior performance over strong baselines, which was one of our primary contributions. Following the reviewer's suggestions, we ran two new experiments, which led us to two new findings: TokenGate’s performance extends to larger models e.g., Qwen3-8b, and exhibits strong multi-task performance. We also clarify how our ablations separate the contribution of token selection from that of the adapter. We respond to each point inline below.
We agree this is an important question and we believe training a general TokenGate model that generalizes across downstream tasks is an exciting opportunity for future work. To study this, we provide a new, controlled multi-task experiment, which demonstrates that a single trained TokenGate model can transfer across tasks. We trained a single multi-task TokenGate model on a mixture of two disparate tasks, Biodex and NIAH. The two tasks call for very different pruning patterns (Appendix E, Table 4). As Table 1 shows, the multi-task model closely tracks the single-task models on both tasks, and on both tasks, it continues to outperform FastKV's accuracy–TTFT frontier. This shows that one gate can learn task-appropriate importance across domains without per-task specialization, which supports the feasibility of training a general TokenGate model on broader task mixtures.
Limitations. Unlike training-free token pruning methods such as FastKV, TokenGate requires a training stage to learn the scoring network jointly with a LoRA adapter. Its cost is comparable to standard LoRA fine-tuning, and a single checkpoint serves all inference-time token budgets and longer contexts. Our experiments train on individual tasks or small task mixtures. Training TokenGate on a broad instruction-tuning mixture and evaluating zero-shot transfer to unseen tasks is an important direction for future work.
This is a great question, and our ablation in Section 5.2 (Figure 5, 2WikiMultihopQA) separates the two effects. FastKV ranks tokens by the attention they receive from the last few queries. At layer 1, this attention-based heuristic carries little signal, so on the same LoRA backbone FastKV's selector performs close to random-K selection at every budget. In contrast, TokenGate's learned scorer substantially outperforms both. For example, at 2k kept tokens, TokenGate reaches 0.70 F1, compared to 0.37 for FastKV and 0.29 for random-K. Holding the selector fixed, we also observe a co-adaptation effect: with the same random-K selection, the TokenGate backbone outperforms the LoRA backbone at every budget (0.37 vs. 0.29 F1 at 2k tokens). The gains therefore come primarily from improved token selection, with a complementary gain from co-adaptation, and not from the adapter alone. We note that all baselines, including FastKV and token-free pruning methods, also run on a backbone fine-tuned with the same LoRA configuration (Section 5).