- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RouteCache, which trains a binary hash indexer to select the top-k most important tokens and computes attention only over the selected tokens. During inference, it further incorporates LRU caching, kernel fusion, and asynchronous execution to improve efficiency. Experiments on Qwen3 models ranging from 4B to 30B parameters show that RouteCache achieves performance generally close to dense attention in perplexity evaluations and on LongBench V2 and RULER.
Critical strengths
- The efficiency analysis experiments in this paper are thorough, showing the execution time of each operator in detail.
- This paper is clearly written, easy to understand and follow.
Critical weaknesses
main
-
The paper’s contributions are not sufficiently clear. Its two main innovations are training a hash indexer for token selection and using LRU caching. The hash-indexer approach is similar to Spotlight, as both train a hash indexer to select tokens. The paper should therefore focus on explaining the limitations of Spotlight and how the proposed method addresses them. LRU has also been used for KV caching in prior work, and its connection to the hash indexer is relatively weak. The novelty appears to take the form of an “A+B” combination. The paper should explain the connections between its contributions in greater detail rather than simply combining different methods.
-
The paper only compares against other hash-based methods. However, among trainable token-level top-k approaches, DSA from DeepSeek-V3.2 is widely used. One of the paper’s motivations is to move beyond layer-wise imitation of dense attention, yet it does not explain the shortcomings of such approaches. A comparison with these methods would make the evaluation more convincing.
minor
-
The contribution of LRU to model quality reported in Table 4 is questionable. The LRU configuration uses a budget of 2K, whereas the eager baseline uses 1K. The observed improvement may therefore result from the increased budget rather than from LRU itself.
-
Although the paper presents detailed experiments on per-layer decoding acceleration, end-to-end acceleration experiments are missing. This raises doubts about the practical effectiveness of the proposed method.
Questions for authors
I will consider raising my rating if the authors can answer the following questions:
-
Could the authors explain the motivation behind the proposed method and the limitations of previous approaches in greater detail?
-
Could the authors train a DSA-based model for comparison? To avoid gains from backbone training, it would be sufficient to perform only Stage 1 (S1), keeping the backbone parameters frozen.
-
Could the authors add an Eager Top-2K baseline to the experiments in Table 4?
-
Could the authors provide experimental results on end-to-end speedups?