acceptodds
Under review as a conference paper at ICLR 2027

RouteCache: Efficient Long-Context Attention via Learnable Retrieval and Temporal Caching

Abstract

Sparse attention can reduce long-context inference costs, but learning discrete token selection from the language modeling objective remains challenging. We introduce RouteCache, which combines end-to-end trained hash-based retrieval with an LRU working cache for efficient decoding. Our key insight is that attention-score gradients provide task-driven supervision for retrieval scores, enabling training beyond layer-wise imitation of dense attention. Starting from oracle initialization, we introduce closed-loop gradient feedback: gated hash scores influence attention during training and receive feedback from the resulting language modeling loss. This jointly optimizes hash networks across layers while keeping the pretrained backbone frozen. At inference, the gates are removed and the hash networks select tokens for sparse attention across all layers. To translate sparse retrieval into practical acceleration, the working cache reuses recently retrieved KV pairs, while fused kernels and asynchronous execution reduce data movement and scheduling overhead. Across Qwen3 models from 4B to 30B parameters, including MoE and reasoning variants, RouteCache achieves quality generally close to dense attention on perplexity, LongBench V2, and RULER evaluations. At 256K context, latency benchmarks show up to 4.9–7x speedups over StaticCache, depending on the retrieval and cache budgets.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Sparse Attention, Long-Context LLM, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Re8NOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RouteCache, which trains a binary hash indexer to select the top-k most important tokens and computes attention only over the selected tokens. During inference, it further incorporates LRU caching, kernel fusion, and asynchronous execution to improve efficiency. Experiments on Qwen3 models ranging from 4B to 30B parameters show that RouteCache achieves performance generally close to dense attention in perplexity evaluations and on LongBench V2 and RULER.

Critical strengths
  1. The efficiency analysis experiments in this paper are thorough, showing the execution time of each operator in detail.
  2. This paper is clearly written, easy to understand and follow.
Critical weaknesses

main

  1. The paper’s contributions are not sufficiently clear. Its two main innovations are training a hash indexer for token selection and using LRU caching. The hash-indexer approach is similar to Spotlight, as both train a hash indexer to select tokens. The paper should therefore focus on explaining the limitations of Spotlight and how the proposed method addresses them. LRU has also been used for KV caching in prior work, and its connection to the hash indexer is relatively weak. The novelty appears to take the form of an “A+B” combination. The paper should explain the connections between its contributions in greater detail rather than simply combining different methods.

  2. The paper only compares against other hash-based methods. However, among trainable token-level top-k approaches, DSA from DeepSeek-V3.2 is widely used. One of the paper’s motivations is to move beyond layer-wise imitation of dense attention, yet it does not explain the shortcomings of such approaches. A comparison with these methods would make the evaluation more convincing.

minor

  1. The contribution of LRU to model quality reported in Table 4 is questionable. The LRU configuration uses a budget of 2K, whereas the eager baseline uses 1K. The observed improvement may therefore result from the increased budget rather than from LRU itself.

  2. Although the paper presents detailed experiments on per-layer decoding acceleration, end-to-end acceleration experiments are missing. This raises doubts about the practical effectiveness of the proposed method.

Questions for authors

I will consider raising my rating if the authors can answer the following questions:

  1. Could the authors explain the motivation behind the proposed method and the limitations of previous approaches in greater detail?

  2. Could the authors train a DSA-based model for comparison? To avoid gains from backbone training, it would be sufficient to perform only Stage 1 (S1), keeping the backbone parameters frozen.

  3. Could the authors add an Eager Top-2K baseline to the experiments in Table 4?

  4. Could the authors provide experimental results on end-to-end speedups?

Reviewer_xwJAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper claims that the current "sparse attention methods retain selected dense layers to help preserve accuracy". To improve the performance of sparse attention under long contexts, RouteCache is proposed. RouteCache combines learned token retrieval with an LRU cache, which fetches only missed cache entries from HBM to reduce data transfer frequency while retaining reasonable token selection performance. Experiments on the Qwen3 model series span a reasonable model size, demonstrating a 4.9-7x speedup over StaticCache baselines.

Critical strengths
  • The experiment span is acceptable. The proposed method has been tested on more than 2 benchmarks, including LongBench and RULER; a relatively rationale range of model size, including one MoE model.
  • The presentation of the paper is clear and organized, which makes it easy for readers to follow and understand.
  • Experiments demonstrate that the performance of the proposed method has increased efficiency with controllable performance degradation.
Critical weaknesses
  • The motivation is not clear enough. The paper claims that learning discrete token selection remains challenging, but no evidence is provided to demonstrate the flaws of the current methods in this direction. Insatisfactory performance is not a solid reason to be a foundation of the method proposing.
  • Only one model family has been evaluated, hindering the generalization of the method.
Questions for authors
  • Do you only use AI tools for paper editing, as claimed in the statement section, actually? No coding or implementation assistance? No other specific questions.
Reviewer_mDaBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a sparse attention method to reduce HBM traffics. First, hash functions (in MLPs) are trained for query and key vectors such that the hash functions align with the attention scores of query/key pairs. Query/key pairs with the higher attention score tend to have similar hash output (binary vector). To do that, two-stage training is proposed. In stage 1, the hash functions (in MLPs) are initialized by using top-K keys and query. In stage 2, the hash alignment is adjusted to account for their impact on the final model output. In test time, the hash vectors are managed for key vectors. Given a query, hash scores (Hamming distance of q/k hash vectors) are computed with a very small amount of HBM traffics (since hash vector size << original key vector size). Then, the LRU working cache (containing recently used key/value vectors under LRU) is looked up for high hash score keys. Only the key (and associated value) vectors with cache miss are fetched from HBM. Reading the hash vectors and the missed key/value vectors from HBM reduces memory traffics thereby reducing latency. Then, attention is performed on the key/value vectors in the cache, which reduces compute cost w.r.t. full attention.

Critical strengths

The idea of aligning hash functions with the contribution of attention and jointly refining hash-based retrieval across layers using the language-modeling objective looks novel.

Other implementation details like fusion of query hashing, binarization, bit-packing, and Hamming-distance computation, overlap of retrieval and working-cache updates with K/V projections using a separate CUDA stream, asynchronous writes of new KV pairs and key hashes, etc., look quite practical.

Critical weaknesses

End-to-end latency comparison is needed. The paper reports per-layer latency comparison. In order to evaluate the utility, it would be better to report the latency of entire model while varying sequence lengths (including very long ones to characterize the effectiveness and scaling behavior) and batch sizes.

Stage 2 performs dense attention with hash scores, whereas inference applies sparse attention. It is interesting to investigate and, if needed, reduce the training-inference gap. For a possible improvement, it would be useful to apply an iterative design where, after obtaining the sparse attention solution (i.e., trained hash functions and LRU working cache mechanism), the hash functions could be further fine-tuned under the obtained sparse attention.

It would be interesting to compare attention patterns between the original full attention and the proposed sparse attention. Especially, task-wise analyses would be useful, e.g., long context tasks (like needle in a haystack) vs. math tasks.

Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.