- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Hierarchical Hash Retrieval (HHR), a learned sparse-attention method for efficient long-context generation. HHR uses Geometry-Aware Key Routing (GKR) for page-level pruning and Learned Hash Projection (LHP) for fine-grained hash retrieval. The method is evaluated on LongBench, RULER, and LongBench-v2, with additional ablation and efficiency experiments.
Critical strengths
1. Well-motivated method design. HHR provides a clear coarse-to-fine framework combining page-level pruning and fine-grained hash retrieval.
2. Interesting analysis of hash-retrieval errors. The paper identifies the mismatch between Hamming distance and QK relevance caused by discarded magnitude information, which provides a reasonable motivation for the proposed method.
Critical weaknesses
1. The empirical evaluation is incomplete and does not sufficiently validate HHR across the intended task settings. The RULER and LongBench evaluations use incomplete benchmark subsets, while most ablation studies are conducted only on a few selected subtasks. In addition, the evaluation is dominated by long-input/short-output workloads and does not include long-generation reasoning tasks such as AIME or MATH-500, which are particularly relevant for decoding-time sparse attention. Together, these limitations make it difficult to assess whether HHR generalizes reliably across tasks and generation regimes.
2. The efficiency and baseline comparisons are not strong enough to substantiate the claimed practical advantage. The paper reports speedups over an unspecified “Dense” baseline without clarifying whether it uses eager attention, SDPA, FlashAttention, etc., and provides little information about the implementation of HHR, batch size, component-wise latency, memory overhead, or batched decoding. Moreover, HHR is a learned sparse-attention method, but it is not compared against recent training-based baselines such as SeerAttention-R [1]. These omissions make it difficult to determine whether HHR offers a clear practical advantage over strong contemporary alternatives.
[1] Gao Y, Guo S, Cao S, et al. Seerattention-r: Sparse attention adaptation for long reasoning[J]. arXiv preprint arXiv:2506.08889, 2025.
Questions for authors
1. Efficiency setup. What implementation is used for the “Dense” baseline in Fig. 7, and how is HHR implemented? Please also clarify the batch size and whether batched decoding is supported.
2. Efficiency breakdown. Can the authors report the latency breakdown of GKR, LHP/hash retrieval, Top-K selection, and sparse attention, as well as the additional memory overhead?
3. Benchmark coverage. Why are only 11 RULER tasks and 16 LongBench datasets reported? Can the authors provide results on the full standard benchmark suites?
4. Ablation coverage. Can the main ablations be reported on the full benchmark, or at least on a fixed representative set of tasks rather than different selected subtasks?
5. Long-generation evaluation. Can the authors evaluate HHR on long-generation reasoning tasks such as AIME or MATH-500?
6. Training-based baselines. Why are recent learned sparse-attention methods such as SeerAttention-R not included in the comparison?
6 comments
Response for W2
W2. The efficiency and baseline comparisons are not strong enough to substantiate the claimed practical advantage.
A2. We thank the reviewer for the feedback. We clarify the Dense baseline implementation, provide additional single-layer and whole-model timing comparisons with HATA, and include SeerAttention-R as a learned sparse-attention baseline on six LongBench datasets.
1. Dense baseline and implementation. The Dense baseline in the original manuscript uses Eager Attention. We select HATA as a representative learned hash-based retrieval method and use its official kernels. We benchmark Dense, HATA, and HHR in vLLM under the same hardware and request configurations.
2. Single-layer and whole-model efficiency. We benchmark single-layer decoding with Llama-2-7B on an RTX6000D GPU using a 32K-token input, a batch size of 8, and FP16 precision. The sparse methods use 128-bit hashes and a token budget of approximately 1.5%.
| Method | Decode latency (ms/step) ↓ | Speedup over Dense ↑ |
|---|---|---|
| Dense | 3.7572 | 1.000× |
| HATA | 1.1527 | 3.259× |
| HHR | 0.9059 | 4.147× |
HHR supports batched decoding when requests share the same configuration and achieves a 1.272× single-layer decoding speedup over HATA.
For the whole-model evaluation, we use Llama-2-7B-32K-Instruct on an RTX6000D GPU with FP16 precision, a batch size of 1, and a page size of 128. We run one warm-up request and one timed request per configuration.
| Input / Output length | Method | Time to first decode (s) ↓ | Decode time (s) ↓ | End-to-end time (s) ↓ | Decode speedup ↑ | End-to-end speedup ↑ |
|---|---|---|---|---|---|---|
| 36K / 3.6K | Dense | 8.153 | 99.821 | 107.974 | 1.000× | 1.000× |
| HATA | 8.185 | 59.881 | 68.067 | 1.667× | 1.586× | |
| HHR | 8.182 | 58.077 | 66.259 | 1.719× | 1.630× | |
| 72K / 7.2K | Dense | 26.032 | 309.875 | 335.907 | 1.000× | 1.000× |
| HATA | 26.078 | 139.874 | 165.952 | 2.215× | 2.024× | |
| HHR | 26.068 | 137.066 | 163.134 | 2.261× | 2.059× |
In these two single-request measurements, HHR achieves end-to-end speedups of 1.630× and 2.059× over Dense, comparable to HATA's 1.586× and 2.024×, respectively. 3. Comparison with SeerAttention-R. We evaluate a decoding adaptation of SeerAttention-R on six representative LongBench datasets. HHR and Oracle Top-K both use a 1.5% retrieval budget.
| Method | NarrativeQA ↑ | 2WikiMQA ↑ | QMSum ↑ | TREC ↑ | PassageRetrieval-en ↑ | LCC ↑ |
|---|---|---|---|---|---|---|
| Dense | 30.01 | 46.70 | 25.30 | 72.50 | 99.50 | 52.48 |
| Oracle Top-K | 29.71 | 46.20 | 25.52 | 72.50 | 99.50 | 51.91 |
| SeerAttention-R | 28.11 | 46.08 | 23.78 | 70.00 | 99.25 | 49.76 |
| HHR | 30.83 | 47.34 | 24.34 | 73.00 | 99.00 | 50.49 |
HHR outperforms SeerAttention-R on five of the six datasets and achieves the highest scores among the evaluated methods on NarrativeQA, 2WikiMQA, and TREC. Response A1 also reports long-generation reasoning results on MATH-500 and AIME.
Response for Q1-Q6
Thank you for these detailed questions. We will incorporate the implementation details and supplementary results below into the revised manuscript.
Q1. Efficiency setup.
A1. The Dense baseline in Fig. 7 uses eager attention. HHR performs the same attention computation over the KV entries selected by its two-stage retrieval pipeline. Both retrieval stages use custom CUDA operators to accelerate retrieval and reduce intermediate buffer allocations. The results in Fig. 7 were obtained with a batch size of 1. HHR also supports homogeneous batched decoding with a batch size greater than 1.
Q2. Efficiency breakdown.
A2. We thank the reviewer for this suggestion. We provide additional whole-model and single-layer benchmarks to quantify HHR’s decoding latency and GPU memory usage relative to HATA.
Whole-model evaluation. In our comparison, HATA uses its official CUDA kernels, and HHR uses custom CUDA kernels that we implemented. We benchmark Dense, HATA, and HHR with Llama-2-7B-32K-Instruct on the same RTX 6000 GPU in FP16, using a batch size of 1 and matched input and output lengths. All speedups are relative to Dense.
| Method | Prefill time (s) ↓ | Decode time (s) ↓ | End-to-end time (s) ↓ | Decode speedup ↑ | End-to-end speedup ↑ | GPU memory (GiB) ↓ |
|---|---|---|---|---|---|---|
| Dense | 8.153 | 99.821 | 107.974 | 1.000× | 1.000× | 50.81 |
| HATA | 8.185 | 54.881 | 63.067 | 1.819× | 1.712× | 53.14 |
| HHR | 8.182 | 53.077 | 61.259 | 1.881× | 1.763× | 53.38 |
Single-layer evaluation. We benchmark single-layer decoding with Llama-2-7B on an RTX 6000 GPU in FP16, using a 32K-token input and a batch size of 8. HHR and HATA both use 128-bit hashes and a token budget of approximately 1.5%.
| Method | Decode latency (ms/step) ↓ | Speedup over Dense ↑ | Peak GPU memory (GiB) ↓ |
|---|---|---|---|
| Dense | 3.7572 | 1.000× | 4.88 |
| HATA | 1.1527 | 3.259× | 5.08 |
| HHR | 0.9059 | 4.147× | 5.11 |
With our custom CUDA kernels, HHR achieves lower decoding latency than HATA in both benchmarks with comparable GPU memory usage. HHR uses only 0.24 GiB more GPU memory in the whole-model benchmark and 0.03 GiB more peak GPU memory in the single-layer benchmark. HHR also achieves higher accuracy than HATA in the comparisons reported in Tables 1 and 2.
Q3. Benchmark coverage.
A3. We evaluate 11 representative RULER subtasks. HATA also reports results on a subset of RULER. Our LongBench evaluation covers 16 English datasets across six task categories, including single-document question answering, multi-document question answering, summarization, few-shot learning, synthetic tasks, and code completion.
Q4. Ablation coverage.
A4. We conduct the main supplementary ablations on two representative datasets, 2WikiMQA and MultiFieldQA-en, and use the same samples across configurations. See W1 (A1) for the experimental settings and results.
Q5. Long-generation evaluation.
A5. We added long-generation reasoning evaluations on MATH-500, AIME 2024, and AIME 2025. See W2 (A2) for the evaluation settings and results.
Q6. Training-based baselines.
A6. Our original comparison of learned sparse-attention methods focused on trainable hashing approaches such as HATA. SeerAttention-R uses learned block-level gating. We have extended the evaluation to include a direct comparison with SeerAttention-R. See W2 (A2) for the results.
Response for W1
W1. The empirical evaluation is incomplete and does not sufficiently validate HHR across the intended task settings.
A1. Thank you for the constructive feedback. Below, we clarify the benchmark coverage and report additional experiments on the contributions of GKR and LHP, performance across model sizes, sensitivity to the token budget, and long-generation reasoning.
Evaluation coverage. We evaluate 11 representative subtasks from RULER. HATA likewise reports results on a subset of RULER subtasks. On LongBench, as stated in the manuscript, we evaluate 16 English datasets spanning six categories: single-document QA, multi-document QA, summarization, few-shot learning, synthetic tasks, and code completion. We will explicitly list the evaluated RULER subtasks and LongBench datasets, along with their task categories, in the revised manuscript.
Additional experiments. We use two representative datasets, 2WikiMQA and MultiFieldQA-en, to evaluate the contributions of GKR and LHP, compare HHR with full attention across model sizes, and measure sensitivity to the token retention budget.
1. Contributions of GKR and LHP
We replace the learned transformation matrix R in GKR or the learned projection matrix W in LHP with a random matrix while retaining the learned parameters of the other stage. We also evaluate GKR-only and LHP-only variants to assess the contribution of combining the two stages.
Randomizing R or W reduces the mean score by 1.86 and 0.70 points, respectively. The ranking of the single-stage variants changes across datasets. Full HHR achieves the highest score on both datasets and improves the mean score by 2.35 points over GKR only and 1.16 points over LHP only. These results support combining page-level routing with token-level retrieval.
2. Performance Across Model Sizes
We compare HHR with full attention (Dense) using Qwen3-4B, Qwen3-8B, and Qwen3-14B.
Across all three model sizes, HHR achieves an average score over the two datasets comparable to that of full attention.
3. Sensitivity to the Top-K Retention Ratio
We vary the final token retention ratio across 1%, 2%, 5%, 10%, and 20% and compare HHR with StreamingLLM, SnapKV, Loki, and HATA at each budget.
2WikiMQA
MultiFieldQA-en
HHR achieves the highest score in 9 of the 10 dataset–budget combinations. At both the 1% and 2% retention ratios, it outperforms all four baselines on both datasets.
4. Long-Generation Reasoning
Our original long-context experiments primarily cover long inputs with short outputs. To examine sparse attention during long reasoning trajectories, we use Qwen3-4B on MATH-500, AIME 2024, and AIME 2025. These benchmarks typically have short prompts, but the model can generate substantially longer intermediate reasoning sequences. For this evaluation, we use a fixed budget of 2K KV tokens during decoding, following the budget setting used by SeerAttention-R.
Results. HHR and SeerAttention-R achieve similar scores on MATH-500 (92.25% vs. 92.20%). With the same 2K decoding-time attention budget, HHR scores 6.98 and 4.48 percentage points higher than SeerAttention-R on AIME 2024 and AIME 2025, respectively. The largest gap between HHR and full attention is 16.20 percentage points on AIME 2025.