acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Sparse Attention Done Right

Abstract

Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention. Chunk-wise sparse attention offers a promising alternative, but all existing methods fall short of full attention because of their inaccurate chunk selection. We propose Hierarchical Landmark Sparse (HiLS) Attention, a chunk-wise sparse attention mechanism that learns chunk selection end-to-end under the language-modeling (LM) loss. HiLS factorizes attention hierarchically: each query performs attention independently with each retrieved chunk to extract chunk-specific information, and the resulting outputs are fused according to chunk retrieval scores. By incorporating retrieval scores into the forward attention computation, HiLS optimizes them directly with the LM loss, enabling end-to-end retrieval learning and native sparse training. Experimental results in Fig fig:main_results show that HiLS-Attention achieves performance comparable to, and in some cases better than, full attention at in-domain context lengths. Meanwhile, HiLS-Attention extrapolates more than the training context length with 90% retrieval accuracy, far beyond full attention. Moreover, existing full-attention models can be converted to HiLS-Attention with lightweight continued pretraining, preserving in-domain performance while acquiring ultra-long-context extrapolation. Together with its sparse KV access and computation, HiLS-Attention breaks the usual efficiency-performance trade-off, enabling long-context LLMs that are both more efficient and more effective on general long-context tasks than their full-attention counterparts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sparse attention, long-context modeling, language modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_L7gFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Hierarchical Landmark Sparse (HiLS) Attention, which introduces a new perspective on block scoring: approximate the full-attention-induced block mass using a learned, attention-weighted key summary and an entropy bias derived from a first-order LogSumExp expansion. Retrieved blocks are attended to independently, and their outputs are fused using the same routing scores. This lets the language-modeling loss directly train the scores of selected blocks. The implementation additionally uses low-rank query calibration (Q-Cal), HoPE positional encoding, and kernels that reuse retrieved KV blocks across adjacent queries.

The paper evaluates 345M models trained from scratch and a 7B model adapted through continued pretraining. The reported results show competitive in-domain perplexity and strong length extrapolation, including 96% single-needle accuracy at 4M tokens after 8K-context training. The 7B results also provide evidence of practical adaptation and improvements on LongBench. These results involve RULER-style synthetic training data, and the strongest 7B results use approximately 50B tokens of full-parameter continued pretraining.

Critical strengths
  1. Connecting block relevance to LogSumExp attention mass motivates an attention-weighted summary with entropy calibration. Using the routing scores in output fusion provides a direct learning signal for selected blocks. This is a meaningful contribution beyond simply changing the retrieval heuristic, while building on the hierarchical gating framework of Landmark Attention.

  2. The small-model experiments combine comparable in-domain perplexity with strong retrieval extrapolation, and the ablations demonstrate the importance of learned summaries, Q-Cal, and positional encoding.

Critical weaknesses
  1. Figure 4 measures 345M inference at batch size 1 on H800 against a shared Triton implementation. Appendix H.1 explains this choice, but it does not establish the advantage over an optimized Hopper dense-attention backend. There is no 7B training throughput/MFU comparison, no disclosure of decode split-KV or GQA-packing settings, and no memory or concurrency evaluation. This matters because practical training and long-context inference efficiency are central claims. Sparse KV access alone does not establish reduced GPU-resident KV storage. Moreover, the O(1) decoding description in Appendix H.1 omits the history-dependent summary scan acknowledged in Section 3.

  2. The evaluation protocol and statistical reporting are insufficiently specified, and the reported RULER tasks provide limited coverage of aggregation and semantic QA. GPQA is included as a 5-shot benchmark, but a long-output CoT evaluation is not established by the disclosed protocol. Equal synthetic-data exposure makes the internal comparison procedurally fair; however, it does not isolate improved associative retrieval from an advantage in suppressing distractors. Finally, the ablations show that the complete method depends strongly on HoPE and Q-Cal, so its advantage should not be attributed entirely to the Taylor-derived score. Average downstream performance also masks decreases on some tasks, including MMLU, HellaSwag, and RACE relative to the matched CPT baseline.

  3. The evidence covers 345M pretraining and continued pretraining of a single 7B hybrid-attention model; it does not establish scaling behavior or readiness for distributed long-context training and deployment. Retrieval accuracy at an extrapolated length is not a validated general-purpose context window: historical KV storage, routing costs, and serving constraints still matter. An explicit discussion and narrower claims would address this concern without additional scaling experiments or diminishing the block-scoring contribution.

Questions for authors
  1. Please report steady-state training step time, tokens/s/GPU, MFU, and peak memory for HiLS and the matched OLMo3-512swa-CPT baseline. Use the same hardware, precision, batch sizes, parallelism, and activation-recomputation settings, including routing, backward, communication, and optimizer costs. After sufficient performance warmup, ten stable optimizer steps would be a reasonable initial measurement window; extend it if timing remains variable. Please specify the FLOPs accounting, since applying the dense-attention formula to HiLS can misrepresent MFU, and lower MFU alone does not imply lower throughput. Megatron-LM is a recommended implementation.

  2. Can you supplement the shared-Triton comparison with FA3 or an equivalently optimized Hopper backend? The supplied implementation appears to use a default Triton path without the relevant Hopper-specific optimizations; please clarify its actual TMA, asynchronous matrix-operation, and warp-specialization usage. For Figure 4's batch=1, Q-heads=16, KV-heads=2 setting, report the dense and HiLS launch configurations, split-KV settings, and appropriate GQA-packing choices. Please provide profiling evidence for SM occupancy, and end-to-end latency. FA3 split-KV and pack-GQA should be tuned, with comparable tuning effort for HiLS. When batch-level parallelism is insufficient to saturate the GPU, shorter selected sequences do not necessarily reduce latency; routing, top-K selection, and other preprocessing overheads may even make sparse attention slower. This is why a measured comparison against a properly tuned dense baseline is necessary.

  3. Is the historical KV cache fully GPU-resident during the reported measurements, or is offloading actually used? Please report persistent and peak memory, distinguishing original KV storage, summary keys, landmark-related storage, indices, and temporary workspaces. A modest concurrency sweep should show both performance at equal concurrency and achievable throughput under an equal GPU-memory budget, preferably with a common latency target. Please also reconcile the O(1)O(1) decode claim with the O(N/S)O(N/S) per-query routing scan and include routing and top-K costs in the measurements.

  4. Please specify toolkit, task configurations, prompts and few-shot selection, answer extraction, sample counts, greedy versus sampling, temperature, decoding top-k/top-p where applicable, generation limits, stopping criteria, input truncation, and landmark-aware length accounting. Which RULER subtasks and parameters are aggregated in Table 3? For synthetic evaluation, report multiple dataset-generation seeds with paired samples across methods and mean/std; deterministic greedy decoding does not remove randomness in synthetic data construction. Standardized tooling such as lm-eval-harness for task evaluation is recommended. DO NOT use custom scripts for evaluation.

  5. Can you provide representative results on aggregation tasks such as RULER CWE/FWE and semantic QA such as RULER QA-SQuAD/QA-Hotpot? To support the broader claim of replacing full attention while preserving capability, please also provide one meaningful long-output reasoning evaluation, such as gpqa_diamond_cot_n_shot, and aime, on which the baseline has nontrivial performance. The existing GPQA 5-shot row does not establish that long generation was evaluated. Use matched prompts and max generation tokens, and report accuracy and output lengths.

  6. Appendix F indicates equal 5% RULER-style augmentation for the relevant training comparisons; I do not interpret this alone as unfairness. Can you clarify how much the retrieval conclusions depend on task-specific augmentation and distractor suppression? I recommend a 7B control in which both HiLS and the matched CPT baseline are trained without RULER-style augmentation, holding the remaining training settings and token budgets fixed. Construct synthetic test sets with five independent dataset-generation seeds, use paired samples across methods within each seed, and report the mean score and standard deviation across the five sets. This would help separate augmentation-dependent gains from generalization to unseen retrieval instances; it would not by itself establish whether distractor suppression explains the extrapolation results. A targeted control, such as isolated MQAR with random-padding and no-padding variants, could help. In Appendix A.1, the prose describes a maximum over normalized head scores, whereas Eq. (12) uses raw logits; which is implemented?

  7. discussion. Could you distinguish the demonstrated block-scoring and retrieval-extrapolation results from the broader prospect of using HiLS as a core building block for ultra-long-context foundation models? Please discuss the untested scaling and distributed-training assumptions, including context-parallelism support, and distinguish empirical retrieval extrapolation from a generally usable, deployable context window. Sparse KV access does not automatically reduce resident KV capacity; if full historical KV is retained, summaries and other auxiliary state add storage overhead. If the evaluation procedures and optimized baseline comparisons requested above do not sustain the reported quality or efficiency gains, please further narrow the claims about HiLS as a core building block for ultra-long-context foundation models to match the evidence that remains supported. I do not request scaling law, CP implementation, or evaluation of reasoning-trajectory drift in long-horizon optimization tasks for this point.

Reviewer_dNHdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Hierarchical Landmark Sparse Attention (HiLS), which uses learned landmark summaries to select context chunks. Attention is computed within each selected chunk, and chunk outputs are combined using the retrieval scores, allowing these landmarks to receive backpropagated gradient from the language modeling loss. The method incorporates positional encoding via HoPE and a low-rank query calibration module. Experiments cover training 345M models from scratch and converting a 7B pretrained model through continued pretraining.

Critical strengths
  1. Incorporating chunk retrieval scores into output aggregation provides a useful way to train chunk summaries directly under the language modeling loss.
  2. The experiments cover perplexity, retrieval, downstream tasks, and end-to-end inference latency at different context lengths.
  3. The paper includes ablations on landmark summaries, query calibration, and positional encoding.
Critical weaknesses
  1. The extrapolation comparisons are not very convincing. HiLS with RoPE fails at 16K, whereas HiLS with HoPE remains effective at much longer lengths. Although a full-attention HoPE baseline is included, comparable HoPE variants of the sparse baselines are missing.
  2. The RULER evaluation covers only three task types. Broader evaluations are needed to support the long-context claims.
  3. The efficiency evaluation in Appendix compares against a dense baseline using the same Triton infrastructure, rather than a widely used optimized attention implementation.
Questions for authors

see weakness

Reviewer_8zEXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Hierarchical Landmark Sparse Attention (HiLS), which represents each historical chunk using a landmark-derived weighted key and an entropy term. These summaries approximate chunk log-masses, enabling selection of a limited number of chunks. The selected chunk scores also enter the hierarchical attention weights, giving the summaries a direct learning signal from the language-model objective. A query-calibration module and an implementation exploiting chunk reuse complete the method. Validation includes a 345M model trained at 8K and evaluated on synthetic retrieval tasks up to 4M, continued pretraining of a 7B model, component ablations, and inference measurements on an H800 GPU.

Critical strengths
  1. The motivation is compelling, and the method offers a concrete refinement of landmark-gated hierarchical attention. Missing relevant blocks is an important failure mode of sparse attention, so improving selection is a meaningful target rather than only optimizing execution. Landmark Attention (Mohtashami and Jaggi, 2023, Section 3.1) already uses within-block softmax multiplied by a learned landmark gate, with gradients from the language-model loss. HiLS's distinguishing contribution is to construct the gate score from a landmark-query-weighted key and an entropy bias motivated by chunk LogSumExp, and retain selected-score learning while applying hard top-k during training. This is a meaningful development of the existing framework, rather than the first introduction of hierarchical weights or LM-trained landmark gates; it does not make hard top-k membership differentiable.
  2. The paper provides substantial evidence beyond a single retrieval demonstration. The 7B experiments include continued-pretraining controls and both retrieval and LongBench results; the ablations examine query calibration, summary variants, positional encoding, and parameter overhead. Appendix H also reports actual inference improvements: at 512K, prefill decreases from 67.0 to 5.0 seconds and decode from 85.9 to 5.5 milliseconds in the 345M, batch-one configuration. These results support practical value within the reported setting.
Critical weaknesses
  1. The suitability of a single landmark-based approximation across diverse queries is insufficiently validated. Although chunk scores depend on the current query, each chunk is represented by a summary constructed at a single landmark-query expansion point. Appendix C motivates this summary through a first-order Taylor expansion, but it remains unclear how accurate the approximation is when a query attends to different tokens within the chunk than the landmark does. Such errors could cause relevant chunks to be missed during selection. Direct measurements of approximation error and routing recall across different query-induced attention patterns would help substantiate this central modeling choice.
  2. The mechanism behind Q-Cal's substantial gains remains insufficiently explained. Removing Q-Cal increases 512K perplexity from 5.95 to 16.93 for the 345M HiLS-HoPE model (Table 9), making it a consequential component of the evaluated method. The authors suggest that token-level attention and chunk-level scoring benefit from different queries, and explicitly acknowledge in Appendix G that “the underlying mechanism is not yet fully understood.” I appreciate this acknowledgement, but the discrepancy being corrected remains insufficiently characterized. For example, does Q-Cal mainly calibrate the scale or offset of aggregated chunk masses, adjust their balance against the local-window mass, or improve which chunks are preferred? Since the surrogate already includes an entropy correction for aggregation, comparisons with simple score-calibration alternatives, alongside mass and ranking diagnostics, would help distinguish these explanations. The existing ablations demonstrate Q-Cal's utility, but do not establish whether its gains support the proposed mass-approximation account or arise from a different learned routing and weighting function. This limits the explanatory strength of the paper without invalidating the observed improvements.
Questions for authors
  1. Targeted evidence concerning the approximation: For future queries that induce substantially different within-chunk attention patterns from the landmark, how well does the single-summary estimator preserve chunk-mass rankings? Approximation-error and routing-recall diagnostics, using the original token-level query as reference and distinguishing the uncalibrated surrogate from the final calibrated scores, would help assess whether the proposed approximation supports reliable selection.
  2. Targeted evidence concerning Q-Cal: Given its substantial effect and the acknowledged uncertainty in Appendix G, is Q-Cal's benefit explained by a simple score scale/offset adjustment, including the retrieved-chunk versus local-window mass balance, or does it require query-dependent changes in chunk preference? A scalar-temperature/bias control and measurements of how calibration changes rankings and remote/local mass allocation would help distinguish these possibilities.
  3. Clarification of landmark context and position indexing: What attention mask is used to contextualize landmark tokens at each layer: all preceding tokens, only the current block, or the same local-window/sparse-retrieval rule as ordinary tokens? This concerns how the landmark hidden state/query is produced, separately from the chunk-local aggregation in Eq. (8). Does each inserted landmark receive a separate position ID and shift the position IDs of subsequent text tokens? If not, how are position IDs assigned?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.