- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Hierarchical Landmark Sparse (HiLS) Attention, which introduces a new perspective on block scoring: approximate the full-attention-induced block mass using a learned, attention-weighted key summary and an entropy bias derived from a first-order LogSumExp expansion. Retrieved blocks are attended to independently, and their outputs are fused using the same routing scores. This lets the language-modeling loss directly train the scores of selected blocks. The implementation additionally uses low-rank query calibration (Q-Cal), HoPE positional encoding, and kernels that reuse retrieved KV blocks across adjacent queries.
The paper evaluates 345M models trained from scratch and a 7B model adapted through continued pretraining. The reported results show competitive in-domain perplexity and strong length extrapolation, including 96% single-needle accuracy at 4M tokens after 8K-context training. The 7B results also provide evidence of practical adaptation and improvements on LongBench. These results involve RULER-style synthetic training data, and the strongest 7B results use approximately 50B tokens of full-parameter continued pretraining.
Critical strengths
-
Connecting block relevance to LogSumExp attention mass motivates an attention-weighted summary with entropy calibration. Using the routing scores in output fusion provides a direct learning signal for selected blocks. This is a meaningful contribution beyond simply changing the retrieval heuristic, while building on the hierarchical gating framework of Landmark Attention.
-
The small-model experiments combine comparable in-domain perplexity with strong retrieval extrapolation, and the ablations demonstrate the importance of learned summaries, Q-Cal, and positional encoding.
Critical weaknesses
-
Figure 4 measures 345M inference at batch size 1 on H800 against a shared Triton implementation. Appendix H.1 explains this choice, but it does not establish the advantage over an optimized Hopper dense-attention backend. There is no 7B training throughput/MFU comparison, no disclosure of decode split-KV or GQA-packing settings, and no memory or concurrency evaluation. This matters because practical training and long-context inference efficiency are central claims. Sparse KV access alone does not establish reduced GPU-resident KV storage. Moreover, the O(1) decoding description in Appendix H.1 omits the history-dependent summary scan acknowledged in Section 3.
-
The evaluation protocol and statistical reporting are insufficiently specified, and the reported RULER tasks provide limited coverage of aggregation and semantic QA. GPQA is included as a 5-shot benchmark, but a long-output CoT evaluation is not established by the disclosed protocol. Equal synthetic-data exposure makes the internal comparison procedurally fair; however, it does not isolate improved associative retrieval from an advantage in suppressing distractors. Finally, the ablations show that the complete method depends strongly on HoPE and Q-Cal, so its advantage should not be attributed entirely to the Taylor-derived score. Average downstream performance also masks decreases on some tasks, including MMLU, HellaSwag, and RACE relative to the matched CPT baseline.
-
The evidence covers 345M pretraining and continued pretraining of a single 7B hybrid-attention model; it does not establish scaling behavior or readiness for distributed long-context training and deployment. Retrieval accuracy at an extrapolated length is not a validated general-purpose context window: historical KV storage, routing costs, and serving constraints still matter. An explicit discussion and narrower claims would address this concern without additional scaling experiments or diminishing the block-scoring contribution.
Questions for authors
-
Please report steady-state training step time, tokens/s/GPU, MFU, and peak memory for HiLS and the matched OLMo3-512swa-CPT baseline. Use the same hardware, precision, batch sizes, parallelism, and activation-recomputation settings, including routing, backward, communication, and optimizer costs. After sufficient performance warmup, ten stable optimizer steps would be a reasonable initial measurement window; extend it if timing remains variable. Please specify the FLOPs accounting, since applying the dense-attention formula to HiLS can misrepresent MFU, and lower MFU alone does not imply lower throughput. Megatron-LM is a recommended implementation.
-
Can you supplement the shared-Triton comparison with FA3 or an equivalently optimized Hopper backend? The supplied implementation appears to use a default Triton path without the relevant Hopper-specific optimizations; please clarify its actual TMA, asynchronous matrix-operation, and warp-specialization usage. For Figure 4's batch=1, Q-heads=16, KV-heads=2 setting, report the dense and HiLS launch configurations, split-KV settings, and appropriate GQA-packing choices. Please provide profiling evidence for SM occupancy, and end-to-end latency. FA3 split-KV and pack-GQA should be tuned, with comparable tuning effort for HiLS. When batch-level parallelism is insufficient to saturate the GPU, shorter selected sequences do not necessarily reduce latency; routing, top-K selection, and other preprocessing overheads may even make sparse attention slower. This is why a measured comparison against a properly tuned dense baseline is necessary.
-
Is the historical KV cache fully GPU-resident during the reported measurements, or is offloading actually used? Please report persistent and peak memory, distinguishing original KV storage, summary keys, landmark-related storage, indices, and temporary workspaces. A modest concurrency sweep should show both performance at equal concurrency and achievable throughput under an equal GPU-memory budget, preferably with a common latency target. Please also reconcile the decode claim with the per-query routing scan and include routing and top-K costs in the measurements.
-
Please specify toolkit, task configurations, prompts and few-shot selection, answer extraction, sample counts, greedy versus sampling, temperature, decoding top-k/top-p where applicable, generation limits, stopping criteria, input truncation, and landmark-aware length accounting. Which RULER subtasks and parameters are aggregated in Table 3? For synthetic evaluation, report multiple dataset-generation seeds with paired samples across methods and mean/std; deterministic greedy decoding does not remove randomness in synthetic data construction. Standardized tooling such as lm-eval-harness for task evaluation is recommended. DO NOT use custom scripts for evaluation.
-
Can you provide representative results on aggregation tasks such as RULER CWE/FWE and semantic QA such as RULER QA-SQuAD/QA-Hotpot? To support the broader claim of replacing full attention while preserving capability, please also provide one meaningful long-output reasoning evaluation, such as
gpqa_diamond_cot_n_shot, andaime, on which the baseline has nontrivial performance. The existing GPQA 5-shot row does not establish that long generation was evaluated. Use matched prompts and max generation tokens, and report accuracy and output lengths. -
Appendix F indicates equal 5% RULER-style augmentation for the relevant training comparisons; I do not interpret this alone as unfairness. Can you clarify how much the retrieval conclusions depend on task-specific augmentation and distractor suppression? I recommend a 7B control in which both HiLS and the matched CPT baseline are trained without RULER-style augmentation, holding the remaining training settings and token budgets fixed. Construct synthetic test sets with five independent dataset-generation seeds, use paired samples across methods within each seed, and report the mean score and standard deviation across the five sets. This would help separate augmentation-dependent gains from generalization to unseen retrieval instances; it would not by itself establish whether distractor suppression explains the extrapolation results. A targeted control, such as isolated MQAR with random-padding and no-padding variants, could help. In Appendix A.1, the prose describes a maximum over normalized head scores, whereas Eq. (12) uses raw logits; which is implemented?
-
discussion. Could you distinguish the demonstrated block-scoring and retrieval-extrapolation results from the broader prospect of using HiLS as a core building block for ultra-long-context foundation models? Please discuss the untested scaling and distributed-training assumptions, including context-parallelism support, and distinguish empirical retrieval extrapolation from a generally usable, deployable context window. Sparse KV access does not automatically reduce resident KV capacity; if full historical KV is retained, summaries and other auxiliary state add storage overhead. If the evaluation procedures and optimized baseline comparisons requested above do not sustain the reported quality or efficiency gains, please further narrow the claims about HiLS as a core building block for ultra-long-context foundation models to match the evidence that remains supported. I do not request scaling law, CP implementation, or evaluation of reasoning-trajectory drift in long-horizon optimization tasks for this point.