- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The manuscript investigates whether hybrid linear-attention language models truly require multiple full-attention layers for long-context retrieval, or whether a single full-attention layer is sufficient. Through controlled experiments at 403M and 1.7B scale, the authors vary the number, placement, and reuse of attention layers in Gated-DeltaNet-based hybrids and evaluate validation loss, downstream tasks, real-data recall, synthetic passkey retrieval, and continued-pretraining for long-context retrieval. Their main finding is that one full-attention layer can largely match the standard multi-attention hybrid recipe while substantially reducing attention KV-cache cost, whereas pure linear models fail on real-data retrieval. They further argue that retrieval performance is governed more by training data and context length than by the number of attention layers, with placement and looping acting as secondary design choices.
Critical strengths
1.) Broad empirical study within the chosen setup: The manuscript systematically varies several architectural factors, including the number and placement of attention layers, attention width, looping, and continued-pretraining conditions, rather than relying on a single configuration.
2.) Evaluation goes beyond synthetic passkey retrieval: The manuscript also includes real-data recall and long-context retrieval experiments after continued pretraining.
Critical weaknesses
1.) The manuscript is difficult to follow and the related-work coverage is incomplete: The manuscript is difficult to follow due to fragmented exposition, abrupt transitions, and an unclear overall narrative. Several relevant works on linear and efficient attention, including Random Feature Attention [1], Performer [2], RACE Attention [3], CosFormer [4], and YOSO [5], are omitted. Discussing these works would provide a more complete view of how linear attention methods have evolved and better contextualize the design choices considered in this manuscript.
2.) The motivation for the single-attention-layer design is underdeveloped: The manuscript motivates the cost of multiple full-attention layers but does not clearly show how retrieval is distributed across those layers. An analysis of which layers matter most for retrieval, or how retrieval changes as individual layers are removed, would better motivate the choice of studying a single full-attention layer.
3.) The empirical evidence supports substantially narrower conclusions than the manuscript’s overall narrative suggests: The main experiments are limited to 403M and 1.7B models in a specific Gated-DeltaNet hybrid setup. These results do not establish that a single full-attention layer is sufficient for hybrid LLMs in general, so the claims should be narrowed or validated across larger scales, other architectures, and broader long-context tasks.
4.) Reproducibility and submission-compliance issues should be addressed: The manuscript does not include the required AI-use disclosure statement, and the code is not available to reviewers. Releasing the code and exact experimental configurations would improve reproducibility, especially given the many architectural variants, seeds, and continued-pretraining runs.
5.) The central premise of the manuscript is not sufficiently established from prior work: The manuscript assumes that more full-attention layers lead to stronger retrieval, but this relationship is not clearly established by prior work. Since challenging this assumption is central to the manuscript, the authors should either provide supporting evidence from the literature or qualify the claim.
In general, I believe the manuscript asks an important question, but the current study reads largely as a sequence of architectural ablations rather than a convincing scientific argument. The manuscript would benefit from a clearer empirical motivation for why a single full-attention layer is the right object to study, for example by showing how retrieval behavior is distributed across existing attention layers, whether removing one layer changes the role of others, and whether some layers are consistently more important than others. I would also encourage the authors to reframe the manuscript around prior empirical evidence and to clearly identify which previous works actually support the claimed "implicit assumption" that more full-attention layers lead to better retrieval. This would make the motivation and overall narrative substantially more convincing.
[1] Peng, H., Pappas, N., et al. Random Feature Attention. ICLR 2021.
[2] Choromanski, K., Likhosherstov, V., et al. Rethinking Attention with Performers. ICLR 2021.
[3] Joshi, S., Chowdhury, A., et al. RACE Attention: A Strictly Linear-Time Attention Layer for Training on Outrageously Large Contexts. ICLR 2026.
[4] Qin, Z., Sun, W., et al. cosFormer: Rethinking Softmax In Attention. ICLR 2022.
[5] Zeng, Z., Xiong, Y., et al. You Only Sample (Almost) Once: Linear Cost Self-Attention Via Bernoulli Sampling. ICML 2021.
Questions for authors
1.) Refer to Weaknesses.
2.) Does the conclusion hold beyond retrieval-oriented tasks? In particular, have you tested tasks involving long-range aggregation, multi-hop reasoning, code, or generation?
3.) How should practitioners choose the placement of the single attention layer? The results suggest placement matters, but the manuscript does not yet provide a clear principle for predicting the best location without running a sweep.
4.) How sensitive is the one-layer result to the specific Gated DeltaNet backbone? Do you expect the same conclusion to hold for other linear-attention or state-space hybrids, and is there any evidence supporting that generalization?