acceptodds
Withdrawn from ICLR 2027

One Full-Attention Layer Is Enough for Hybrid Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Hybrid language models replace most softmax attention with linear mixers such as Gated DeltaNet and keep about 25% of the layers as full attention, the recipe of Jamba, Samba, and Qwen3-Next. Those attention layers exist for one capability, long-context retrieval, and they are also the part of the model whose serving cost grows with context: at 32K the six attention layers of a 24-layer 25% hybrid hold 0.38 GB of KV cache per sequence. We ask whether the recipe can be replaced by a hybrid with a *single* full-attention layer. Using 65 pretrained hybrids at 403M and 1.27B (identical data and optimization, 2–6 seeds per cell) and over 80 short continued-pretraining runs, we find that it can, without giving anything up. Against the 25% recipe, one layer matches loss (2.619 vs. 2.616), downstream average, real-data recall (SWDE and FDA), and, after a few hundred needle-augmented continued-pretraining steps, real-text retrieval to 32K, with 1/6 of the attention KV cache (1/24 of full attention). One layer is *necessary*: pure linear models collapse on real-data recall and cannot learn real-text needle retrieval even when trained on it. It is *sufficient* because what bounds real-text retrieval in every architecture we test, from one attention layer to twenty-four, is data and training length: under plain pretraining every model sits on a recency floor beyond 4K, and 500 needle-augmented steps (5% of pretraining tokens) lift a single layer from 0.16 to 0.96 at 8K while 12 or 24 layers learn no better. The remaining choices are secondary knobs with mapped effects: placement (mid-stack for loss and FDA, top for a reliably forming pretrained circuit), looping the layer with shared weights (matches three stacked layers everywhere and beats them when needle data is scarce, at 3× the single-layer KV cache), and inference-time looping (a free unlock for an unformed circuit that becomes harmful once the circuit is trained).

Primary area:
foundation or frontier models, including LLMs
Keywords:
hybrid language models, inference-time depth recurrence, long-context retrieval, KV cache efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_QH2QOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The manuscript investigates whether hybrid linear-attention language models truly require multiple full-attention layers for long-context retrieval, or whether a single full-attention layer is sufficient. Through controlled experiments at 403M and 1.7B scale, the authors vary the number, placement, and reuse of attention layers in Gated-DeltaNet-based hybrids and evaluate validation loss, downstream tasks, real-data recall, synthetic passkey retrieval, and continued-pretraining for long-context retrieval. Their main finding is that one full-attention layer can largely match the standard multi-attention hybrid recipe while substantially reducing attention KV-cache cost, whereas pure linear models fail on real-data retrieval. They further argue that retrieval performance is governed more by training data and context length than by the number of attention layers, with placement and looping acting as secondary design choices.

Critical strengths

1.) Broad empirical study within the chosen setup: The manuscript systematically varies several architectural factors, including the number and placement of attention layers, attention width, looping, and continued-pretraining conditions, rather than relying on a single configuration.

2.) Evaluation goes beyond synthetic passkey retrieval: The manuscript also includes real-data recall and long-context retrieval experiments after continued pretraining.

Critical weaknesses

1.) The manuscript is difficult to follow and the related-work coverage is incomplete: The manuscript is difficult to follow due to fragmented exposition, abrupt transitions, and an unclear overall narrative. Several relevant works on linear and efficient attention, including Random Feature Attention [1], Performer [2], RACE Attention [3], CosFormer [4], and YOSO [5], are omitted. Discussing these works would provide a more complete view of how linear attention methods have evolved and better contextualize the design choices considered in this manuscript.

2.) The motivation for the single-attention-layer design is underdeveloped: The manuscript motivates the cost of multiple full-attention layers but does not clearly show how retrieval is distributed across those layers. An analysis of which layers matter most for retrieval, or how retrieval changes as individual layers are removed, would better motivate the choice of studying a single full-attention layer.

3.) The empirical evidence supports substantially narrower conclusions than the manuscript’s overall narrative suggests: The main experiments are limited to 403M and 1.7B models in a specific Gated-DeltaNet hybrid setup. These results do not establish that a single full-attention layer is sufficient for hybrid LLMs in general, so the claims should be narrowed or validated across larger scales, other architectures, and broader long-context tasks.

4.) Reproducibility and submission-compliance issues should be addressed: The manuscript does not include the required AI-use disclosure statement, and the code is not available to reviewers. Releasing the code and exact experimental configurations would improve reproducibility, especially given the many architectural variants, seeds, and continued-pretraining runs.

5.) The central premise of the manuscript is not sufficiently established from prior work: The manuscript assumes that more full-attention layers lead to stronger retrieval, but this relationship is not clearly established by prior work. Since challenging this assumption is central to the manuscript, the authors should either provide supporting evidence from the literature or qualify the claim.

In general, I believe the manuscript asks an important question, but the current study reads largely as a sequence of architectural ablations rather than a convincing scientific argument. The manuscript would benefit from a clearer empirical motivation for why a single full-attention layer is the right object to study, for example by showing how retrieval behavior is distributed across existing attention layers, whether removing one layer changes the role of others, and whether some layers are consistently more important than others. I would also encourage the authors to reframe the manuscript around prior empirical evidence and to clearly identify which previous works actually support the claimed "implicit assumption" that more full-attention layers lead to better retrieval. This would make the motivation and overall narrative substantially more convincing.

[1] Peng, H., Pappas, N., et al. Random Feature Attention. ICLR 2021.

[2] Choromanski, K., Likhosherstov, V., et al. Rethinking Attention with Performers. ICLR 2021.

[3] Joshi, S., Chowdhury, A., et al. RACE Attention: A Strictly Linear-Time Attention Layer for Training on Outrageously Large Contexts. ICLR 2026.

[4] Qin, Z., Sun, W., et al. cosFormer: Rethinking Softmax In Attention. ICLR 2022.

[5] Zeng, Z., Xiong, Y., et al. You Only Sample (Almost) Once: Linear Cost Self-Attention Via Bernoulli Sampling. ICML 2021.

Questions for authors

1.) Refer to Weaknesses.

2.) Does the conclusion hold beyond retrieval-oriented tasks? In particular, have you tested tasks involving long-range aggregation, multi-hop reasoning, code, or generation?

3.) How should practitioners choose the placement of the single attention layer? The results suggest placement matters, but the manuscript does not yet provide a clear principle for predicting the best location without running a sweep.

4.) How sensitive is the one-layer result to the specific Gated DeltaNet backbone? Do you expect the same conclusion to hold for other linear-attention or state-space hybrids, and is there any evidence supporting that generalization?

Reviewer_ijKfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper trains a variety of hybrid LMs at small scale, showing that using just one attention layer recovers most of the gains of using many, including an investigation of several weight-tied and looped variants.

Critical strengths

-The findings are convincing: that one attention layer is sufficient for relatively small models trained on relatively few tokens, benchmarked on small tasks.

Critical weaknesses
  • The paper is essentially unreadable.
  • The tables and graphs are impenetrable.
  • This has been noted by many other papers (e.g., Mamba-2).
  • Any standout findings are hard for a human reader to notice.
  • In Table 1 and Figure 1, it's kind of unclear which (if any) cells correspond to the one-attention-layer case.
  • In Table 2, it's unclear why only the mid/top/loop arms have ranges while the baseline experiments do not.
  • Table 3: why so many slashes? Difficult to read.
  • Needle augmented pretraining sounds a bit like training on test?
Questions for authors

N/A

Reviewer_adKvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

authors find that one full-attention layer in a GDN hybrid can approach hybrids that contain multiple full attention layers -- specifically on language modeling and single needle retrieval tasks

Critical strengths
  • study is performed rigorously across different scales and seeds! the evaluation is done across different tasks ranging from real-world recall, to long-context single needle recall and short-context reasoning tasks
Critical weaknesses
  • the evidence that single layer is enough for hybrid is rather weak due to:
    • authors use very similar looking sequences of single needle recall during continued pretraining, and evaluate on the same sequences -- this training and testing on similar looking sequences may not imply general recall abilities -- clearly, FDA task performance drops as one decreases the number of attention layers, meaning single layer attention may not be enough for general purpose recall
Questions for authors
  • can the authors evaluate on more "OOD" tasks (like FDA or SWDE) to confirm whether single layer attention is enough

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.