acceptodds
Under review as a conference paper at ICLR 2027

The Effect of Layer Ordering in Hybrid Architectures

Abstract

Hybrid attention is rapidly becoming the dominant architectural choice for frontier models. Despite their successes, such architectures complicate model design by introducing additional choices, including the types and locations of layers. This combinatorially large design space has been studied empirically, but given the vast expense of training and evaluating models with different configurations, a principled understanding of the effects of layer selection and ordering is needed. We provide the first such analysis, showing that the dominant hybrid architecture is highly task-specific. Concretely, for any layer configuration, we prove that there exists a task where we can construct a model with this configuration that provably outperforms any other model of any different configuration under fixed resource constraints. Empirically, we observe that for standard training, learned hybrid architectures exhibit separations between layer orderings that match our theoretical findings. For instance, under the same memory budget, the optimal architecture predicted by our theory for a segmented retrieval task consistently achieves perfect accuracy, whereas the next strongest plateaus below 0.5. These results extend to deeper models, where "containing" the dominant architecture for a task indicates whether the model attains high performance. In addition, we investigate the mechanisms learned by hybrids, observing that Transformer layers contribute the largest share of performance on associative recall across architectures. This work establishes insights into task-dependent hybrid architecture design relevant to the training and scaling of frontier-scale hybrids.

Primary area:
foundation or frontier models, including LLMs
Keywords:
hybrid attention, expressivity

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6gpaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies (theoretically and empirically) how ordering of state-space and transformer layers in a hybrid model affect task performance. The results show theoretically that different tasks will have different optimal layer orderings, and empirically validate some of these predictions.

Critical strengths
  • The question is timely, and these contributions are (to my knowledge) novel.
  • The theoretical results are very clear, and the simple examples provide nice intuitions.
  • The empirical results help to show that this is not merely a theoretical point.
Critical weaknesses
  • The experiments (and general arguments) are toy, both in the sense of the scale of the architectures considered in the empirical tests, but also in the overall arguments, and that makes them less relevant to the frontier case than the authors would seem to imply (e.g., end of abstract). For example, as the authors note, any model A can be simulated by a model B which contains its layer sequence as a subsequence. Now, if we consider a frontier model like Qwen3.5 which contains repeating patterns of 3:1 S and T (for up to 48 layers), it clearly contains every sequence considered in this paper as a subsequence. So, the theoretical results tell us absolutely nothing about constraints on a model Qwen3.5, and the empirical results similarly. I still find this paper to be useful for the intuitions and insights it provides, but I think it should be presented more clearly as a toy / theory investigation that is maybe a starting place for doing more frontier-relevant work.

  • The theory results are difficult to check (even the authors apparently had an issue with their proof and had to entrust the correct version to an LM), but I find the intuitive and empirical contributions to be sufficient to make the claims broadly plausible.

Questions for authors
  • What practical implications could these results actually have for frontier models that contain all patterns studied in this paper as subsequences?
    • Failing that, what future directions would lead from these results to frontier-relevant insights?
2 comments

We thank the reviewer for their positive review, highlighting the novelty and the clarity of the theoretical results.

W1 and Q1) Connection to natural language and frontier models. We agree that the work here provides lightweight, controlled experiments of synthetic tasks that are not precisely seen in natural language. However, these insights are relevant to both natural language tasks and to frontier models. We show, both empirically and theoretically, that certain tasks require one layer ordering over another. For complex enough tasks, which are considered in Theorem 1, even a 48-layer model will begin to fail the more complex tasks. We experiment with small, representative tasks to make the theory-to-experiment connection the cleanest, but deeper tasks that require many more layers are certainly possible.

With the insights of i) subarchitecture containment and ii) every architecture having an optimal task, we can infer that under no prior knowledge of the training task, the best architectures are those with the most varying subarchitectures. Thus, this work provides a theoretical basis for common architecture designs.

W2) Parsing the theoretical results. Theorem 1 and the related results for deeper architectures are a first step towards capturing the different tasks each architecture can implement efficiently. We leave to future work identifying other task types that provide greater intuitions and efficient implementations.

Thanks for this response; I maintain my weak positive assessment.

Reviewer_1KhqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study hybrid sequence models that combine softmax-attention (T) layers with SSM (S) layers, and ask how the order of layer types affects what a model can tractably compute. Formally, T and S layers are generalized beyond what is typically used in practice. Models are compared by {S,T}^* subsequence containment (i.e., can one model emulate another), and by compression. The authors prove that any layer ordering profile has tasks which equal-or-shallower profiles not containing its compression can only compute with exponentially more state or precision (or polynomially more context window). The authors also present experiments on Transformer and Mamba hybrids on synthetic tasks.

Critical strengths
  1. The paper contains a general ordering-separation theorem: this goes beyond earlier hybrid-vs-pure separations (Cooper et al, 2026) to a statement about ordering of hybrid models. It gives a formal reason why no single hybrid layout can be the best on all tasks, which is the paper's main contribution.
  2. Controlled experiments with conservative reporting: all 16 four-layer and all 32 five-layer profiles are trained under a matched memory proxy. Results are reported conservatively with the worst seed for the treatment group and the best seed for the control groups.
Critical weaknesses
  1. Theorem 1(3) depends on Theorem 4 → Lemma 11, whose Case 4 (Lemmas 15-16, pp.32-38) is given only as "a brief and compressed version of the proof. We refer the reader to the supplementary materials for a complete rigorous proof". I would urge the authors to provide a complete, self-contained proof of Lemmas 15 and 16 with all constants defined, or restrict Theorem 1(3) to the three cases proven in the paper.
  2. Experiments do not directly test predictions by the theory
  • the authors say that "the learned hybrids exhibit identical separations as predicted theoretically" and "we show a separation in learnability between different layer orderings" (p.2). But the theorems concern expressivity under per-layer bit budgets whereas the experiments fix a summed memory proxy which does not correspond to real constraints.
  • Moreover, the ST lower bounds are tiny and the ST models in Fig2 have Mamba states far above these bounds. So the theory does not predict the observed ST failures, as I understand it.
  • Also, some abstract claims are not supported: "Consistently achieves perfect accuracy" (some TS curves peak at only 0.88 or 0.74), also "Transformer layers contribute the largest share of performance on associative recall" (inferred from probe decodability, so the claim holds only per layer but not for the aggregate)
Questions for authors
  1. Can you include in the appendix a complete, self-contained proof of Lemmas 15 and 16?
  2. In the 2-layer experiments, the ST lower bounds are well below the tested Mamba states. What explains the ST failures? Can you show a regime (sweeping lambda or I against N) where ST accuracy tracks the predicted threshold?
  3. Please give the training protocol: optimizer, batch size, steps or epochs, and the number of training and evaluation sequences. Which split did learning-rate selection use (app. C2)? Did the TST experiment in Fig9 fix total memory, or fix W and N? The caption (All models used the same window size and state size) and the text of app D4 (we also hold memory constant) disagree.
  4. How do the T-first findings relate to Bae et al (2026) who find that early attention placement underperforms in LLM pretraining? Do natural-language tasks fall closer to the TS or the ST regime of your theory?
3 comments

We thank the reviewer for their positive review, noting the careful reporting we have done in showing results in a conservative manner.

W1 and Q1) Lemmata 15 and 16 for Case 4. We agree that having all of the relevant proofs within the paper itself is ideal. Our one concern here is that given the size of some of the components, clarity might be reduced. While it is possible to include the complete proof of Case 4 within the main paper, we placed its rigorous detail in another file for brevity. The proofs are lengthy as is, with the proof of Theorem 1 amounting to 20 pages. The additional 28 pages for only Case 4 are necessary for completeness, but would bloat the manuscript.

W2a) Connection between Theory and Experiments. See W1 in the common response. In summary, we have modified the wording of the introduction to more closely align with the theoretical and empirical results. We also note that making the theory closer to the total memory, or memory proxy, likely requires further specifying the SSM architecture, where not all bits can be used independently of each other. In this way, we constrain the models theoretically slightly differently than empirically for the sake of clarity and conciseness.

W2b and Q2) Difference in scale for ST models. The current paper tests both the situations where Mamba has significantly more memory than needed as well as when Mamba has the smallest memory possible. For example, in Figure 2a, the right edge of each curve corresponds to a state expansion factor of 1, the smallest Mamba state for a given token dimension. Figure 8a shows the case where the total available memory is further decreased. Comparing the two figures, we see that the allotted memory in Figure 2a allows for one model to have consistent 1.0 accuracy, while the smaller memory in Figure 8a does not. Thus, we find that for the smallest total memory that allows one model to have perfect accuracy, the smallest Mamba state is the model that performs best.

However, we agree that there are failures of the Mamba architecture not covered in the theory. Our work focuses on a broader family without needing to fix precise architectural details; this broader family includes both Mamba and DeltaNet. This provides a more general upper bound, but tighter ones for specific architectures are certainly possible and are an excellent direction for future work.

W2c) Abstract comments. We first would like to clarify our intent behind "consistently achieves perfect accuracy". We mean to say that for some architecture setting (token dimension, window size, state dimension), all training runs reach 1.0 accuracy, and any other layer ordering with a similar memory budget fails to get close. The different curves in Figure 2a that do not reach 1.0 are only a different ratio of window size to state dimension.

For the probing experiments, we agree and have softened the language in the abstract to more accurately reflect that this result comes from layerwise decoding.

Q3) Experiment details. For completeness, we have added the mentioned training settings, also discussed in W2 of the common response. We used the training split for learning rate selection. Figure 9's caption was incorrect; it came from a previous experiment where window and state size were held constant. The caption has been corrected in the current version.

Q4) Connection to prior work about T-layer location. There exist both tasks that favor TS, as studied in Section 2.2, and ST, as briefly experimented on in Figure 11. Both exhibit tasks present in natural language: recalling inferred information from early in the context (better for TS), and using information from the early context to extract the correct token near the end of the context (better for ST). Thus, we suspect natural language is a mixture of both, supporting architectures that have both ST and TS motifs. An architecture with T layers in the middle do precisely this, matching the results in Bae et al. (2026).

Thank you for the detailed response. The added training details (C1), the convergence curves (D8), the corrected Fig. 9 caption, the softened probing claim, and the discussion of Bae et al. address my points well. Two concerns remain. Theorem 1(3) still depends on Lemmas 15-16, whose complete proof sits only in supplementary material - a self-contained proof in the appendix or a restriction of Theorem 1(3) to the three cases proven in the paper would be beneficial. Also, the response confirms that the observed ST failures are not predicted by the theory, and the requested regime sweeping λ\lambda or II against NN was not provided. I also note that the learning rate was selected on the training split rather than a held-out split. Since my main concerns are only partially resolved, I maintain my score of 3.

We thank the reviewer for their positive view of our work. We address the remaining concern about W2b and Q2 below.

On the λ\lambda against NN experiment, one challenge here is that this experiment's utility is contingent on whether an ST model with a Mamba SSM is likely to perform the task well at some scale. We note that we do not theoretically expect the ST model to always perform well beyond the established bound; this is only a necessary condition, not a sufficient one. The requested sweep would only trace the threshold if the Mamba layers were close to optimal.

We ran a small related experiment to explore these failures further. We tested two settings with more than λlog⁡∣V∣\lambda \log |V| state: the first was a normal training run, and in the second, we forced the λ\lambda-previous token to be placed in the SSM state. The difference between these tests is whether the ST model can learn a Mamba layer that buffers the last λ\lambda tokens to perform the lookup. Indeed, the forced model consistently achieves accuracy close to 1.0 while the learned model does no better than random. We suspect this is due to Mamba having a diagonal recurrence, which has been previously shown to be particularly bad at recalling the order of tokens it received. Specific SSM architecture failures are outside the scope of this work, as there are a variety, each with its own drawbacks; we use Mamba due to its common use as an SSM in practice.

We hope that this addresses this remaining concern. Thank you again!

Reviewer_uDkuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how layer ordering affects hybrid sequence models combining state-space and windowed-attention layers. It presents expressivity separations showing that, for any target architecture profile, a corresponding task can favor that profile over alternative profiles of no greater depth under specified resource constraints. The paper also develops simpler separating tasks for shallow architectures and evaluates trained hybrids on retrieval, copying, and associative-recall tasks. Additional experiments examine whether containing particular architecture profiles as subsequences predicts performance, and use linear probes to investigate layerwise representations.

Critical strengths
  1. The theoretical framework distinguishes different sources of architectural limitations. Theorem 1 separates architectures with an incompatible ordering from those with insufficient repetitions of otherwise compatible layer types. This provides a structured analysis of ordering beyond the overall attention-to-SSM ratio.

  2. The shallow architecture comparisons are systematic. Section 3.2 evaluates all four-layer profiles on segmented retrieval and all five-layer profiles on tensor lookup. These experiments provide evidence that subsequence containment is informative within the tested settings, rather than relying only on a few selected architectures.

  3. The experiments include several useful controls. The paper reports multiple seeds, learning-rate sweeps, alternative memory budgets, a Gated DeltaNet comparison, and a selective-copying task favoring ST rather than TS (Appendices C–D). These controls strengthen the evidence that the reported behavior is not simply a universal preference for one ordering in the implementation.

Critical weaknesses

1. The connection between the theoretical separations and practical hybrid architecture design remains limited.

The theoretical model allows arbitrary finite-state transitions and arbitrary nonlinear windowed functions, while constraining state size, representation precision, and attention-window length (Section 2). The separating constructions exploit these information-access restrictions, but do not comparably constrain the parameterization or computation required to implement the layer functions. As acknowledged in Section 2.2, the general constructions need not be efficiently realizable by standard neural layers.

Moreover, the windowed-attention setting does not establish that similar separations persist in hybrids retaining global causal attention. Appendix F.3 also explicitly notes that its conversion to convolutional recurrent layers does not establish the constant-factor separation in Theorem 1, part 3. These limitations do not invalidate the stated abstract results, but they substantially restrict their implications for practical architecture selection. The paper would benefit from a sharper distinction between its formal expressivity results and its broader architectural conclusions.

2. The experiments do not establish whether ordering sensitivity persists with greater depth and representative periodic interleaving.

The depth extensions use four layers for segmented retrieval and five layers for tensor lookup, while the associative-recall experiments use eight layers (Sections 3–4). This is not a systematic depth study on fixed tasks. Indeed, Table 1 shows that all eleven four-layer profiles containing TS attain perfect accuracy, suggesting that some ordering differences disappear once the necessary substructure is available. The architecture-dependent task construction in Theorem 1 does not resolve this fixed-task question.

The main eight-layer performance sweep also focuses on T∗S∗T^\ast S^\ast and S∗T∗S^\ast T^\ast. At six S-layers and two T-layers, this covers TTSSSSSS and SSSSSSTT, but not a repeated 3:1 profile such as SSSTSSST. The four-layer exhaustive sweep and the alternating 1:1 profile used in probing do not substitute for this comparison. Since SSSTSSST already contains both TS and TST, it is an especially relevant baseline for determining whether task-specific fine-grained ordering offers benefits beyond a simple periodic architecture.

3. Training sufficiency is not documented well enough to distinguish persistent capability gaps from convergence-speed differences.

The manuscript does not provide a complete account of parameter counts, training-set sizes, batch sizes, final training durations, and total training tokens for the principal configurations. Appendix C.2 describes the budgets used for learning-rate selection, but these should not be confused with the duration of the final training runs.

Learning-rate sweeps help address sensitivity to step size, but do not establish convergence. Without learning curves or extended-training comparisons, the observed gaps—particularly in the eight-layer associative-recall experiments—may partly reflect differences in optimization or sample efficiency. I am not claiming that the models are necessarily undertrained. Rather, the current reporting does not allow that explanation to be assessed. This distinction matters when interpreting empirical performance differences through expressivity limitations.

Questions for authors
  1. Attention regime and theoretical scope: Do the main empirical ordering gaps persist when the Transformer layers use global causal attention? Which conclusions are intended to apply beyond the generalized windowed-attention setting?

  2. Depth and periodic baselines: On a fixed task, how does a repeated 3:1 architecture compare with the block-ordered baselines as depth increases beyond eight layers, with matched layer counts and clearly specified resource budgets? A targeted small-model study would address this concern without requiring frontier-scale pretraining.

  3. Training sufficiency: Can the authors provide parameter counts, final training budgets, and learning curves for the key comparisons, together with evidence on whether the rankings and accuracy gaps persist under substantially longer training with appropriately adjusted learning-rate schedules?

3 comments

We thank the reviewer for the constructive comments and, in particular, for noting the depth of experiments used, ranging from varying the depths of models, varying available memory, and testing other tasks as controls.

W1 and Q1) Connection between Theory and Experiments. Thank you for the comment. We respond to this in W1 in the common response and Q1. In addition, given the large design space and complexity of hybrid model architectures, any analysis necessarily requires some abstractions. Using a specific parameterization would limit the scope of the work to a small group of architectures rather than providing general insights.

With regard to global attention vs. windowed attention, our work focuses on comparing models under fixed memory budgets. In practice, available GPU memory to run inference for models significantly limits the scale of models that can be used. Under these budgets, full-window attention uses significantly more memory than windowed variants, and we have shown that windowed attention with SSM layers is sufficient for some tasks. In this way, all results apply to full attention where the window grows with the sequence length; in these situations, full attention is an undesirable choice.

W2 and Q2) Testing increased depth and periodic interleaving. Our choice of depth for experiments with more than the number of layers that is sufficient was chosen to achieve different goals. For the synthetic tasks where we have a clean characterization in our theory, namely segmented retrieval and the TST task, we tested hybrids with two additional layers beyond the smallest architecture that performs each task. With more layers, increasing depth introduces many more architectures containing the relevant subarchitecture compared to those that do not, unbalancing the results. For associative recall, we chose the smallest depth of model that would give us access to both a 3:1 and 7:1 layer ratio, both proposed as optimal in prior work.

With regard to periodic layer combinations, i.e., other than S∗T∗S^*T^* and T∗S∗T^*S^*, we have updated Figure 4 and Figure 12 to include two interleaved architectures. On (2,1)-AR, both perform on average better than the other architectures, while on (1,2)-AR, neither model outperforms SSSSSTTT or TTSSSSSS.

W3) Establishing convergence. We agree, see W2 in the common response. For experiments with associative recall and segmented retrieval, we have added figures showing the convergence. These have been added to Appendix D.8. They show that all of the models, except one, have been trained to convergence. The one model that does not appear to converge is the highest-performing model, and further training will only improve its position among the rest of the architectures.

Thank you for the authors’ response, which has partially addressed my concerns. However, the following critical issues remain unresolved:

(1) Use of windowed attention rather than global attention in the experiments. The authors insist on using windowed attention for memory-control purposes, whereas most open-weight industrialized LLMs incorporate global attention in a proportion of their layers (e.g., Kimi-K3, GLM-5.3). Whether the paper's conclusions generalize to global-attention settings remains unverified, and the strictly memory-constrained regime studied here appears to have limited applicability to practical architectures.

(2) Shallow depth increments and periodic interleaving. During the rebuttal, the authors added only "two additional layers beyond the smallest architecture that performs each task", whereas modern LLMs typically stack tens of layers. Consequently, these supplementary synthetic experiments remain insufficient to explain or predict the complex layer-ordering dynamics of pretraining-scale LLMs.

Given that these core limitations remain unaddressed, I believe the current manuscript has not yet met the acceptance bar, and I will maintain my negative assessment.

Thank you for the discussion! We would like to address the two remaining concerns.

C1. Studying the memory-constrained regime has important implications for practical architectures and model designs. This is one of the central motivations for studying hybrid architectures, because memory constraints are critical bottlenecks even for frontier models. Kimi K3 incorporates recurrent layers precisely to reduce the memory overhead associated with global attention, while GLM-5.3 employs sparse attention and memory-efficient inference techniques. These developments highlight the practical importance of understanding models under memory constraints.

Similarly, the use of global attention does not imply having unlimited memory. The global attention layers add an Ω(L)\Omega(L) amount of memory. This is a serious bottleneck on inference, taking up a large fraction of memory that could be allocated to a deeper architecture with more SSM layers. Additionally, production models running on fixed hardware often run out of memory due to increasing context lengths. When this takes place, some form of context compression is necessary, losing fidelity of the early tokens. This effectively windows what the model can see. Studying windowed transformers is a natural choice and provides clean intuitions and experiments.

C2. We emphasize that our theory and experiments serve two different purposes. Theorem 1 establishes that tasks exist that are suited for each layer ordering, while the experiments are meant to show that these differences are learnable. Our contribution is establishing general theoretical principles and demonstrating their effects in controlled experiments, rather than claiming a complete predictive theory of frontier-scale LLM training.

To this end, instead of experimenting on complex and less-intuitive tasks only solvable by very deep architectures, we focus on simple, natural tasks that are suited for small-scale architectures. These both show that these differences are empirically observable and provide intuition towards the type of task suited for each architecture. We find that identifying other natural tasks suited for specific deeper models is an interesting direction of future work building on both the theoretical results for architectures of all depths and empirical results for natural tasks for small depth. We believe the current results constitute a substantive step toward understanding the role of layer ordering in hybrid architectures.

Comments on the paper

1 comment

We thank the reviewers for their careful engagement with our work. We are glad to see the reviewers' recognition of the importance of studying hybrid layer orderings. The useful intuition this work provides is mentioned by multiple reviewers as a strong selling point of the work. Among all reviews, two points were consistently brought up as potential concerns: the alignment between the empirical and theoretical results, and certain missing experimental details from the work. We address both of these in this common response.

W1) Alignment between experiments and theory. Indeed, the theoretical results and empirical results constrain slightly different measures of used memory. Specifically, our work provides a theoretical treatment of deep hybrid models from the perspective of constraining memory through available bits to each layer, bits per token, and window sizes. The experiments constrain a proxy that measures total memory used by each layer, combining all three. However, a separation under total memory, with any model performing perfectly, implies there is a separation when all three sources are individually constrained. Extending the theory to be constrained under the summed memory proxy requires breaking the assumption that each bit of state can be accessed symmetrically. If we were to do so, we would need to integrate specific architecture choices into the theory, sacrificing generality and clarity---and substantially narrowing the resulting insights. Instead, our choice enables a much more general understanding of hybrid layer ordering.

For the phrasing used in the paper, terms like "identical separations" are meant to convey that when theory predicts one architecture is optimal, such as, for example, TS, then we observe the same optimal architecture experimentally. We have reworded the second page of the work to clarify this.

W2) Experimental details. We appreciate the reviewers spotting this and have added in all remaining details. We have placed these in Appendix C.1. We briefly comment on some of the choices here. These settings have been selected so that for all models, the number of training epochs used was sufficient to train these models to convergence at each setting. The vocabulary size and training data size were chosen to let the top-performing model achieve 1.0 accuracy and none of the rest, providing a strong testbed for the different architectures. The number of epochs was selected to be large enough to allow all models to converge and train past convergence, in case any of the architectures needed significantly longer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.