- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies (theoretically and empirically) how ordering of state-space and transformer layers in a hybrid model affect task performance. The results show theoretically that different tasks will have different optimal layer orderings, and empirically validate some of these predictions.
Critical strengths
- The question is timely, and these contributions are (to my knowledge) novel.
- The theoretical results are very clear, and the simple examples provide nice intuitions.
- The empirical results help to show that this is not merely a theoretical point.
Critical weaknesses
-
The experiments (and general arguments) are toy, both in the sense of the scale of the architectures considered in the empirical tests, but also in the overall arguments, and that makes them less relevant to the frontier case than the authors would seem to imply (e.g., end of abstract). For example, as the authors note, any model A can be simulated by a model B which contains its layer sequence as a subsequence. Now, if we consider a frontier model like Qwen3.5 which contains repeating patterns of 3:1 S and T (for up to 48 layers), it clearly contains every sequence considered in this paper as a subsequence. So, the theoretical results tell us absolutely nothing about constraints on a model Qwen3.5, and the empirical results similarly. I still find this paper to be useful for the intuitions and insights it provides, but I think it should be presented more clearly as a toy / theory investigation that is maybe a starting place for doing more frontier-relevant work.
-
The theory results are difficult to check (even the authors apparently had an issue with their proof and had to entrust the correct version to an LM), but I find the intuitive and empirical contributions to be sufficient to make the claims broadly plausible.
Questions for authors
- What practical implications could these results actually have for frontier models that contain all patterns studied in this paper as subsequences?
- Failing that, what future directions would lead from these results to frontier-relevant insights?
2 comments
Thanks for this response; I maintain my weak positive assessment.
We thank the reviewer for their positive review, highlighting the novelty and the clarity of the theoretical results.
W1 and Q1) Connection to natural language and frontier models. We agree that the work here provides lightweight, controlled experiments of synthetic tasks that are not precisely seen in natural language. However, these insights are relevant to both natural language tasks and to frontier models. We show, both empirically and theoretically, that certain tasks require one layer ordering over another. For complex enough tasks, which are considered in Theorem 1, even a 48-layer model will begin to fail the more complex tasks. We experiment with small, representative tasks to make the theory-to-experiment connection the cleanest, but deeper tasks that require many more layers are certainly possible.
With the insights of i) subarchitecture containment and ii) every architecture having an optimal task, we can infer that under no prior knowledge of the training task, the best architectures are those with the most varying subarchitectures. Thus, this work provides a theoretical basis for common architecture designs.
W2) Parsing the theoretical results. Theorem 1 and the related results for deeper architectures are a first step towards capturing the different tasks each architecture can implement efficiently. We leave to future work identifying other task types that provide greater intuitions and efficient implementations.