acceptodds
Under review as a conference paper at ICLR 2027

Library of Layers (LoL): Adaptively Ordering Transformer Layers at Runtime

Abstract

Computer programmers chain together the same reusable computational primitives, for example, conditional statements or loops, but in different orders to implement different algorithms. In contrast, existing language model architectures always execute their layers in the same order. We introduce Library of Layers (LoL), an architecture that dynamically orders its layers during inference to solve the problem at hand. LoL has a single hyperparameter that allows it to transition seamlessly between parameter reuse and sparsity, generalizing the parameter-efficiency and FLOPs-efficiency tradeoffs of dense, depth-recurrent, and mixture of experts architectures in one unified family. We show that LoL achieves competitive performance with all three above baselines when pretrained on text data. On synthetic sudoku and maze data, we observe that the model employs distinct layer orderings to implement different solution strategies, and we can intervene on the layer ordering to force a model to apply a specific solution strategy to a problem instance. In a multilingual natural language setting, we find that LoL chooses different layer order patterns based on language and script. LoL provides a path toward a future in which models dynamically compose their own computational graph, enabling more efficient and controllable reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language Model, Reorder, looped, mixture-of-experts

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_UBd2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Library of Layers (LoL), a decoder-only Transformer architecture whose routed core adaptively selects the next Transformer block from a globally shared library. By varying the library size relative to the number of executed core depths, the architecture supports both parameter-reuse and sparse-computation regimes. To accommodate different routing decisions across a sequence, LoL uses strided window attention: tokens within a window share an expert, routing depends on preceding windows, and window boundaries shift across depths. Full causal attention is retained in the fixed-order prelude and coda. The architecture also incorporates depth-dependent FiLM conditioning and, in some experiments, progressive depth growth during training.

Experiments investigate multilingual routing, strategy control on Sudoku and maze tasks, mathematical reasoning at the 1B-parameter scale, and comparisons with looped Transformers and mixture-of-experts models. The principal findings are that routing paths can encode controllable task-dependent behavior and that LoL achieves broadly competitive performance across the evaluated architectural regimes.

Critical strengths

1. A coherent architecture connecting parameter reuse and sparse computation.

Separating library size from execution depth provides a useful framework for studying parameter sharing and conditional computation within one model family. The comparisons span dense, looped, and sparse baselines rather than a single reference architecture. In particular, Figure 4 provides encouraging results against parameter-matched looped models, with evaluation over three random seeds. This supports the feasibility of learning flexible execution paths without necessarily sacrificing language-modeling quality.

2. The routing-intervention experiments provide meaningful evidence of controllability.

The synthetic experiments go beyond visualizing routing correlations: the authors extract strategy-specific paths and force those paths at inference after replacing the strategy description with neutral input. The maze experiment is particularly useful because the strategies share the same output syntax and differ in exploration order, reducing the possibility that the intervention merely selects a superficial formatting convention. The near-perfect strategy adherence is a substantive result, although its interpretation should remain tied to the controlled training setup.

Critical weaknesses

1. Central architectural choices are not reasonable enough and need stronger justification.

The arguments against first-token sequence-level routing and token-level routing lack controlled empirical comparisons. Attention-space mismatch and optimization instability should also be distinguished from causal correctness. Existing attention-mask comparisons do not establish the superiority of window-level routing or the need to route attention rather than retain fixed full causal attention and route only feed-forward blocks.

Natural-language experiments use only 1,024-token contexts, leaving long-range dependencies and sensitivity to window size and stride untested. Restricted attention in the routed core could limit long-context performance.

The globally shared library and depth-dependent FiLM need matched-parameter/FLOPs comparisons against depth-wise or stage-wise libraries and simpler or no depth conditioning. Observed neighboring-depth expert specialization makes these alternatives particularly relevant.

2. The benefits specifically attributable to adaptive ordering remain unclear.

Performance and baseline. Tables 1 and 4 demonstrate competitiveness rather than consistent gains relative to reported uncertainties, with full attention performing best on GSM8K. The large ARC-E improvement already achieved by the windowed baseline warrants investigation, especially since Figure 9 favors full attention in language-modeling loss. Exact token budgets, comparable optimization, and repeated training seeds are needed.

Efficiency. The early loss advantage is attributed to progressive depth growth rather than adaptive ordering alone, while the MoE baseline achieves slightly lower final training loss. FLOPs comparisons do not establish practical speedups; throughput, decoding latency, and memory measurements should account for routing overhead.

Questions for authors

Please refer to Weaknesses.

Additionally, I would like to ask the authors to discuss the connections between LoL and [1, 2], particularly regarding parameter sharing and model reuse in [1] and adaptive routing and composition of reusable computational blocks in [2]. Should these works be cited and as related work to clarify LoL’s distinctive contributions?

[1] Language Model Networks: Supervision-Efficient Learning through Dense Communication. ICML 2026

[2] Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning. ICLR 2018

2 comments

Thanks again for your feedback. We made a strong effort to address all your points, including substantial edits to our manuscript, and we would greatly appreciate it if you would consider increasing your score accordingly. Do you have any other questions?

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns, particularly through the new ablation studies and comparisons with related architectures.

The new results address several of my earlier questions. However, the comparisons with MoM and UniPool primarily focus on language modeling performance and do not establish whether adaptive complete-block ordering provides distinct advantages over simpler routing alternatives in terms of controllability. In addition, while the interleaved full-attention experiment is useful, the effectiveness of the proposed architecture on longer contexts remains unverified.

Given these remaining concerns, I will maintain my original rating. Thank you again for the constructive discussion.

Reviewer_dm6qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LoL replaces a transformer's middle layers with a shared library of complete Transformer blocks selected by a top-1 router at each executed depth. Routing is shared within token windows, uses causally available information, and is combined with strided window attention, depth-dependent normalization, and depth growing. Varying the library size enables parameter reuse or sparse execution. Experiments demonstrate path-conditioned control of synthetic sudoku and maze traces, competitive downstream performance against dense baselines, and a small held-out loss advantage over the tested looped configurations. The reported sparse MoE comparison has a small training-loss disadvantage for LoL.

Critical strengths
  1. The architecture addresses a concrete difficulty in routing complete attention blocks. Section 3 and Algorithm 1 provide a clear construction that preserves causality in both attention and routing decisions. The combination of a shared layer library and shifted attention windows is a useful implementation to investigate across parameter-reuse and sparsity regimes.

  2. The intervention experiments demonstrate that execution paths can serve as a behavioral control interface. After removing the strategy information at evaluation, forcing a strategy-specific mode path produces the requested behavior on unseen problems. The maze setting is particularly informative because the strategies share the same output syntax. Successful generalization under these interventions is a meaningful result.

  3. The empirical study includes substantial tuning and relevant comparisons. Figure 4 shows a small, consistent held-out loss advantage over the tested Huginn configurations at three library sizes, with results from three seeds. Section 4.3 describes learning-rate sweeps, and the MoE comparison also uses three seeds. Table 4 and Figure 19 include a strided-window dense baseline; Section 4.3.2 matches the MoE baseline's attention pattern and approximately matches active parameters. Appendix A studies growing, load balancing, router sharing, and attention patterns in dense baselines. These are meaningful controls, and the paper reports its MoE loss disadvantage.

Critical weaknesses
  1. The benefit of adaptive block selection is not sufficiently isolated. The dense and looped comparisons are relevant evidence against fixed execution architectures, and Figure 4 provides a positive result. However, they compare complete designs rather than isolate adaptive selection within LoL. The highest-priority control is a small model trained with an input-independent expert-selection sequence using the same shared library, attention mask, executed depth, depth-dependent normalization, growing policy, and a comparable budget. The handling of adaptive output weights should be explicit. This would test whether the input-dependent selection and execution paths add value beyond parameter sharing and the surrounding design. Replacing routes only after training is a useful diagnostic but can introduce distribution shift. I do not require exhaustive ablations of every component; a focused test of this central architectural choice would be more decisive.

  2. The comparisons cover broad architecture families but do not sufficiently test the closest alternatives. Mixture-of-Modules (Gong et al., 2024) dynamically selects reusable attention and FFN modules, UniPool (Huang et al., 2026b) studies globally shared expert pools, and Ternovtsii and Bilak (2026) route smaller complete Transformer blocks. Section 2 acknowledges these connections. An incremental extension can be a worthwhile contribution if its benefits are established rigorously. Here, a representative comparison such as a shared-pool FFN MoE would help distinguish the value of routing complete blocks from that of sharing experts across depths. The existing conventional MoE comparison matches attention and approximately matches active parameters, but does not isolate these choices. This need not entail reproducing every related system: one carefully matched alternative, with explicit budgets and tuning, could substantially strengthen the case.

  3. The control results are convincing within their setting, but their architectural specificity remains uncertain. The maze evaluation uses 20,000 unseen mazes per strategy and shared output syntax, so this is meaningful behavioral evidence. Restricting strategy information to the router encourages path-conditioned control but does not guarantee the observed generalization. If controllability is the main benefit justifying the architecture, applying the same protocol to a simpler routed baseline would clarify what the complete-block library adds. The results establish control through execution paths, which change expert identities, repetition counts, and order together. They do not independently establish reusable algorithmic functions of individual blocks. A pure ordering intervention would be optional diagnostic evidence, not an acceptance requirement; appropriately bounded claims would also address this interpretive issue.

  4. The accuracy and efficiency results leave the overall tradeoff unresolved. Table 4 shows small numerical improvements over the strided-window dense baseline, including GSM8K 27.9 versus 25.9 and PIQA 59.7 versus 59.0, while the sliding-window baseline is closer. The reported benchmark standard errors do not quantify variation across training runs. Figure 4 is positive held-out evidence, but its three-seed min/max bars should not be interpreted as confidence intervals. In the sparse setting, only the 3:2 ratio is tested, with final training loss approximately 2.80 for LoL versus 2.75 for MoE. The suggested downstream compensation is not evaluated for these checkpoints; Table 1 concerns different models. Held-out loss and a compact downstream evaluation of the existing LoL/MoE checkpoints would clarify this tradeoff. Same-hardware timing is needed only to substantiate practical speed claims; otherwise, efficiency conclusions should remain limited to FLOPs and parameters. I do not require LoL to outperform every baseline: comparable accuracy plus a rigorously established additional capability could suffice.

  5. Reporting clarifications are needed, but are secondary to the experimental questions above. Appendix B provides useful details, but a consolidated table should specify the selected learning rates, batch sizes, schedules, router parametrization, score transformation, and load-balancing bias updates. Although Appendix B explains that gradients reach the router through the selected score without differentiating the discrete choice, the score transformation and resulting range of alpha_k should be explicit. Please also briefly motivate the first-window depth vector e^(i) beyond the first-token state s_0^(i-1), which already changes across depths. Report total and active parameters, tokens processed, depth-growing settings, and FLOP accounting for each main comparison. These quantities need not all be identical; specify the matched budgets and explain remaining differences.

Questions for authors
  1. Under matched architectural settings and a comparable budget, how does adaptive LoL compare with a small model trained with input-independent expert selections? How are the output weights handled?

  2. Can one representative shared-pool FFN or modular-routing baseline clarify the benefit of routing complete blocks, using either language-model quality or the same maze-control protocol?

  3. Can the existing sparse LoL/MoE checkpoints be evaluated on held-out loss and a compact downstream suite, with uncertainty reported in a way that distinguishes benchmark sampling from training-seed variation?

  4. Can the authors consolidate the selected configurations, budget matching, and router score definition, and state whether the efficiency claims concern FLOPs or measured execution time?

I am willing to increase the score if the authors can provide strong evidence to address the critical weaknesses and my questions.

3 comments

Thanks again for your feedback. We made a strong effort to address all your points, including substantial edits to our manuscript, and we would greatly appreciate it if you would consider increasing your score accordingly. Do you have any other questions?

Thanks for the detailed response.

I believe that W5 and Q4 are solved. Tables 2-4 are good.

For W1, the random routing run in Figure 12 is no problem. But the two curves almost overlap. How is alpha set when the expert is drawn at random? My reading is that input-dependent routing has only a small effect on the LM loss, and its real value is the control interface. I think that is a fine position, but the paper should say it directly.

On W2, LoL ends up close to both in training loss, so I don't see a clear quality gain from routing complete blocks over sharing FFN experts across depths. The case for complete blocks then rests on the path-level control, right? That control has not been tested on a routed baseline such as the global-pool MoE. I understand this may not be feasible during the discussion period.

For W3, I couldn't find it in the revised PDF. The abstract and the introduction still credit the layer ordering. I hope this wording can go into the paper.

I'm less convinced by the new safety result. Score-0 and score-4/5 documents are very different in content, so I guess most content features would also give a high AUROC. A baseline probe, for example on the router inputs or on a dense model's hidden states, would show whether routing adds anything beyond content.

W4 is now answered, but not in LoL's favor. The MoE has lower held-out loss and higher accuracy on all four benchmarks, even though LoL sees more tokens under the same FLOP budget (Table 3).

The accuracy is not comparable in the sparse regime, and the control capability has not been compared with other routed models. The path-level control on sudoku and mazes remains the main contribution, and my original score already credited it. So I will keep my score at 2.

AuthorsOpenReview ↗

Official comment by Authors

We thank the reviewer for their response, but we want to push back on several points:

LoL’s real value lies in dynamically ordering its layers during inference to solve the problem at hand. Our main objective is to present this new architecture and the axis of control and interpretability it opens, which we demonstrate on synthetic maze and Sudoku data as well as on real multilingual text. Performance was a supporting goal: we wanted to show the community that this design also works in practice and is competitive with dense, depth-recurrent and MoE models, not that it surpasses them. The paper is consistent on this point.

How is alpha set when the expert is drawn at random?

The router weight is set to one for the random routing run.

For W3, I couldn’t find it in the revised PDF.

From our understanding of W3, the reviewer asked a pure ordering intervention would be optional diagnostic evidence, we provided random ordering and we showed that LoL solves the puzzles and mazes using mainly one dominant strategy and it does not follow any strategy even when it is clearly asked in the prompt, fig 22 and 26 show this.

W4 is now answered, but not in LoL’s favor. The MoE has lower held-out loss and higher accuracy on all four benchmarks, even though LoL sees more tokens under the same FLOP budget (Table 3).

The path-level control on sudoku and mazes remains the main contribution

Thank you for recognising our key contribution of path-level control on Sudoku and mazes. We would also like to highlight that our evidence goes beyond the maze, Sudoku and safety experiments: the multilingual experiment also shows clear evidence of interpretable routing on real text. Regarding W4, the goal of the paper is not to outperform existing models on efficiency or benchmarks, but to show that LoL remains competitive while adding an important and novel aspect, controllability and interpretability, that the reviewer acknowledges. Controllability is shown on Sudoku and mazes, and interpretability on multilingual data as well. We also compared against many relevant baselines, including those suggested by the reviewers: per-token routing, per-context routing, MoM and MoE. Overall, we contribute a new architecture, new insights and new ideas while remaining competitive.

We respectfully disagree with the reviewer conclusion and we believe that our paper insights are more than what is summarized by the reviewer, we present an axis on which you can pretrain models and have control on them at inference time. We believe that our idea could be the beginning of many future work in this direction.

Reviewer_iD91OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Library of Layers (LoL): Adaptively Ordering Transformer Layers at Runtime

This paper introduces Library of Layers a Transformer architecture that dynamically selects and orders complete Transformer blocks from a shared library during inference. Unlike Transformers, which run layers in a fixed order Library of Layers uses a learned router to decide which layer should be executed at each routed depth, based on the input.

The main design problem is how to keep causality while letting different parts of the input sequence follow execution paths. To solve this the authors create Strided Window Attention. This method splits the sequence into windows uses information from windows to make routing decisions and moves the window boundaries as the routing depth increases. This movement allows information to flow between the windows.

The size of the layer library is a control knob. A small library makes the model reuse parameters across routed depths. A larger library lets the model skip layers on a given pass creating more sparsity. The authors present Library of Layers as an architecture that links traditional dense Transformers, depth‑recurrent models and sparse mixture‑of‑experts approaches.

The paper tests Library of Layers in three scenarios. First language‑modeling experiments look at whether different languages trigger different layer execution patterns. Second synthetic Sudoku and maze tasks check whether routing paths match algorithmic strategies and whether a forced learned path can steer the model’s output strategy. Third language‑modeling experiments compare Library of Layers with Transformers looped Transformers and sparse mixture‑of‑experts models.

The results show that Library of Layers delivers language‑modeling loss and downstream benchmark performance at roughly 360 million and 1 billion parameters. In the tasks forcing strategy‑specific routing paths produces outputs that match the requested solving strategies with nearly 100 % accuracy. The paper ends by saying that dynamic layer ordering offers an alternative, for reusing parameters and doing sparse computation while giving researchers a way to explore and control a model’s internal computational pathways.

Critical strengths

Strength 1 : A flexible architectural approach to adaptive layer execution. The highlighting part is choosing and ordering Transformer blocks from a shared library which provides a potential valuable link between parameter reuse and sparse calculation. The proposed Strided Window Attention, a mechanism that solves an important technical challenge: preserving causal information flow while enabling input-dependent routing. Thus, the architecture provides a coherent solution to adaptive computation, instead of just adding a routing mechanism and ignoring how it interacts with sequence processing.

Strength 2 : Controllability is stated by Interpretable synthetic experiments .The Sudoku and maze experiments demonstrate that different algorithmic behaviors can be associated with different routing paths .The maze experiment felt interesting because evaluated strategies share the same output syntax but differ in their cell-exploration order. The reported success of mandatory routing methods indicates that intervention methods through execution path can impact observable model behavior. This makes the paper more convincing than an architecture paper that relies only on overall language modelling parameters.

Critical weaknesses

Weakness 1 : The strategy-specific routing fails to establish completely establish the interpretation of routing paths as internal algorithms.The synthetic experiments demonstrate that forcing representative routing paths can produce the intended solving behavior. However, we can see that router receives explicit information about the strategy being followed in the training phase, and the representative paths are directly derived from held-out cases. As a result, the controlled experiments are conclusive in proving controllability under the described conditions . More evidence is required to see how robust these results are under other routing paths, model seeds, and distinctions between random execution and forced pathways.

Weakness 2 : The paper lacks sufficient confidence in its claims regarding practical efficiency and scalability. While it mentions the performance of language models comparable to looped Transformers and MoE baselines, such comparisons do not show their practical advantages regarding inference efficiency. The MoE baseline has lower training loss in this comparison, while LoL has significantly more variance across random seeds in the looped-transformer experiments. Besides, the authors acknowledge routing instability and the small scale of their experiments. More explicit evidence on inference latency, throughput, memory usage, and scalability will make their claims about LoL practically advantageous compared to existing architecture more convincing.

Questions for authors

Question 1: The robustness and interpretation of routing paths By performing synthetic experiments, it is possible to derive representative routing paths and enforce those paths for evaluation. How sensitive is the adherence to the strategy with respect to the representative path and the training seed? Moreover, how often do we observe a difference between the forced path and the ultimate selected path for the exact input? The responses to these questions will help in understanding whether the obtained controllability is robust or it is specific to routing situations.

**Question 2:**Practical efficiency relative to existing architecture .The competitive language-modeling is demonstrated in the paper but failed to show the practical efficiency advantage of LoL. Do the authors have measurements of inference latency, throughput, and memory consumption for LoL compared with the matched looped-transformer and MoE baselines? If the primary benefit is architectural flexibility rather than faster inference, could the authors clarify this distinction?

Question 3: The paper reviews multiple approaches where routing can be done through shared experts or layers, but it does not clarify how unique the combination of full block routing and Strided Window Attention is in comparison with other similar existing methods. It would be useful if the authors provided a comparison and explained how their technique belongs to a new class of methods.

1 comment

Thanks again for your feedback. We made a strong effort to address all your points, including substantial edits to our manuscript, and we would greatly appreciate it if you would consider increasing your score accordingly. Do you have any other questions?

Reviewer_GtVeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LoL (Library of Layers), in which a fixed prelude and a fixed coda surround a “routed core” that can be dynamically scheduled. LoL makes layer selection, ordering, and the number of times each layer is used conditional on the input, introducing more flexibility of inference of a decoder-only transformer architecture. The authors apply the method to different domain and empirically show the LoL router learns input-adaptive patterns from data.

Critical strengths
  • The idea sounds interesting with an elegant and unifying design, in which formulating layer ordering as an input-dependent decision bridges looped transformers and MoEs via a single knob.
  • Experiments on several domains demonstrate the interpretability and controllability of the proposed method.
Critical weaknesses
  • The design choices behind the core mechanism may not be fully ablated. Like per-token routing vs stride window, window size and stride size, etc.
  • Attention within a window may harm the model's ability to handle long-term dependency. And the efficiency like the actual throughput of LoL is not fully discussed. How does dynamically ordering layers influence the efficiency of the model?
Questions for authors
  • Is there any evidence to support the claim: per-token routing "makes training unstable and harms performance"?
  • More analysis and comparison about model's efficiency and the ability to handle long-term dependency may be helpful to understand the effectiveness of the proposed method.
1 comment

Thanks again for your feedback. We made a strong effort to address all your points, including substantial edits to our manuscript, and we would greatly appreciate it if you would consider increasing your score accordingly. Do you have any other questions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.