- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Library of Layers (LoL), a decoder-only Transformer architecture whose routed core adaptively selects the next Transformer block from a globally shared library. By varying the library size relative to the number of executed core depths, the architecture supports both parameter-reuse and sparse-computation regimes. To accommodate different routing decisions across a sequence, LoL uses strided window attention: tokens within a window share an expert, routing depends on preceding windows, and window boundaries shift across depths. Full causal attention is retained in the fixed-order prelude and coda. The architecture also incorporates depth-dependent FiLM conditioning and, in some experiments, progressive depth growth during training.
Experiments investigate multilingual routing, strategy control on Sudoku and maze tasks, mathematical reasoning at the 1B-parameter scale, and comparisons with looped Transformers and mixture-of-experts models. The principal findings are that routing paths can encode controllable task-dependent behavior and that LoL achieves broadly competitive performance across the evaluated architectural regimes.
Critical strengths
1. A coherent architecture connecting parameter reuse and sparse computation.
Separating library size from execution depth provides a useful framework for studying parameter sharing and conditional computation within one model family. The comparisons span dense, looped, and sparse baselines rather than a single reference architecture. In particular, Figure 4 provides encouraging results against parameter-matched looped models, with evaluation over three random seeds. This supports the feasibility of learning flexible execution paths without necessarily sacrificing language-modeling quality.
2. The routing-intervention experiments provide meaningful evidence of controllability.
The synthetic experiments go beyond visualizing routing correlations: the authors extract strategy-specific paths and force those paths at inference after replacing the strategy description with neutral input. The maze experiment is particularly useful because the strategies share the same output syntax and differ in exploration order, reducing the possibility that the intervention merely selects a superficial formatting convention. The near-perfect strategy adherence is a substantive result, although its interpretation should remain tied to the controlled training setup.
Critical weaknesses
1. Central architectural choices are not reasonable enough and need stronger justification.
The arguments against first-token sequence-level routing and token-level routing lack controlled empirical comparisons. Attention-space mismatch and optimization instability should also be distinguished from causal correctness. Existing attention-mask comparisons do not establish the superiority of window-level routing or the need to route attention rather than retain fixed full causal attention and route only feed-forward blocks.
Natural-language experiments use only 1,024-token contexts, leaving long-range dependencies and sensitivity to window size and stride untested. Restricted attention in the routed core could limit long-context performance.
The globally shared library and depth-dependent FiLM need matched-parameter/FLOPs comparisons against depth-wise or stage-wise libraries and simpler or no depth conditioning. Observed neighboring-depth expert specialization makes these alternatives particularly relevant.
2. The benefits specifically attributable to adaptive ordering remain unclear.
Performance and baseline. Tables 1 and 4 demonstrate competitiveness rather than consistent gains relative to reported uncertainties, with full attention performing best on GSM8K. The large ARC-E improvement already achieved by the windowed baseline warrants investigation, especially since Figure 9 favors full attention in language-modeling loss. Exact token budgets, comparable optimization, and repeated training seeds are needed.
Efficiency. The early loss advantage is attributed to progressive depth growth rather than adaptive ordering alone, while the MoE baseline achieves slightly lower final training loss. FLOPs comparisons do not establish practical speedups; throughput, decoding latency, and memory measurements should account for routing overhead.
Questions for authors
Please refer to Weaknesses.
Additionally, I would like to ask the authors to discuss the connections between LoL and [1, 2], particularly regarding parameter sharing and model reuse in [1] and adaptive routing and composition of reusable computational blocks in [2]. Should these works be cited and as related work to clarify LoL’s distinctive contributions?
[1] Language Model Networks: Supervision-Efficient Learning through Dense Communication. ICML 2026
[2] Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning. ICLR 2018
2 comments
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns, particularly through the new ablation studies and comparisons with related architectures.
The new results address several of my earlier questions. However, the comparisons with MoM and UniPool primarily focus on language modeling performance and do not establish whether adaptive complete-block ordering provides distinct advantages over simpler routing alternatives in terms of controllability. In addition, while the interleaved full-attention experiment is useful, the effectiveness of the proposed architecture on longer contexts remains unverified.
Given these remaining concerns, I will maintain my original rating. Thank you again for the constructive discussion.
Thanks again for your feedback. We made a strong effort to address all your points, including substantial edits to our manuscript, and we would greatly appreciate it if you would consider increasing your score accordingly. Do you have any other questions?