Mixture of Layers with Hybrid Attention: Parallel Thin Blocks for Sparse Transformer Compute

We propose parallel attention processing with a new block-decomposed model, Mixture of Layers (MoL). Comparing with MoE models, the routing in MoL operates with parallel attention blocks instead of experts inside one block. MoL uses $K$ parallel thin blocks of width $thin\ll model$, connected to the full block width $model$ by learned down and up projections, and a top-$k$ routing that selects which of these blocks are processing which token. Sparse routing over multiple blocks degrade attention quality, as each block is processing fewer tokens. To address this, we introduce a hybrid attention design, with one shared block processing all input tokens using dense softmax attention (holds global context), and Gated DeltaNet linear attention in the routed blocks handling sparse token subsets. We scaled MoL to 2.08B (0.63B active) on FineWeb-Edu and found that the model outperforms an iso-active dense baseline by 0.17 PPL, while it underperforms a 1.3B dense baseline by 3.21 PPL and trains about $2\times$ slower. Evaluation on 8-task lm-eval-harness shows scores following PPL results, and scores model between the two dense baselines. We tested single-GPU prefill on A100, H100 SXM and H200 GPUs. At 128K context it is $1.2\times$ faster than the 1.3B dense model on A100, and up to $1.5\times$ faster at 256K tokens on H100 SXM and H200, while short-context prefill and decode remain slower.