Mechanics of Long-Context Hybrid Models 1.1: From Hybrid Attention to Hybrid Position

The architectural design of Large Language Models (LLMs) is shifting from traditional full-attention-only models to hybrid models, which combine different attention modules to improve long-context efficiency and performance in length extrapolation and context extension. To explain why hybrid models work and how to design them better, we propose **Mechanics of Long-Context Hybrid Models**. As Part 1.1 of this series, we begin with hybrids of full attention and either sliding-window attention (SWA) or gated variants of linear attention (LA), represented by GLA and GDN. We first observe a **Seesaw Effect in Context Extension**: SWA hybrids perform better in length extrapolation, while LA hybrids perform better after context extension. We interpret this from the perspective of hybrid position. We find that SWA hybrids suffer from a **Short-Context Learning Trap**, **Short-Window Weariness**, and **Long-Window Laziness**, requiring extended windows to enhance performance in continual long-context pretraining. In contrast, we summarize the **Matthew Effect of Hybrid Position Extrapolation** and propose **Sliding-Window Linear Attention** for LA hybrids, achieving up to 16$\times$ training-free length extrapolation with 100% retrieval accuracy on NIAH-SK1.