Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design

Hybrid architectures combining Full Attention (FA) and Linear Attention (LA) are increasingly prominent, yet their allocation remains largely heuristic. We seek an evidence-grounded basis in the head-level functional organization learned by modern RoPE-based Transformers. Behavioral retrieval and local-streaming probes reveal useful tendencies but do not yield a complete taxonomy. We therefore propose two faithful intervention-based metrics: RoPE Frequency Importance Score (RFIS), which measures how each frequency contribution affects a head's complete attention distribution, and RoPE Positional Dependence (RPD), which isolates the effect of rotary positional modulation. Applied to Qwen3-series models and Llama3.1, RFIS suggests and RPD verifies a complete two-type taxonomy comprising retrieval and positional heads in RoPE Transformers, separated by a salient mid-low-frequency band. Controlled Transformers show that this functionally separating band follows the training-length positional scale; we term this mechanism-level boundary the Global Positional Band (GPBand). Its global positional dependence suggests a potential cause of zero-shot length-extrapolation failure and, together with the layer-specific head distribution, yields two design principles: (i) positional modeling should operate only locally, while global access should be implemented through position-independent retrieval; and (ii) retrieval and positional functions should be assigned at head granularity with layer-specific allocation. We instantiate these principles in the Head-wise Hybrid Architecture (HwH), using NoPE FA for global retrieval and LA for local positional modeling. With an overall FA-to-LA ratio less than 1:3, HwH retains strong language modeling and commonsense reasoning while improving retrieval and substantially strengthening zero-shot long-context extrapolation over Transformer, LA, and a 1:3 layer-wise hybrid baseline. Ablations validate both principles and the component roles, highlighting principled hybrid architecture design as a promising route toward future foundation models.