Decoupled Semantic Routing: Role-Specific Residual Stream Readout for Attention

Queries, keys, and values serve different purposes in attention, yet multi-stream Transformers typically construct all three from the same mixture of residual streams. We study whether they benefit from separate mixtures. A gradient probe on a trained Manifold-Constrained Hyper-Connections (mHC) model reveals aligned query and key preferences but opposing value preferences in deeper layers. Motivated by this observation, we introduce Decoupled Semantic Routing (DSR), which forms a separate stream mixture for each attention role. DSR computes the mixture weights in a low-dimensional space and applies them to the original hidden representations, using only streams at the current layer. On 1B and 3B language models trained for 1.64B tokens, DSR lowers validation loss relative to mHC by 0.00977 and 0.01830. Static mixtures and partial routing also retain the improvement, with several variants outperforming full dynamic routing. The results distinguish two choices in residual readout: which roles share a mixture, and whether that mixture varies across tokens. Separate mixtures remain effective even when their weights are static.