GABA: Geometry-Aware Bilinear Attention for Linear Transformers

Linear attention makes global context tractable in transformers by replacing dense pairwise attention with a compressed $K^{\top }V$ state. Recent work has improved what this state encodes through rank augmentation, gating, or multiple local summaries, but the readout remains a coordinate-aligned query-key contraction whose interaction geometry is shared across output channels, limiting output-dependent cross-channel interactions and the attainable representation rank. A second, orthogonal gap concerns fusion: the value representation feeding global aggregation and its locally enhanced counterpart evolve independently and meet only at the output, leaving their relative geometry unconstrained. We introduce Geometry-Aware Bilinear Attention (GABA), which addresses both within a single associative block. An output channel-conditioned low-rank bilinear refinement allows different output channels to realize distinct query–key geometries and contributes complementary state directions. A contrast-guided anti-symmetric interaction displaces the global and local pathways equally and oppositely about their shared midpoint; although their sum is preserved, subsequent branch-specific transformations induce a non-trivial refinement. The all-linear formulation retains linear sequence complexity while keeping model-level FLOPs essentially unchanged. GABA consistently outperforms strong linear-attention baselines across model scales; the hybrid model achieves $82.03\%$ Top-1 test accuracy on ImageNet-1K, outperforming the second-closest baseline by $0.50$, as shown in Fig. 1.