FUSED BUT NOT RELATIVE: LOW-RANK ATTENTION BIASES SILENTLY DROP TRANSLATION INVARIANCE

A rank-$R$ additive attention bias written as $B\approx {\varphi }_q{\varphi }_k^{\top }$, with ${\varphi }_q,{\varphi }_k\in {\mathbb{R}}^{N\times R}$, can be folded into the queries and keys, allowing compatible unmodified FlashAttention kernels to compute biased attention without materializing the $N\times N$ bias. Alternative fused solutions require bias-specific kernel support or compilation. Low-rank factorization supports fused training but does not guarantee relativity: ${\varphi }_q(i{)}^{\top }{\varphi }_k(j)$ may depend on $(i,j)$ rather than only on $i-j$. % Prior work proposes learning these factors from initialization but does not evaluate this variant; its reported instantiations use closed forms, factorize a pretrained table, or fit an existing bias, thereby enforcing or inheriting relativity. When trained freely in vision, only $37.2\%$ of the learned bias energy remains displacement-dependent, compared with $100\%$ for the unfactorized relative bias it replaces. % We restore relativity by generating factors from learned frequencies, whose inner product reduces to ${\sum }_k{a}_k{b}_k\cos ({\mathbf{w}}_k^{\top }({\mathbf{p}}_i-{\mathbf{p}}_j))$ and therefore depends only on displacement by algebraic identity. A learned phase lifts the resulting even-symmetry restriction, still depending only on displacement, without increasing factor width. Kernel and computational cost are unchanged, while positional parameters become independent of window size and context length; in Swin-T at window $32$, they fall from $1,277,952$ to $4,416$. % At matched factor width, architecture, and training recipe, restoring relativity reduces language-model perplexity degradation at $16\times$ the training context by roughly $7\times$ across ranks $16$–$128$, while using $683\times$ fewer positional parameters and improving in-distribution perplexity. In vision, it improves transfer to semantic segmentation and detection, and out-of-domain generalization at the larger window, while the phase variant strengthens robustness at both. These results provide a practical route to learned relative biases within fused attention across vision, language, and other position-structured domains.