A Dual-Domain Vision Transformer with Spectral Positional Bias

Positional encoding in Vision Transformers presents a recurring trade-off: spatial relative-position tables (Swin) carry thousands of parameters per block and require ad-hoc interpolation to transfer across resolutions, while no-bias designs leave the attention logits geometry-agnostic. We propose a relative positional bias parameterized directly in the frequency domain. This parameterization yields three benefits over the standard spatial-table position encoding: (i) it expresses the same translation-equivariant bias with $3.25\times$ fewer parameters than Swin's spatial table at the same head count; (ii) it admits a closed-form, structurally exact transfer to a different patch grid, in contrast to bilinear interpolation of spatial tables which smooths the learned structure; and (iii) it shares the same parameterization style as a frequency-domain spectral mixer, enabling co-design of positional encoding and token mixing in a common substrate. We demonstrate the bias inside our **Dual-Domain Residual (DDR)** framework, in which a spectral mixer and a gated attention head act as parallel residuals within each block. Head-to-head, the frequency-domain bias recovers $+1.98$ pp over Swin RelPos and $+2.74$ pp over no positional bias; removing the entire attention branch (which carries the bias) drops accuracy by $5.35$ pp, while removing the spectral branch drops it by $2.22$ pp. On ImageNet-1K from scratch, DDR outperforms SpectFormer at every comparable parameter budget, by $+2.0$ pp at Tiny (DDR-Ti-SF, 10.7M, 78.9%) and $+0.4$ pp at Base (DDR-B-Deep, 63.1M, 82.5%); at sub-parity parameters, DDR-Ti+ (8.9M, 77.9%) exceeds SpectFormer-Ti (9.0M, 76.9%) by $+1.0$ pp. The learned per-layer gates concentrate spectral processing in early blocks and attention in late blocks, qualitatively matching the schedules that are hand-coded in the previous hybrid models.