Symmetric Attention Improves Optimization Geometry for Image Restoration

Self-attention has become the workhorse of Transformer-based image restoration, yet its optimization behavior remains largely underexplored. We identify two structural issues arising from the standard asymmetric score matrix $S=Q{K}^{\top }$: (i) the gradients with respect to $W_Q$ and $W_K$ are governed by structurally different operators, allowing the two projections to drift along uncoupled directions and creating flat valleys in the loss landscape; (ii) the asymmetric score matrix may admit complex eigenvalues, whose imaginary components induce rotational dynamics during gradient descent. Together, these effects lead to ill-conditioned optimization and unstable training. Our main contribution, the Balanced Kernel (BK), resolves both issues through a lightweight symmetric score construction that couples the gradient dynamics of $W_Q$ and $W_K$ and guarantees a real eigenvalue spectrum. BK introduces negligible additional FLOPs while achieving $\sim 3\times$ faster convergence, $>4\times$ oscillation reduction, and a substantially more compact optimization basin. When integrated into existing architectures such as Restormer and Uformer, BK consistently improves performance, demonstrating largely architecture-agnostic optimization benefits. We further introduce ChannelRouter, which generates instance-adaptive head weights from degradation-aware channel statistics, and Gated Feature Modulation (GFM), which enables spatially adaptive processing through multiplicative gating. Integrating these components, our Balformer achieves state-of-the-art performance on deraining, deblurring, and denoising benchmarks.