Improved Belief-Attention in Vision Tasks

Recently, Belief-Attention was proposed to improve Transformer performance by orthogonally projecting the softmax-weighted sum of $V$ vectors relative to the original $V$ vectors and retaining the perpendicular component as a residual signal. In this paper, we first present an ablation study demonstrating that the projected component also encodes valuable token correlation information that should not be overlooked. Based on this insight, we propose Belief2-Attention, an extension that effectively leverages both components. Specifically, the projected component passes through an activation function and a linear mapping before being merged with the target token—effectively forming an internal two-layer feedforward network (FFN) within the attention mechanism. Furthermore, while standard attention models token correlations solely via the inner product $Q{K}^{\top }$, we modify this formulation to $(Q+K)K^{\top }$ to enrich pairwise token representations for vision tasks. This allows $K{K}^{\top }$ to capture symmetric correlations, leaving $Q{K}^{\top }$ to model complementary directional dependencies. We demonstrate that Belief2-Attention possesses greater expressive capacity than standard attention under mild conditions. Finally, we validate the effectiveness of Belief2-Attention on vision tasks, including image classification and image segmentation.