- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Belief2-Attention, an extension of Belief-Attention for vision tasks. The method retains both the perpendicular and projected components obtained from the orthogonal projection used in Belief-Attention, and processes the projected component through an activation function and an additional linear mapping. The paper also replaces the standard attention logits with , motivated as a way to jointly capture directional and symmetric token correlations. Experiments on ADE20K, ImageNet, and CIFAR-100 report consistent improvements over standard attention and the original Belief-Attention.
Critical strengths
-
The paper investigates an interesting limitation of the original Belief-Attention formulation. The ablation results indicate that the projected component discarded by Belief-Attention still contains useful information, motivating its explicit reuse in Belief2-Attention.
-
The empirical improvements are consistent across several vision settings. Belief2-Attention improves over standard attention on ADE20K segmentation, ImageNet classification, and CIFAR-100 classification while maintaining similar parameter counts and relatively similar training cost.
Critical weaknesses
-
The claimed additional expressiveness of the attention formulation is not convincing. Since and , the proposed logits can be rewritten as which is exactly a standard attention score with . Therefore, the added term does not appear to enlarge the function class of the attention scores; it is essentially a reparameterization of the query projection. This weakens the paper’s motivation that provides more degrees of freedom or richer token-correlation modeling than standard . The observed empirical improvement may instead arise from optimization effects induced by this parameterization, which would lead to a substantially different interpretation of the method.
-
The experiments do not fully isolate whether the improvement comes from the proposed projection mechanism or simply from reallocating nonlinear capacity. Belief2-Attention adds an activation and an additional linear mapping for the projected component, effectively introducing an internal FFN, while the conventional FFNs are reduced to maintain overall parameter parity. Thus, the comparison changes both the attention mechanism and the distribution of nonlinear capacity within the Transformer. A stronger control would compare against standard attention with an equivalently sized additional nonlinear/gated branch or a similarly redistributed FFN budget. Without such controls, it is difficult to attribute the gains specifically to the belief/projection formulation.
Questions for authors
-
How do the authors reconcile the claimed additional expressiveness of with the fact that it can be rewritten as standard attention using ? If the two formulations have the same attention-score function class, is the intended benefit primarily an optimization or parameterization effect rather than richer token-correlation modeling?
-
Can the authors provide a parameter- and compute-matched control that adds comparable nonlinear capacity to standard attention without using the orthogonal projection? Such a comparison would help determine whether the gains arise specifically from processing the projected component or simply from relocating FFN capacity into the attention block.
2 comments
We understand that the reviewer is busy. However, the discussion period is running short, and about six days have passed since we posted our rebuttal. We have tried to respond to the reviewer's constructive comments sincerely and carefully, and we sincerely hope for a constructive exchange of views with each other. We would greatly appreciate it if the reviewer could let us know whether our responses have mitigated the concerns or point out any remaining issues, while there is still time to address them.
1. How do the authors reconcile the claimed additional expressiveness of (Q+K)KT with the fact that it can be rewritten as standard attention using WQ′=WQ+WK ? If the two formulations have the same attention-score function class, is the intended benefit primarily an optimization or parameterization effect rather than richer token-correlation modeling?
Many thanks for the comment. It is a very insightful question.
(a) To verify whether this is due to an optimization effect, we conducted an additional experiment with the Belief2-Attention QKT setup on CIFAR-100. Before training, we initialized WQ′=WQ+WK as inspired by Reviewer QiCM. The experimental results are shown below. As can be seen, this initialization strategy does not preserve the performance gains achieved by the (Q+K)KT setup.
The above analysis indicates that we need to enforce either (Q+K)KT or equivalently WQ′=WQ+WK across training iterations to maintain the performance gains. If we do so, our motivation that KKT captures symmetric token-correlation and QKT captures complementary token-correlation holds true.
(b) To further investigate the impact of the KKT term in Belief2-Attention, we trained two ViT models with Belief2-Attention on CIFAR-100, where the first model employs QKT and the second uses (Q+K)KT. We then fed a batch of images to the two models and computed the entropies H(softmax(QKT)) and H(softmax((Q+K)KT)) per Transformer block.
Fig.~8 in the revision shows the scatter plots of the entropies. We observe that on average, the entropy H(softmax((Q+K)KT)) is smaller than H(softmax(QKT)) across the three considered Transformer blocks. This suggests that the formulation softmax((Q+K)KT) sharpens the distribution, allowing tokens with high correlation to contribute more while suppressing those with low correlation.
2. Can the authors provide a parameter- and compute-matched control that adds comparable nonlinear capacity to standard attention without using the orthogonal projection? Such a comparison would help determine whether the gains arise specifically from processing the projected component or simply from relocating FFN capacity into the attention block.
The question is insightful. By following the reviewer's suggestions, we also conducted additional experiments on CIFAR-100. In particular, we trained two new ViT models. In the first model (referred to as attention with an internal FFN), we introduce an internal two-layer FFN to the standard attention, where the output of the new attention layer is X←X+MH(X)W+gelu(V(X))Wf. In the second model (referred to as Belief2-Attention (QKT) by setting α=1 in internal FFN), we deliberately set α=1 in the internal FFN of Belief2-Attention. Consequently, the output becomes X←X+Δ(X)W+gelu(V(X))Wp.
The experimental results are summarized in the table above. It is clear that both new models result in degraded performance. We can conclude from the table that naively introducing a two-layer FFN into the standard attention layer is insufficient and harms results. Furthermore, we can conclude that the gains of Belief2-Attention (QKT) indeed arise from the coefficient α obtained via orthogonal projection.
We will incorporate all of the above new results into the final revision to further clarify the impact of Belief2-Attention (these updates are not yet reflected in the current manuscript). We welcome any further comments or questions.