acceptodds
Under review as a conference paper at ICLR 2027

Improved Belief-Attention in Vision Tasks

Abstract

Recently, Belief-Attention was proposed to improve Transformer performance by orthogonally projecting the softmax-weighted sum of vectors relative to the original vectors and retaining the perpendicular component as a residual signal. In this paper, we first present an ablation study demonstrating that the projected component also encodes valuable token correlation information that should not be overlooked. Based on this insight, we propose Belief2-Attention, an extension that effectively leverages both components. Specifically, the projected component passes through an activation function and a linear mapping before being merged with the target token—effectively forming an internal two-layer feedforward network (FFN) within the attention mechanism. Furthermore, while standard attention models token correlations solely via the inner product , we modify this formulation to to enrich pairwise token representations for vision tasks. This allows to capture symmetric correlations, leaving to model complementary directional dependencies. We demonstrate that Belief2-Attention possesses greater expressive capacity than standard attention under mild conditions. Finally, we validate the effectiveness of Belief2-Attention on vision tasks, including image classification and image segmentation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
orthogonal projection, belief-attention, belief2-attention

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QiCmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Belief2-Attention, an extension of Belief-Attention for vision tasks. The method retains both the perpendicular and projected components obtained from the orthogonal projection used in Belief-Attention, and processes the projected component through an activation function and an additional linear mapping. The paper also replaces the standard attention logits QK⊤QK^\top with (Q+K)K⊤(Q+K)K^\top, motivated as a way to jointly capture directional and symmetric token correlations. Experiments on ADE20K, ImageNet, and CIFAR-100 report consistent improvements over standard attention and the original Belief-Attention.

Critical strengths
  1. The paper investigates an interesting limitation of the original Belief-Attention formulation. The ablation results indicate that the projected component discarded by Belief-Attention still contains useful information, motivating its explicit reuse in Belief2-Attention.

  2. The empirical improvements are consistent across several vision settings. Belief2-Attention improves over standard attention on ADE20K segmentation, ImageNet classification, and CIFAR-100 classification while maintaining similar parameter counts and relatively similar training cost.

Critical weaknesses
  1. The claimed additional expressiveness of the (Q+K)K⊤(Q+K)K^\top attention formulation is not convincing. Since Q=XWQQ=XW_Q and K=XWKK=XW_K, the proposed logits can be rewritten as (Q+K)K⊤=X(WQ+WK)WK⊤X⊤,(Q+K)K^\top=X(W_Q+W_K)W_K^\top X^\top, which is exactly a standard Q′K⊤Q'K^\top attention score with WQ′=WQ+WKW'_Q=W_Q+W_K. Therefore, the added KK⊤KK^\top term does not appear to enlarge the function class of the attention scores; it is essentially a reparameterization of the query projection. This weakens the paper’s motivation that (Q+K)K⊤(Q+K)K^\top provides more degrees of freedom or richer token-correlation modeling than standard QK⊤QK^\top. The observed empirical improvement may instead arise from optimization effects induced by this parameterization, which would lead to a substantially different interpretation of the method.

  2. The experiments do not fully isolate whether the improvement comes from the proposed projection mechanism or simply from reallocating nonlinear capacity. Belief2-Attention adds an activation and an additional linear mapping for the projected component, effectively introducing an internal FFN, while the conventional FFNs are reduced to maintain overall parameter parity. Thus, the comparison changes both the attention mechanism and the distribution of nonlinear capacity within the Transformer. A stronger control would compare against standard attention with an equivalently sized additional nonlinear/gated branch or a similarly redistributed FFN budget. Without such controls, it is difficult to attribute the gains specifically to the belief/projection formulation.

Questions for authors
  1. How do the authors reconcile the claimed additional expressiveness of (Q+K)K⊤(Q+K)K^\top with the fact that it can be rewritten as standard attention using WQ′=WQ+WKW'_Q=W_Q+W_K? If the two formulations have the same attention-score function class, is the intended benefit primarily an optimization or parameterization effect rather than richer token-correlation modeling?

  2. Can the authors provide a parameter- and compute-matched control that adds comparable nonlinear capacity to standard attention without using the orthogonal projection? Such a comparison would help determine whether the gains arise specifically from processing the projected component or simply from relocating FFN capacity into the attention block.

2 comments

1. How do the authors reconcile the claimed additional expressiveness of (Q+K)KT(Q+K)K^T with the fact that it can be rewritten as standard attention using WQ′=WQ+WKW_Q'=W_Q+W_K ? If the two formulations have the same attention-score function class, is the intended benefit primarily an optimization or parameterization effect rather than richer token-correlation modeling?

Many thanks for the comment. It is a very insightful question.

(a) To verify whether this is due to an optimization effect, we conducted an additional experiment with the Belief2-Attention QKTQK^T setup on CIFAR-100. Before training, we initialized WQ′=WQ+WKW_Q' = W_Q + W_K as inspired by Reviewer QiCM. The experimental results are shown below. As can be seen, this initialization strategy does not preserve the performance gains achieved by the (Q+K)KT(Q+K)K^T setup.

Belief2-Attention w. (Q+K)KT(Q+K)K^TBelief2-Attention (QKTQK^T) w. initialized WQ′=WQ+WKW_Q'=W_Q+W_K
75.90$\pm$0.5073.72$\pm$0.35

The above analysis indicates that we need to enforce either (Q+K)KT(Q+K)K^T or equivalently WQ′=WQ+WKW_Q' = W_Q + W_K across training iterations to maintain the performance gains. If we do so, our motivation that KKTKK^T captures symmetric token-correlation and QKTQK^T captures complementary token-correlation holds true.

(b) To further investigate the impact of the KKTKK^T term in Belief2-Attention, we trained two ViT models with Belief2-Attention on CIFAR-100, where the first model employs QKTQK^T and the second uses (Q+K)KT(Q+K)K^T. We then fed a batch of images to the two models and computed the entropies H(softmax(QKT))H(\textrm{softmax}(QK^T)) and H(softmax((Q+K)KT))H(\textrm{softmax}((Q+K)K^T)) per Transformer block.

Fig.~8 in the revision shows the scatter plots of the entropies. We observe that on average, the entropy H(softmax((Q+K)KT))H(\textrm{softmax}((Q+K)K^T)) is smaller than H(softmax(QKT))H(\textrm{softmax}(QK^T)) across the three considered Transformer blocks. This suggests that the formulation softmax((Q+K)KT)\textrm{softmax}((Q+K)K^T) sharpens the distribution, allowing tokens with high correlation to contribute more while suppressing those with low correlation.

2. Can the authors provide a parameter- and compute-matched control that adds comparable nonlinear capacity to standard attention without using the orthogonal projection? Such a comparison would help determine whether the gains arise specifically from processing the projected component or simply from relocating FFN capacity into the attention block.

The question is insightful. By following the reviewer's suggestions, we also conducted additional experiments on CIFAR-100. In particular, we trained two new ViT models. In the first model (referred to as attention with an internal FFN), we introduce an internal two-layer FFN to the standard attention, where the output of the new attention layer is X←X+MH(X)W+gelu⁡(V(X))WfX \leftarrow X + MH(X)W + \operatorname{gelu}(V(X))W^f. In the second model (referred to as Belief2-Attention (QKT)(QK^T) by setting α=1\alpha=1 in internal FFN), we deliberately set α=1\alpha=1 in the internal FFN of Belief2-Attention. Consequently, the output becomes X←X+Δ(X)W+gelu⁡(V(X))WpX \leftarrow X + \Delta(X)W + \operatorname{gelu}(V(X))W^p.

attentionattention w. internal FFNBelief2-Attention (QKTQK^T)Belief2-Attention (QKTQK^T) by setting α=1\alpha=1 in internal FFN
72.36$\pm$0.1371.79$\pm$0.1973.81$\pm$0.2472.97$\pm$0.15

The experimental results are summarized in the table above. It is clear that both new models result in degraded performance. We can conclude from the table that naively introducing a two-layer FFN into the standard attention layer is insufficient and harms results. Furthermore, we can conclude that the gains of Belief2-Attention (QKTQK^T) indeed arise from the coefficient α\alpha obtained via orthogonal projection.

We will incorporate all of the above new results into the final revision to further clarify the impact of Belief2-Attention (these updates are not yet reflected in the current manuscript). We welcome any further comments or questions.

We understand that the reviewer is busy. However, the discussion period is running short, and about six days have passed since we posted our rebuttal. We have tried to respond to the reviewer's constructive comments sincerely and carefully, and we sincerely hope for a constructive exchange of views with each other. We would greatly appreciate it if the reviewer could let us know whether our responses have mitigated the concerns or point out any remaining issues, while there is still time to address them.

Reviewer_M7pZOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Belief2-Attention, aiming to enhance the performance of Belief-Attention in visual tasks. Through ablation studies, the authors found that the projected components discarded by the original method still contain useful information. Therefore, they applied these components, after being passed through an activation function and a linear mapping, together with the orthogonal components to update the word elements. Additionally, the attention scores in the paper were changed from QK⊤QK^\top to (Q+K)K⊤(Q+K)K^\top to combine symmetry and directionality. The experiments covered CIFAR100, ImageNet image classification, and ADE20K image segmentation. Under the condition of comparable parameter quantities, the proposed method outperformed the standard attention and the original Belief-Attention.

Critical strengths
  • The motivation for retaining the projection component is quite clear. The author first demonstrated through ablation that this component has classification value, and then proposed to handle the projection component separately from the orthogonal component. The experimental observations and method design have a relatively direct connection.
  • The method modification is relatively simple, and it reduces the total parameter quantity of the traditional FFN. This enables the performance improvement not to be solely based on increasing the model parameters, reflecting the consideration of fairness.
Critical weaknesses
  • Firstly, the criteria for choosing between global and head-wise decomposition are not clear enough. The original Belief-Attention uses both global and head-wise orthogonal components, but Equation (14) is modified to only handle the two components obtained from the global decomposition. Head-wise decomposition can also retain both orthogonal and projection parts, but the existing ablation studies do not compare these two options. It is recommended to explain the reason for choosing the global decomposition and provide a comparison with a parameter budget that is comparable.
  • Secondly, the explanations for symmetry and direction correlation are lacking direct support. KK⊤KK^\top has a symmetrical form, but QK⊤QK^\top does not rule out symmetrical components. Tables 5 and 6 show that adding this term can improve performance, but they do not explain whether the two items after training respectively perform the functions described by the authors. The authors are hoped to clarify whether this is a design intuition or a mechanism explanation supported by model analysis.
Questions for authors
  • Why choose global rather than head-by-head projection decomposition? What are the differences in performance between the two versions under the same parameter budget?
  • Are there any analyses that can support that the trained KK⊤KK^\top and QK⊤QK^\top respectively model symmetric and direction-related correlations, rather than just providing an interpretation of the formula form?
  • ADE20K uses identity activation, while classification tasks use GELU or SiLU. What considerations led to this choice? Did the authors compare the performance of non-linear activations in segmentation tasks?
4 comments

Why choose global rather than head-by-head projection decomposition? What are the differences in performance between the two versions under the same parameter budget?

Many thanks for the comment. In the revision, we have included evaluations of both Belief2-Attention-G and Belief2-Attention-L, where G and L denote global and local projections, respectively. Additional experiments indicate that Belief2-Attention-L consistently outperforms Belief2-Attention-G across both image segmentation and classification tasks (see Table 1, 2, and 3 in the revised manuscript).

Are there any analyses that can support that the trained and respectively model symmetric and direction-related correlations, rather than just providing an interpretation of the formula form?

Many thanks for the comment. Reviewer~QiCm also raises a similar question, asking if (Q+K)KT(Q+K)K^T can be alternatively realized by reparameterization of the weight matrix WQ′=WQ+WKW_Q' = W_Q + W_K, where WQW_Q and WKW_K are the weight matrices for computing QQ and KK vectors, respectively.

(1) We conducted an additional experiment with the Belief2-Attention QKTQK^T setup on CIFAR-100. Before training, we initialized WQ′=WQ+WKW_Q' = W_Q + W_K. The experimental results are shown below. As can be seen, this initialization strategy does not preserve the performance gains achieved by the (Q+K)KT(Q+K)K^T formulation.

Belief2-Attention w. (Q+K)KT(Q+K)K^TBelief2-Attention (QKTQK^T) w. initialized WQ′=WQ+WKW_Q'=W_Q+W_K
75.90$\pm$0.5073.72$\pm$0.35

The above analysis indicates that we need to enforce either (Q+K)KT(Q+K)K^T or equivalently WQ′=WQ+WKW_Q' = W_Q + W_K across training iterations to maintain the performance gains. If we do so, our motivation that KKTKK^T captures symmetric token-correlation and QKTQK^T captures complementary token-correlation holds true.

(2) To further investigate the impact of the KKTKK^T term in Belief2-Attention-G, we trained two ViT models with Belief2-Attention-G on CIFAR-100, where the first model employs QKTQK^T and the second uses (Q+K)KT(Q+K)K^T. We then fed a batch of images to the two models and computed the entropies H(softmax(QKT))H(\textrm{softmax}(QK^T)) and H(softmax((Q+K)KT))H(\textrm{softmax}((Q+K)K^T)) per Transformer block.

Fig.~8 in the revision shows the scatter plots of the entropies. We observe that on average, the entropy H(softmax((Q+K)KT))H(\textrm{softmax}((Q+K)K^T)) is smaller than H(softmax(QKT))H(\textrm{softmax}(QK^T)) across the three considered Transformer blocks. This suggests that the formulation softmax((Q+K)KT)\textrm{softmax}((Q+K)K^T) sharpens the distribution, allowing tokens with high correlation to contribute more while suppressing those with low correlation.

ADE20K uses identity activation, while classification tasks use GELU or SiLU. What considerations led to this choice? Did the authors compare the performance of non-linear activations in segmentation tasks?

Many thanks for the comment. We agree with the reviewer that it is of great interest to find out how non-linear activation affect the performance of the segmentation task. We have performed an additional experiment by using GELU in the segmentation task for Belief2-Attention-G, where GG denotes global projection. As shown in the table below, it is found that the obtained validation mIoU with GELU is comparable to that that with the identity function.

Belief2-Attention w. GELUBelief2-Attention w. Identity
val mIoU38.8238.95

We understand that the reviewer is busy. However, the discussion period is running short, and about six days have passed since we posted our rebuttal. We have tried to respond to the reviewer's constructive comments sincerely and carefully, and we sincerely hope for a constructive exchange of views with each other. We would greatly appreciate it if the reviewer could let us know whether our responses have mitigated the concerns or point out any remaining issues, while there is still time to address them.

Reviewer_M7pZOpenReview ↗

Official Comment by Reviewer M7pZ

Thank you for your response. After carefully reviewing the rebuttal and taking the discussion into account, I am afraid I still have to maintain my original score. I will continue to discuss this with the Area Chair.

Many thanks for the reply.

We really appreciate if the reviewer could elaborate more on the remaining concerns of the paper to give us an opportunity to clarify them.

Thank you and we look forward to your detailed comments.

Reviewer_ZTCdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper conducts an ablation study of the earlier proposed Belief-Attention mechanism for transformers based on orthogonalized residual connection with respect to the V vector in the attention output. In addition, the paper introduces an extended realization coined Belief2-Attention by extending the softmax inner product QKTQK^T to (Q+K)KT(Q+K)K^T and adding a 2-layer FFN component to further process the projected component. Both frameworks are studied in their application to visual transformers. The Belief-Attention ablation study is performed on CIFAR100 classification task using a standard ViT. The experiments with Belief2-Attention include image segmentation on ADE20K with JiT-B, image classification on ImageNet-1K and CIFAR100 and an ablation study on CIFAR100 with standard ViT. The results of these experiments demonstrate a modest improvement in accuracy and IoU (for segmentation task) with a little computational overhead compared to the baseline.

Critical strengths

The proposed Belief2-Attention mechanism is an interesting novel extension of the standard attention, which is easily applicable and demonstrates a potential to improve the performance of visual transformers with limited computational overhead.

The language and the style of the paper is very clear, concise and easy to follow.

Critical weaknesses

The main weaknesses of the paper are related to its empirical limitations.

  1. All the experiments in the paper are underdescribed. It is not very clear which exact ViT realizations were used in the experiments in Sec. 2.2., 4.3, and 4.4. I assume that all of the transformers used across all of the experiments described in the paper are trained from scratch, although it would be nice to clarify if the used transformers are untrained or pre-trained. The hyperparameters (e.g. learning rates, batch sizes, number of epochs, etc.) and other experimental settings used in each study (image preprocessing, optimizers, LR schedule, warmup, regularization, etc.) are mainly underreported.

  2. Each experiment is performed only on one type of model at a time. It would be beneficial to use at least two different types of models for each experiment, preferably with different model sizes. Recovering a performance of a given baseline on a given dataset reported in the literature would also be a plus.

  3. Even with a one type of model for each experiment the methodology seems insufficient. The results with several seeds are reported for only a few experiments. The model tuning seem to lack hyperparameter scans on the same search space and budget (or at least it is not reported in the paper). It would be useful if the report of such scan included the results of different runs with different hyperparameters (and seeds) in Appendix, not only the run with the best performance.

  4. I am not quite convinced that the results of Belief-Attention ablation study (Fig. 3 and the discussion in Section 2.2) prove that the orthogonal projection component carries critical (additional) information. As far as I understand, both the projected and perpendicular components by construction include the cross-token information through the scalar α\alpha (or β\beta). The construction with V vectors as residuals does not include it and predictably fails. The difference in validation accuracy between the standard attention and Belief-Attention seems to be marginal and does not constitute a rigorous proof in my opinion, taking into account the concerns I raised above in this section.

Questions for authors

In addition to the points raised in "Critical Weaknesses" section I have the following questions:

  1. What was the motivation for using JiT-B for image segmentation task? As the authors mention themselves this model is designed for image generation. While the core attention mechanism is the same as in the standard ViT, it would be nice to comment why a standard ViT (or a different model commonly adopted for image segmentation) was not used.
7 comments

All the experiments in the paper are underdescribed. It is not very clear which exact ViT realizations were used in the experiments in Sec. 2.2., 4.3, and 4.4.

Many thanks for the comment. We have updated the manuscript, describing how the hyperparameters were set in all four experiments (see Appendix~C in the revision). We have also clarified in the main paper that the segmentation task fine-tunes a pretrained ViT model while image classification tasks train the ViT models from scratch.

Each experiment is performed only on one type of model at a time. It would be beneficial to use at least two different types of models for each experiment, preferably with different model sizes.

Many thanks for the comment. Following the reviewer's suggestion, we have conducted additional experiments on CIFAR-100. We trained ViT models with 9 and 12 Transformer blocks. The validation accuracy is summarized in the table below. The suffix 'G' in Belief2-Attention-G denotes the global projection variant. It is clear that Belief2-Attention-G outperforms standard attention for both 9 and 12 Transformer blocks.

AttentionBelief2-Attention-G
12 blocks (new)72.83$\pm$ 0.2276.16$\pm$0.36
9 blocks (from the paper)72.36$\pm$ 0.3175.90$\pm$ 0.50

We will also test other model sizes for the task of image segmentation and image classification on ImageNet and update the paper accordingly in the final version.

Regarding the hyperparameter search

Many thanks for the comment. In fact, when conducting the experiments, we first swept several candidate values to identify the optimal initial learning rate for both the baseline model and Belief-Attention during the initial 5 training epochs (see Appendix~C). For image segmentation and image classification on CIFAR-100, the optimal initial learning rates matched those provided in the open-source repositories. For image classification on ImageNet, the optimal initial learning rate for both the baseline model and Belief-Attention was 0.0010.001. We then used these optimal task-specific learning rates to evaluate the baseline model, Belief-Attention, and Belief2-Attention, of which the results are presented in the main paper.

In addition to the above hyperparameter search, we also tested a non-optimal initial learning rate of 5×10−45 \times 10^{-4} for both the image segmentation task and the image classification task on CIFAR-100. The table below summarizes the validation performance for the two tasks. We observe that even with a non-optimal initial learning rate, Belief2-Attention-G still outperforms standard attention.

AttentionBelief2-Attention-G
val mIoU for segmentation37.3437.77
val accuracy for classification70.86$\pm$ 0.4074.05$\pm$0.10

Regarding more random seeds in the experiment

We have conducted additional experiments on the segmentation task with two new random seeds. Table~1 in the revision is updated to reflect the overall performance. Again it is found that Belief2-Attention performs better than Attention and Belief-Attention.

We will also test other random seeds for the task of image classification on ImageNet and update the paper accordingly in the final version.

I am not quite convinced that the results of Belief-Attention ablation study (Fig. 3 and the discussion in Section 2.2) prove that the orthogonal projection component carries critical (additional) information. As far as I understand, both the projected and perpendicular components by construction include the cross-token information through the scalar α\alpha(or β\beta). The construction with V vectors as residuals does not include it and predictably fails.

Many thanks for the comment. We agree with the reviewer that the scalar α\alpha (or β\beta) should, in principle, carry cross-token information. However, without empirical investigation, it remains unclear whether this cross-token information is significant. The experimental results in Fig.~3 and Section 2 show that by using only the projected component as the residual signal, we can achieve reasonable validation performance.

The difference in validation accuracy between the standard attention and Belief-Attention seems to be marginal and does not constitute a rigorous proof in my opinion, taking into account the concerns I raised above in this section.

Many thanks for the comment. While Fig. 3 provides a qualitative overview where the visual margins may appear tight, Table~3 in the paper details the exact numerical values. Specifically, validation accuracy improves from 72.36±0.1272.36\pm 0.12 to 74.18±0.2574.18\pm 0.25. Given the minimal architectural overhead introduced by our modification, we consider this consistent improvement to be meaningful.

Building upon Belief-Attention, we propose Belief2-Attention in this work to advance performance even further, raising validation accuracy from 74.18±0.2574.18\pm 0.25 to 75.90±0.5075.90\pm 0.50.

What was the motivation for using JiT-B for image segmentation task? As the authors mention themselves this model is designed for image generation. While the core attention mechanism is the same as in the standard ViT, it would be nice to comment why a standard ViT (or a different model commonly adopted for image segmentation) was not used.

Thank you for the helpful comment. To clarify, our experiments with the JiT-B model are strictly for image classification on ImageNet rather than segmentation.

We selected JiT-B because its standard ViT-based attention mechanism provides a clean, well-established framework for evaluating our proposed modifications, aligning with our parallel research efforts. We agree with the reviewer that its core attention design matches standard ViT.

I thank the authors for their detailed response and for conducting additional experiments and modifying the text according to my requests.

I greatly appreciate their efforts, however my concerns have been addressed only partially. In particular, Table 2 in the revised version of the manuscript does not provide any variance on the validation accuracy.

In addition, regarding the concern about using only one type of model: I admire that the authors have conducted an additional experiment with a ViT model that has a different number of layers, but what I envisioned was using a different type of visual transformer, such as DeiT or a transformer with self-supervised pretraining (e.g. DINO), at least for one of the experiments, e.g. image classification.

I realize there is little time left in the discussion period, however I am maintaining my original score for now.

Comments on the paper

2 comments

Dear Reviewers,

Please carefully review the authors’ rebuttal and engage in further discussion with the authors to clarify any remaining concerns and questions. Your active participation during the discussion period is greatly appreciated.

Best, AC

Dear Associate Editor,

We apologize for the interruption and appreciate your time. We would be grateful for your support in encouraging a discussion with reviewers.

More than five days have passed since we posted our rebuttal comments, and we have not yet received their responses. We believe we have addressed all critical concerns raised by the reviewers.

Thank you very much for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.