- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Geometry-Aware Bilinear Attention (GABA) to improve the retrieval expressiveness and global–local feature integration of linear attention. The method introduces two main components: an output-channel-conditioned low-rank bilinear interaction that enables different output channels to use distinct query–key matching rules, and a contrast-guided anti-symmetric interaction that coordinates global and local representations while preserving their sum. The authors provide theoretical analyses and evaluate the proposed approach on image classification, object detection, image segmentation, and language modeling tasks.
Critical strengths
- A novel mechanism supported by theoretical analysis. The paper identifies a structural limitation arising from channel-shared query–key interactions in conventional linear attention and introduces an output-channel-conditioned low-rank bilinear retrieval mechanism. The theoretical analysis demonstrates that the proposed mechanism can enhance the representational capacity of the associative state while preserving linear complexity with respect to sequence length.
- Comprehensive and competitive experimental results. The proposed method is evaluated across multiple vision and language modeling tasks and compared with recent approaches, including RAVLT, MAViT, and MHLA. GABA achieves consistent improvements across different model scales. In particular, GABA-M1 outperforms strong baselines under comparable parameter and computational budgets.
Critical weaknesses
- Insufficient validation of the practical necessity of output-channel-conditioned bilinear interactions. The theoretical analysis establishes the expressiveness advantages of output-channel-conditioned bilinear interactions, while the ablation and representation analyses provide supporting empirical evidence. However, the paper does not sufficiently demonstrate which specific performance limitations in practical tasks arise from channel-shared query–key interactions. Moreover, direct comparisons with simpler alternatives, such as channel-wise feature scaling or diagonal feature reweighting, are missing. Consequently, it remains unclear how much additional practical benefit the proposed cross-dimensional bilinear interaction provides over these simpler designs.
- Insufficient justification for the anti-symmetric global–local interaction mechanism. The paper introduces an anti-symmetric update to coordinate global and local representations while preserving the sum of the two branches. However, it does not sufficiently explain why this sum-preserving constraint should improve downstream predictions. The current experiments mainly demonstrate the effectiveness of adding the AntiSym module, without direct comparisons against simpler alternatives such as single-branch updates or non-anti-symmetric two-branch updates. Therefore, it remains unclear whether the observed improvements specifically result from the anti-symmetric constraint itself.
Questions for authors
- Compared with simpler alternatives such as channel-wise feature scaling or diagonal feature reweighting, what specific practical advantages does the proposed output-channel-conditioned cross-dimensional bilinear interaction provide?
- Why should preserving the sum of the global and local representations improve downstream predictions? What specific advantages does the proposed anti-symmetric interaction offer over conventional single-branch updates or non-anti-symmetric two-branch updates?
4 comments
Response 2/2
W2/Q2: Why preserving the sum helps, and comparison with single-branch and non-anti-symmetric updates.
Why sum preservation should help. We see three reasons, which we will add to Sec. 2.2:
- It redistributes rather than injects. By Eq. 8, the fused output changes by A_{Q,K}(Δ) − Δ, which depends only on how differently the global and local pathways process the same displacement, and vanishes when the global aggregation leaves Δ unchanged. The exchange can therefore adjust the relative global–local geometry, but cannot add an independent residual signal to the fused representation.
- It keeps a stable anchor. The midpoint (V + L)/2 is fixed throughout training, so the output gate, projection, and residual path downstream always receive a representation whose combined content is unchanged. Non-sum-preserving updates shift V + L by Δ (single-branch) or 2Δ (same-sign) and add A(Δ) or A(Δ) + Δ to the output, so they can drift the fused representation away from what downstream layers expect.
- It is a strong inductive bias. A single displacement with a fixed opposite sign has fewer degrees of freedom than independent corrections to the two branches.
Direct comparison. We ran exactly the requested controls: all variants keep the same zero-initialized, contrast-guided shift generator (so all start identical to the model without a shift) and differ only in how the shift is applied (bilinear branch enabled in all rows; single runs due to the rebuttal time budget; GABA's seed std is 0.03).
Table R3 (all-linear Tiny).
| Update | Params (M) | FLOPs (G) | Online (%) | EMA (%) |
|---|---|---|---|---|
| None (V, L) | 14.873 | 2.545 | 81.84 | 81.90 |
| Single-branch global (V + Δ, L) | 14.900 | 2.554 | 81.55 | 81.62 |
| Single-branch local (V, L − Δ) | 14.900 | 2.554 | 81.52 | 81.59 |
| Same-sign (V + Δ, L + Δ) | 14.900 | 2.554 | 81.45 | 81.56 |
| Unconstrained (V + α_V Δ, L + α_L Δ) | 14.900 | 2.554 | 81.60 | 81.67 |
| Unconstrained (independent convs) | 14.927 | 2.563 | 81.39 | 81.46 |
| Anti-symmetric (GABA) | 14.900 | 2.554 | 82.03 | 82.10 |
The anti-symmetric exchange is the only update that improves on the model without a shift (+0.19/+0.20), and every alternative falls 0.43–0.64 points below it. The pattern is consistent with the reasoning above: all updates that change the sum V + L fall below even the no-shift model, suggesting that shifting the fused representation is harmful, while redistributing between the pathways is beneficial. Notably, the two unconstrained variants contain the anti-symmetric form as a special case (α_L = −α_V, or identical convolutions with opposite signs), yet training does not recover it, so the constraint acts as an inductive bias rather than a restriction that optimization would otherwise relax. Since these are single runs, we regard this explanation as consistent with the evidence rather than conclusive, and we will add seed-averaged results and drift diagnostics in the revision.
We appreciate the reviewer's thoughtful and constructive comments, which have substantially strengthened the experimental evidence. Following them, we added a channel-shared counterpart of the bilinear interaction at equal and matched parameter budgets, single-branch, same-sign, and unconstrained two-branch updates with the same shift generator, and an explanation of why sum preservation helps, which we will add to Sec. 2.2. If any further control or baseline would help, we would be glad to run it. If these additions address the reviewer's concerns, we would sincerely appreciate a reconsideration of the score.
We have uploaded a revised version of our paper with additional details and experiments, with quantitative and qualitative results addressing the reviewers' feedback. We welcome any further questions or suggestions and would be happy to provide additional clarification.
Thank you for the detailed response and additional experiments.
The three-seed results and shared-conditioning comparisons provide stronger evidence for the proposed bilinear mechanism. The new single-branch and unconstrained-update controls also address much of my concern about AntiSym.
My main reservation concerns the necessity of the cross-dimensional bilinear interaction. The shared-conditioning comparison demonstrates the benefit of output-channel conditioning, but does not fully establish its advantage over simpler alternatives, such as output-conditioned diagonal reweighting. Proposition D.2 concerns a transformation shared across output channels and does not rule out rank augmentation through channel-dependent diagonal transformations. The AntiSym results are encouraging, although the alternative-update comparisons are still based on single runs.
The additional experiments have strengthened the paper, but I remain concerned about the practical significance of the improvements, particularly given the measured inference overhead.
I would also appreciate it if the authors could make the revised PDF available as soon as possible for easier accessing.
For now, I will maintain my rating of 2 (Weak Rejection). That said, I consider the paper borderline and would not oppose its acceptance.
Response 1/2
We thank the reviewer for the thoughtful review. Both requested comparisons were also raised by other reviewers, and we ran them during the rebuttal; we report them here in full.
W1/Q1: Output-channel conditioning vs. simpler channel-wise alternatives.
Why channel-wise alternatives cannot provide the same benefit. Channel-wise feature scaling and diagonal reweighting, whether applied to the queries, the keys, or the output channels of the state, are channel-shared transformations: every output channel is retrieved through the same interaction. By Prop. D.2, such transformations act on the state as a common multiplication and can rescale or reorient existing state directions but cannot add new ones; for any diagonal D, rank(Q D S) ≤ rank(S) and rank(S D) ≤ rank(S). Output-channel conditioning removes this restriction: each output channel e uses its own interaction matrix M⁽ᵉ⁾, the family cannot be collapsed into a single shared projection (Prop. D.3), and the correction can contribute up to R new state directions (Prop. D.4), a bound that is attainable (Prop. D.7). The two designs also differ in what they weight: a diagonal reweighting only rescales coordinate-aligned products Q_ic K_jc, whereas M⁽ᵉ⁾ also weights cross-dimensional products Q_ic K_ja with c ≠ a.
Empirical comparison with a channel-shared counterpart. To test this directly, we tie the output conditioning (shared C), so that all output channels use one interaction matrix while everything else is unchanged. We test it at the same rank (R = 4) and at a parameter-matched rank (R = 6).
Table R2 (all-linear Tiny, AntiSym enabled in all rows; mean ± std over 3 seeds; rank metrics at layers.2.blocks.5 on 1,000 class-balanced validation images).
The practical limitation of channel-shared interaction. The table shows what a shared interaction fails to deliver in practice. Consistent with Prop. D.2, the shared variant does not raise the numerical rank of the state above the no-branch model, whereas channel conditioning does, and the effective rank of the retrieved output follows the same order as accuracy across all four variants. At identical parameters and fewer FLOPs, channel conditioning improves accuracy by +0.20 (online) and +0.23 (EMA) over the parameter-matched shared variant, while the shared variant recovers only +0.06 to +0.11 over no bilinear branch, and enlarging its rank from 4 to 6 does not help. The practical benefit therefore comes from the output-channel conditioning itself, not from the added capacity. At the representation level, GABA also lowers channel redundancy relative to RALA (Table 11).
We will add an explicit diagonal channel-reweighting control to Appendix F in the revision.