acceptodds
Under review as a conference paper at ICLR 2027

GABA: Geometry-Aware Bilinear Attention for Linear Transformers

Abstract

Linear attention makes global context tractable in transformers by replacing dense pairwise attention with a compressed state. Recent work has improved what this state encodes through rank augmentation, gating, or multiple local summaries, but the readout remains a coordinate-aligned query-key contraction whose interaction geometry is shared across output channels, limiting output-dependent cross-channel interactions and the attainable representation rank. A second, orthogonal gap concerns fusion: the value representation feeding global aggregation and its locally enhanced counterpart evolve independently and meet only at the output, leaving their relative geometry unconstrained. We introduce Geometry-Aware Bilinear Attention (GABA), which addresses both within a single associative block. An output channel-conditioned low-rank bilinear refinement allows different output channels to realize distinct query–key geometries and contributes complementary state directions. A contrast-guided anti-symmetric interaction displaces the global and local pathways equally and oppositely about their shared midpoint; although their sum is preserved, subsequent branch-specific transformations induce a non-trivial refinement. The all-linear formulation retains linear sequence complexity while keeping model-level FLOPs essentially unchanged. GABA consistently outperforms strong linear-attention baselines across model scales; the hybrid model achieves Top-1 test accuracy on ImageNet-1K, outperforming the second-closest baseline by , as shown in Fig. 1.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Linear attention, Bilinear attention, Vision Transformer, Representation rank

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zh7vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Geometry-Aware Bilinear Attention (GABA) to improve the retrieval expressiveness and global–local feature integration of linear attention. The method introduces two main components: an output-channel-conditioned low-rank bilinear interaction that enables different output channels to use distinct query–key matching rules, and a contrast-guided anti-symmetric interaction that coordinates global and local representations while preserving their sum. The authors provide theoretical analyses and evaluate the proposed approach on image classification, object detection, image segmentation, and language modeling tasks.

Critical strengths
  1. A novel mechanism supported by theoretical analysis. The paper identifies a structural limitation arising from channel-shared query–key interactions in conventional linear attention and introduces an output-channel-conditioned low-rank bilinear retrieval mechanism. The theoretical analysis demonstrates that the proposed mechanism can enhance the representational capacity of the associative state while preserving linear complexity with respect to sequence length.

  1. Comprehensive and competitive experimental results. The proposed method is evaluated across multiple vision and language modeling tasks and compared with recent approaches, including RAVLT, MAViT, and MHLA. GABA achieves consistent improvements across different model scales. In particular, GABA-M1 outperforms strong baselines under comparable parameter and computational budgets.
Critical weaknesses
  1. Insufficient validation of the practical necessity of output-channel-conditioned bilinear interactions. The theoretical analysis establishes the expressiveness advantages of output-channel-conditioned bilinear interactions, while the ablation and representation analyses provide supporting empirical evidence. However, the paper does not sufficiently demonstrate which specific performance limitations in practical tasks arise from channel-shared query–key interactions. Moreover, direct comparisons with simpler alternatives, such as channel-wise feature scaling or diagonal feature reweighting, are missing. Consequently, it remains unclear how much additional practical benefit the proposed cross-dimensional bilinear interaction provides over these simpler designs.

  1. Insufficient justification for the anti-symmetric global–local interaction mechanism. The paper introduces an anti-symmetric update to coordinate global and local representations while preserving the sum of the two branches. However, it does not sufficiently explain why this sum-preserving constraint should improve downstream predictions. The current experiments mainly demonstrate the effectiveness of adding the AntiSym module, without direct comparisons against simpler alternatives such as single-branch updates or non-anti-symmetric two-branch updates. Therefore, it remains unclear whether the observed improvements specifically result from the anti-symmetric constraint itself.
Questions for authors
  1. Compared with simpler alternatives such as channel-wise feature scaling or diagonal feature reweighting, what specific practical advantages does the proposed output-channel-conditioned cross-dimensional bilinear interaction provide?

  1. Why should preserving the sum of the global and local representations improve downstream predictions? What specific advantages does the proposed anti-symmetric interaction offer over conventional single-branch updates or non-anti-symmetric two-branch updates?
4 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for the thoughtful review. Both requested comparisons were also raised by other reviewers, and we ran them during the rebuttal; we report them here in full.

W1/Q1: Output-channel conditioning vs. simpler channel-wise alternatives.

Why channel-wise alternatives cannot provide the same benefit. Channel-wise feature scaling and diagonal reweighting, whether applied to the queries, the keys, or the output channels of the state, are channel-shared transformations: every output channel is retrieved through the same interaction. By Prop. D.2, such transformations act on the state as a common multiplication and can rescale or reorient existing state directions but cannot add new ones; for any diagonal D, rank(Q D S) ≤ rank(S) and rank(S D) ≤ rank(S). Output-channel conditioning removes this restriction: each output channel e uses its own interaction matrix M⁽ᵉ⁾, the family cannot be collapsed into a single shared projection (Prop. D.3), and the correction can contribute up to R new state directions (Prop. D.4), a bound that is attainable (Prop. D.7). The two designs also differ in what they weight: a diagonal reweighting only rescales coordinate-aligned products Q_ic K_jc, whereas M⁽ᵉ⁾ also weights cross-dimensional products Q_ic K_ja with c ≠ a.

Empirical comparison with a channel-shared counterpart. To test this directly, we tie the output conditioning (shared C), so that all output channels use one interaction matrix while everything else is unchanged. We test it at the same rank (R = 4) and at a parameter-matched rank (R = 6).

Table R2 (all-linear Tiny, AntiSym enabled in all rows; mean ± std over 3 seeds; rank metrics at layers.2.blocks.5 on 1,000 class-balanced validation images).

Bilinear variantRParams (M)FLOPs (G)Online (%)EMA (%)Num. rank of S̃r_eff of Y_bil
None–14.8652.54681.76 ± 0.0281.81 ± 0.0459.80 ± 0.076.70 ± 0.05
Shared C414.8882.55481.87 ± 0.0681.92 ± 0.0759.76 ± 0.107.07 ± 0.04
Shared C (param-matched)614.9002.55981.83 ± 0.0781.87 ± 0.0559.67 ± 0.086.98 ± 0.03
Channel-conditioned (GABA)414.9002.55482.03 ± 0.0382.10 ± 0.0360.10 ± 0.067.50 ± 0.05

The practical limitation of channel-shared interaction. The table shows what a shared interaction fails to deliver in practice. Consistent with Prop. D.2, the shared variant does not raise the numerical rank of the state above the no-branch model, whereas channel conditioning does, and the effective rank of the retrieved output follows the same order as accuracy across all four variants. At identical parameters and fewer FLOPs, channel conditioning improves accuracy by +0.20 (online) and +0.23 (EMA) over the parameter-matched shared variant, while the shared variant recovers only +0.06 to +0.11 over no bilinear branch, and enlarging its rank from 4 to 6 does not help. The practical benefit therefore comes from the output-channel conditioning itself, not from the added capacity. At the representation level, GABA also lowers channel redundancy relative to RALA (Table 11).

We will add an explicit diagonal channel-reweighting control to Appendix F in the revision.

AuthorsOpenReview ↗

Response 2/2

W2/Q2: Why preserving the sum helps, and comparison with single-branch and non-anti-symmetric updates.

Why sum preservation should help. We see three reasons, which we will add to Sec. 2.2:

  • It redistributes rather than injects. By Eq. 8, the fused output changes by A_{Q,K}(Δ) − Δ, which depends only on how differently the global and local pathways process the same displacement, and vanishes when the global aggregation leaves Δ unchanged. The exchange can therefore adjust the relative global–local geometry, but cannot add an independent residual signal to the fused representation.
  • It keeps a stable anchor. The midpoint (V + L)/2 is fixed throughout training, so the output gate, projection, and residual path downstream always receive a representation whose combined content is unchanged. Non-sum-preserving updates shift V + L by Δ (single-branch) or 2Δ (same-sign) and add A(Δ) or A(Δ) + Δ to the output, so they can drift the fused representation away from what downstream layers expect.
  • It is a strong inductive bias. A single displacement with a fixed opposite sign has fewer degrees of freedom than independent corrections to the two branches.

Direct comparison. We ran exactly the requested controls: all variants keep the same zero-initialized, contrast-guided shift generator (so all start identical to the model without a shift) and differ only in how the shift is applied (bilinear branch enabled in all rows; single runs due to the rebuttal time budget; GABA's seed std is 0.03).

Table R3 (all-linear Tiny).

UpdateParams (M)FLOPs (G)Online (%)EMA (%)
None (V, L)14.8732.54581.8481.90
Single-branch global (V + Δ, L)14.9002.55481.5581.62
Single-branch local (V, L − Δ)14.9002.55481.5281.59
Same-sign (V + Δ, L + Δ)14.9002.55481.4581.56
Unconstrained (V + α_V Δ, L + α_L Δ)14.9002.55481.6081.67
Unconstrained (independent convs)14.9272.56381.3981.46
Anti-symmetric (GABA)14.9002.55482.0382.10

The anti-symmetric exchange is the only update that improves on the model without a shift (+0.19/+0.20), and every alternative falls 0.43–0.64 points below it. The pattern is consistent with the reasoning above: all updates that change the sum V + L fall below even the no-shift model, suggesting that shifting the fused representation is harmful, while redistributing between the pathways is beneficial. Notably, the two unconstrained variants contain the anti-symmetric form as a special case (α_L = −α_V, or identical convolutions with opposite signs), yet training does not recover it, so the constraint acts as an inductive bias rather than a restriction that optimization would otherwise relax. Since these are single runs, we regard this explanation as consistent with the evidence rather than conclusive, and we will add seed-averaged results and drift diagnostics in the revision.

We appreciate the reviewer's thoughtful and constructive comments, which have substantially strengthened the experimental evidence. Following them, we added a channel-shared counterpart of the bilinear interaction at equal and matched parameter budgets, single-branch, same-sign, and unconstrained two-branch updates with the same shift generator, and an explanation of why sum preservation helps, which we will add to Sec. 2.2. If any further control or baseline would help, we would be glad to run it. If these additions address the reviewer's concerns, we would sincerely appreciate a reconsideration of the score.

We have uploaded a revised version of our paper with additional details and experiments, with quantitative and qualitative results addressing the reviewers' feedback. We welcome any further questions or suggestions and would be happy to provide additional clarification.

Thank you for the detailed response and additional experiments.

The three-seed results and shared-conditioning comparisons provide stronger evidence for the proposed bilinear mechanism. The new single-branch and unconstrained-update controls also address much of my concern about AntiSym.

My main reservation concerns the necessity of the cross-dimensional bilinear interaction. The shared-conditioning comparison demonstrates the benefit of output-channel conditioning, but does not fully establish its advantage over simpler alternatives, such as output-conditioned diagonal reweighting. Proposition D.2 concerns a transformation shared across output channels and does not rule out rank augmentation through channel-dependent diagonal transformations. The AntiSym results are encouraging, although the alternative-update comparisons are still based on single runs.

The additional experiments have strengthened the paper, but I remain concerned about the practical significance of the improvements, particularly given the measured inference overhead.

I would also appreciate it if the authors could make the revised PDF available as soon as possible for easier accessing.

For now, I will maintain my rating of 2 (Weak Rejection). That said, I consider the paper borderline and would not oppose its acceptance.

Reviewer_KNnyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces GABA, combining output-channel-conditioned low-rank bilinear retrieval with an anti-symmetric exchange between global and local branches. It provides a state-rank gain bound, sufficient conditions for strict rank increase, and a construction that attains the bound. Experiments report improvements in image classification, detection, and segmentation, with additional small-scale language experiments evaluating only the bilinear component.

Critical strengths

Clear motivation. The paper identifies two concrete limitations: shared query–key geometry across output channels and limited coordination between global and local branches. The proposed readout and anti-symmetric exchange directly address these issues.

Broad evaluation with clear limitations. Experiments cover classification, detection, segmentation, and language modeling, with consistent reported gains across ImageNet and ADE20K model scales. The authors acknowledge the single-run ablations and the lack of long-sequence efficiency measurements.

Critical weaknesses

W1. No measured efficiency. The efficiency argument relies on operation counts, with no throughput, latency, or memory measurements. Yet the authors expect some added operations to be bandwidth-bound (p. 25). The conclusion’s “at most 0.33%” overhead applies only to the bilinear branch, excluding the anti-symmetric exchange and M2 gate (p. 28). Please report runtime and memory against RAVLT, with a breakdown by component.

W2. The language results are difficult to verify. Table 5 shows small gains over MHLA, but the 390M architecture and the source of each baseline result are unclear. These experiments test only the bilinear component (p. 8, l. 427–431), so they do not validate the full GABA method. Please provide model configurations, identify cited and rerun baselines, and release raw evaluation outputs.

W3. Limited statistical support on ImageNet. GABA-M1 improves over the strongest reproduced baseline at each scale by 0.25–0.39 percentage points, without results across seeds. The reproduced RAVLT/MAViT baselines also fall 0.24–0.49 points below their original results, although MHLA reports similar reproduced values. Please report several seeds at the Tiny scale and clarify the baseline protocol, including EMA and stochastic depth.

W4. The ablations do not isolate the proposed geometry. Tables 8 and 9 test module inclusion and bilinear rank only on the all-linear Tiny model. Comparisons with extra heads or width under similar parameter and compute budgets would help separate the proposed structure’s benefit from added capacity. The anti-symmetric exchange also needs controls that retain the shift generator but use single-branch or same-sign updates. Without these controls, the causal interpretation in the conclusion seems too strong.

W5. The abstract misattributes its headline result. The reported 82.03% accuracy and +0.50-point gain match the all-linear Tiny ablation in Table 8, using non-EMA weights and RALA as the baseline. The abstract instead assigns them to a hybrid model; Table 1 reports 82.80% and 82.95% for the hybrid Tiny variants.

Questions for authors

Q1. How do GABA-M1 and GABA-M2 compare with RAVLT in measured throughput, latency, and peak memory at 224 × 224 and longer sequence lengths? How much overhead does each component add?

Q2. Which baselines in Table 5 did you rerun, and which results come from the MHLA paper? Could you also provide the 390M model configuration?

Q3. Do the Table 1 gains hold across multiple seeds, at least for Tiny? Could differences in EMA evaluation, stochastic depth, or other settings explain why the reproduced RAVLT and MAViT results fall below their original values?

Q4. How does the bilinear branch compare with extra heads or wider projections under similar parameter and compute budgets? With the same shift generator, how does the proposed opposite-sign exchange compare with single-branch or same-sign updates? These controls would help clarify whether the gains come from the proposed geometry.

6 comments
AuthorsOpenReview ↗

Response 1/4

We thank the reviewer for the detailed review and the concrete requests. Several of the requested controls overlap with those requested by another reviewer; we report them here in full so that this response is self-contained.

W1/Q1: Measured efficiency and per-component overhead. We agree, and we first correct our accounting: the "at most 0.33%" in the conclusion refers to the bilinear branch alone. Including the anti-symmetric exchange (a 3×3 depthwise convolution, 9ND multiply–accumulates per block), GABA-M1 remains below 1% of model FLOPs and adds 0.062 M parameters at Tiny; the M2 gate adds substantially more (Appendix E). We will correct the conclusion.

We measured inference cost for RAVLT and for each component added separately, under identical conditions.

Table R5: Measured inference cost with per-component breakdown (Tiny, 224²). Single GCD of an AMD Instinct MI250X, PyTorch 2.9.1, ROCm 6.4, fp32, synthetic input; throughput at batch 128, latency at batch 1 (median); 50 iterations after 20 warmup.

ModelAdded params (M)Throughput (img/s)Latency (ms)Peak mem (GB)
RAVLT_L–728.219.541.84
+ exchange only+0.026615.823.562.31
+ bilinear only+0.035682.622.922.03
GABA-M1_L+0.062594.426.402.42
GABA-M2_L+1.98552.428.492.42
RAVLT_NL–732.416.941.84
GABA-M1_NL+0.008629.619.922.42
GABA-M2_NL+1.93584.621.522.42

The breakdown shows where the cost arises. The measured cost is higher than the operation count alone would suggest, mainly because of the anti-symmetric exchange, whose element-wise operations and full-width intermediates at the highest-resolution stage are memory-bound in the current unfused implementation; batch-1 latency in a small model is additionally dominated by the number of kernel launches. The bilinear branch is the lighter of the two components, and the M2 gate adds no peak memory over M1. The overhead is lower in the hybrid models of Table 1, and it does not increase with sequence length: in additional measurements at 1024² (65,536 stage-1 tokens), the relative overhead remained at a similar level for the all-linear model and decreased markedly for the hybrid models.

GABA's aim is to improve the accuracy of linear attention while preserving linear complexity, not to reduce wall-clock cost, and it achieves consistent gains at every scale and across seeds. Our conclusion already lists the cost of the M2 gate and the open question of efficiency at long sequences among the limitations; we will extend that paragraph with these measurements, add Table R5 and the higher-resolution results to Appendix E, and state that the efficiency claims refer to parameters and operation counts.

AuthorsOpenReview ↗

Response 2/4

W3/Q3: Seeds and baseline protocol.

Seeds. We trained RAVLT and GABA on the all-linear Tiny backbone with three independent seeds each (seeds 0/1/2, shared between the two models). We chose the all-linear setting because it isolates the proposed mechanism without Softmax blocks.

Table R1: Seed robustness (all-linear Tiny). Mean ± std over 3 runs; Δ computed per run.

ModelOnline Top-1 (%)EMA Top-1 (%)
RAVLT_L81.52 ± 0.0481.68 ± 0.03
GABA_L82.03 ± 0.0382.10 ± 0.03
Δ (per run)+0.50 ± 0.07 (min +0.44)+0.42 ± 0.06 (min +0.37)

GABA outperforms RAVLT in every run, and the smallest per-run gain is about ten times the run-to-run std of either model. We will add seeds for the hybrid Tiny models of Table 1 in the revision.

Baseline protocol. RAVLT and MAViT were reproduced with their official repositories, each with its own released training configuration, without modification. GABA uses the RAVLT configuration: 300 epochs, AdamW (peak learning rate 10⁻³, weight decay 0.05), global batch 1024, cosine decay with 5 warmup epochs, DeiT augmentation and regularization, EMA with decay 0.99996, and stochastic depth rates of 0.1/0.1/0.4/0.55 for Tiny/Small/Base/Large. All entries marked * and all GABA entries in Tables 1–2 report the best-epoch accuracy of the EMA weights, evaluated identically for all models. The full configurations are available in the anonymized repository linked in the paper.

Since the recipes are unchanged, the gap to the published values does not stem from training settings, and we attribute it to differences in the training environment (hardware and library versions). As the reviewer notes, MHLA independently reports reproduced values similar to ours, which indicates that the published numbers are not reproduced by third parties either. Because all models in our comparison share the same reproduction setting, our claims are relative to it.

AuthorsOpenReview ↗

Response 3/4

W4/Q4: Isolating the proposed geometry from added capacity.

(a) Exchange controls with the same shift generator. We ran exactly the requested controls: all variants keep the same zero-initialized, contrast-guided shift generator and differ only in how the shift is applied (bilinear branch enabled in all rows; single runs due to the rebuttal time budget; GABA's seed std is 0.03).

Table R3 (all-linear Tiny).

UpdateParams (M)FLOPs (G)Online (%)EMA (%)
None (V, L)14.8732.54581.8481.90
Single-branch global (V + Δ, L)14.9002.55481.5581.62
Single-branch local (V, L − Δ)14.9002.55481.5281.59
Same-sign (V + Δ, L + Δ)14.9002.55481.4581.56
Unconstrained (V + α_V Δ, L + α_L Δ)14.9002.55481.6081.67
Unconstrained (independent convs)14.9272.56381.3981.46
Opposite-sign (GABA)14.9002.55482.0382.10

The opposite-sign exchange is the only update that improves on the model without a shift (+0.19/+0.20). Single-branch and same-sign updates with the identical generator fall 0.43–0.58 points below it, and both unconstrained variants, which contain the opposite-sign form as a special case, also fall short.

(b) Structure vs. capacity for the bilinear branch. Three results indicate that the gain does not come from added capacity:

  • Parameter-matched control. Tying the output conditioning (shared C), which by Prop. D.2 cannot add state directions, at a parameter-matched rank (R = 6; 14.900 M params, more FLOPs than GABA) reaches 81.83 ± 0.07 / 81.87 ± 0.05, versus 82.03 ± 0.03 / 82.10 ± 0.03 for GABA (3 seeds each). Its numerical state rank does not rise above the no-branch model (59.67 vs. 59.80), whereas GABA's does (60.10).
  • Rank sweep. Increasing the bilinear rank to R = 8 and R = 16 (2× and 4× the added parameters) lowers accuracy (Table 9).
  • M2. GABA-M2 adds about 1.9 M parameters (+13%) over M1 but gains only +0.11 Top-1 (Table 2).

The bilinear branch adds only 0.035 M parameters (0.24%). A width- or head-matched control at this budget would change the backbone by well under 1%; the parameter-matched shared-conditioning control above tests the same question while keeping everything else fixed. We will add an explicit width-matched control in the revision.

(c) Causal wording. We agree that the conclusion overstates causality and will rephrase it as "consistent with," citing the controls above.

Reviewer_STM5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Geometry-Aware Bilinear Attention (GABA), which augments associative linear attention with an output-channel-conditioned low-rank bilinear readout and a contrast-guided anti-symmetric exchange between global and local value pathways. The bilinear component admits an equivalent compressed-state refinement, with an attainable algebraic rank increase bounded by the factorization rank. The paper distinguishes an economical M1 variant from an M2 variant containing an additional gated token-wise correction, and evaluates both all-linear and hybrid backbones. Experiments cover image classification, dense prediction, and a language-modeling adaptation using the bilinear component.

Critical strengths
  1. The associative refinement is technically coherent and carefully scoped. Equations 2–3 introduce output-channel-dependent query–key interactions without constructing a token-pair attention matrix.

  2. The empirical improvements extend beyond hybrid attention and the more expensive M2 variant. Table 2 reports all-linear M1 improvements of 0.41/0.40/0.39/0.37 percentage points over the reproduced RALA baseline from Tiny to Large, with closely matched parameter and operation counts. Table 4 additionally reports M1 improvements under both segmentation decoders. Reporting M1 separately from M2 is valuable because it exposes the additional cost of the gated token-specific correction and provides evidence that the gains do not depend entirely on that correction.

Critical weaknesses
  1. The evidence does not sufficiently establish that the specific geometric constraints explain the improvements. In Table 8, the individual mechanisms improve EMA accuracy by only 0.16 and 0.21 percentage points, and all entries are single training runs. The paper acknowledges this limitation, but online/EMA agreement and persistence across checkpoints are correlated observations from the same runs. They do not establish robustness across independent training repetitions. Furthermore, the ablation does not compare output-channel conditioning against shared conditioning, or the anti-symmetric exchange against a comparably sized one-sided or unconstrained value correction. Table 10 demonstrates an immediate spectral change within a trained model, but cannot establish that these particular constraints are responsible for the downstream accuracy improvement. This distinction matters because the proposed geometric interpretation is a central contribution.

  2. The practical accuracy–efficiency advantage over the closest methods remains insufficiently demonstrated. Appendix E provides thorough arithmetic accounting and acknowledges bandwidth-sensitive operations, but there are no measured latency, throughput, or peak-memory comparisons. Consequently, the small operation-count overhead does not yet establish a similarly small practical overhead. The reproduced performance frontier also requires clarification: Table 1 reports MAViT accuracies of 82.42/84.33/85.40/85.64, whereas the original MALA paper reports 82.9/84.7/85.7/86.0. Standardized retraining is useful, but these differences are comparable to the claimed gains. Without reconciling the configurations and measuring actual execution costs, the improvement over the existing accuracy–efficiency frontier remains uncertain.

Questions for authors
  1. Can you provide independent repetitions of the principal component ablation, together with focused controls using shared output conditioning and a comparably sized alternative to the anti-symmetric exchange? Do the proposed constraints retain an advantage?

  2. What architectural or training differences explain the reproduced RAVLT/MAViT results relative to their published results? Please specify which baseline operations, including the query–value refinement in Equation 9, are shared across the Table 8 variants.

  3. Can you provide measured accuracy–latency and peak-memory comparisons for M1 and M2 against the closest baselines, including a higher-token-count setting? These measurements would materially clarify the practical value of the proposed lightweight refinement.

6 comments
AuthorsOpenReview ↗

Response 1/4

We thank the reviewer for the careful and constructive critique. Below we address both concerns with new experiments.

Weakness 1: whether the geometric constraints explain the improvements

We agree that single runs, and agreement between online and EMA evaluation of the same run, do not establish robustness across independent training repetitions, and that Table 10 alone does not link the spectral change to accuracy. We therefore ran three new controlled experiments, all on the all-linear Tiny backbone with the identical 300-epoch schedule.

(a) Seed robustness. We trained RALA and GABA with three independent seeds each (runs 1/2/3 use seeds 0/1/2, shared between the two models).

Table R1: Seed robustness (all-linear Tiny). Mean ± std over 3 runs (seeds 0/1/2, shared between models); Δ computed per run.

VariantRunOnline Top-1 (%)EMA Top-1 (%)
RALA1 / 2 / 381.53 / 81.48 / 81.5681.69 / 81.65 / 81.71
Mean ± std81.52 ± 0.0481.68 ± 0.03
GABA1 / 2 / 382.03 / 82.05 / 82.0082.10 / 82.13 / 82.08
Mean ± std82.03 ± 0.0382.10 ± 0.03
ΔMean ± std+0.50 ± 0.07+0.42 ± 0.06
Min / Max+0.44 / +0.57+0.37 / +0.48

GABA outperforms RALA in every run. The smallest per-run gain (+0.37 EMA) exceeds the run-to-run std of either model (≤ 0.04) by roughly an order of magnitude.

(b) Output-channel conditioning vs. shared conditioning. We tie the columns of C so that every output channel uses the same interaction matrix M. By Prop. D.2, this variant provably cannot increase the rank of the state, so it isolates the role of output-channel conditioning. We test it at the same rank (R = 4) and at a parameter-matched rank (R = 6).

Table R2 (AntiSym enabled in all rows; mean ± std over 3 seeds; rank metrics at layers.2.blocks.5 on 1,000 class-balanced validation images, protocol of Sec. H.10; for "None," they refer to S and Y_base).

Bilinear variantRParams (M)FLOPs (G)Online (%)EMA (%)Num. rank of S̃r_eff of Y_bil
None–14.8652.54681.76 ± 0.0281.81 ± 0.0459.80 ± 0.076.70 ± 0.05
Shared C414.8882.55481.87 ± 0.0681.92 ± 0.0759.76 ± 0.107.07 ± 0.04
Shared C (param-matched)614.9002.55981.83 ± 0.0781.87 ± 0.0559.67 ± 0.086.98 ± 0.03
Channel-conditioned (GABA)414.9002.55482.03 ± 0.0382.10 ± 0.0360.10 ± 0.067.50 ± 0.05
  • Accuracy. At identical parameters and fewer FLOPs, GABA exceeds the parameter-matched shared variant by +0.20 (online) and +0.23 (EMA), about 3–5× the larger seed std. Shared conditioning gains only +0.06 to +0.11 over no bilinear branch, and raising its rank from 4 to 6 does not help.
  • State rank. Consistent with Prop. D.2, the numerical rank of S̃ under shared conditioning stays within seed variation of the no-branch model, whereas channel conditioning raises it.
  • Effective rank. The effective rank of the retrieved output follows the same order as accuracy across all four variants.
AuthorsOpenReview ↗

Response 2/4

(c) Anti-symmetric exchange vs. alternative value corrections. With the channel-conditioned bilinear branch enabled in all rows, we keep the same zero-initialized, contrast-guided displacement generator and vary only how the displacement is applied. These are single runs (seed 0) due to the rebuttal time budget; for reference, GABA's seed std is 0.03 (Table R1).

Table R3 (all-linear Tiny, single run per variant).

Value correctionUpdateParams (M)FLOPs (G)Online (%)EMA (%)
None (+Bilinear only)V, L14.8732.54581.8481.90
One-sided globalV + Δ, L14.9002.55481.5581.62
One-sided localV, L − Δ14.9002.55481.5281.59
Same-signV + Δ, L + Δ14.9002.55481.4581.56
Unconstrained (decoupled scale)V + α_V Δ, L + α_L Δ14.9002.55481.6081.67
Unconstrained (independent conv)V + Δ_V, L + Δ_L14.9272.56381.3981.46
Anti-symmetric (GABA)V + Δ, L − Δ14.9002.55482.0382.10

The anti-symmetric exchange is the only correction that improves on the bilinear-only model (+0.19 online, +0.20 EMA). Every alternative falls 0.43–0.64 points below GABA, more than ten times GABA's seed std. This includes two unconstrained variants that contain the anti-symmetric form as a special case, one of which has more parameters and FLOPs than GABA. Single runs do not allow us to identify the cause of the degradation with confidence. A plausible explanation is the loss of midpoint preservation: every alternative shifts the combined representation V + L rather than redistributing it between the two pathways. We will report the corresponding drift diagnostics and seed-averaged results in the revision.

(d) On linking spectral change to accuracy. We agree that Table 10, a within-model forward-pass comparison, cannot by itself attribute the accuracy gain to the bilinear constraint. Table R2 provides complementary cross-model evidence: the variant that the theory says cannot add state directions (shared C) shows no rise in numerical rank, a smaller effective-rank increase, and a smaller accuracy gain, even at matched parameters. We regard this as consistent co-variation across controlled variants, not as proof of causation, and will phrase it as such.

Revisions for Weakness 1. We will (i) add Tables R1–R3 to Appendix F and report Table 8 as seed averages; (ii) remove the statement that online/EMA agreement and checkpoint persistence support robustness, since they are not independent of the runs they come from; (iii) soften causal wording in Sec. 1 and Appendix F to "consistent with"; and (iv) add seeds and drift diagnostics for Table R3.

AuthorsOpenReview ↗

Response 3/4

Weakness 2: practical accuracy–efficiency advantage

(a) Measured execution cost. We agree that operation counts do not establish practical overhead. We measured inference cost under identical conditions.

Table R4: Measured inference cost (Tiny, 224²). Single GCD of an AMD Instinct MI250X, PyTorch 2.9.1, ROCm 6.4, fp32, synthetic input; throughput at batch 128, latency at batch 1 (median); 50 iterations after 20 warmup.

ModelEMA Top-1 (%)Throughput (img/s)Latency (ms)Peak mem (GB)
RAVLT_L81.69728.219.541.84
GABA-M1_L82.10594.426.402.42
GABA-M2_L82.21552.428.492.42
RAVLT_NL82.31732.416.941.84
GABA-M1_NL82.80629.619.922.42
GABA-M2_NL82.95584.621.522.42

The measured cost is higher than the operation count alone would suggest, mainly because the anti-symmetric exchange consists of element-wise operations with full-width intermediates at the highest-resolution stage, which are memory-bound in the current unfused implementation, as anticipated in Appendix E. This overhead is lower in the hybrid setting of Table 1, and it does not increase with input resolution: in additional measurements at 1024² (65,536 stage-1 tokens), the relative overhead remained at a similar level for the all-linear model and decreased markedly for the hybrid models, consistent with linear scaling.

GABA's aim is to improve the accuracy of linear attention while preserving linear complexity, not to reduce wall-clock cost, and it achieves consistent gains at every scale and across seeds. Our conclusion already lists the cost of the M2 gate and the open question of efficiency at long sequences among the limitations; we will extend that paragraph with these measurements, report them (including the higher-resolution results and MAViT and MHLA) in Appendix E, and state that the efficiency claims refer to parameters and operation counts.

(b) Reproduced vs. originally reported baselines. RAVLT and MAViT were reproduced with their official repositories, each with its own released training configuration, without modification. GABA uses the RAVLT configuration (300 epochs, AdamW with peak learning rate 10⁻³ and weight decay 0.05, global batch 1024, EMA decay 0.99996, stochastic depth 0.1/0.1/0.4/0.55 for Tiny/Small/Base/Large). All starred entries and all GABA entries report the best-epoch EMA accuracy, evaluated identically. Since the recipes are unchanged, we attribute the gap to the published values to differences in the training environment (hardware and library versions); as another reviewer also notes, MHLA reports reproduced values similar to ours. Because all models in our comparison share the same reproduction setting, our claims are relative to it. The full configurations are in the anonymized repository.

Abstract corrections. We found two errors in the abstract, which we will correct: 82.03% is the online accuracy of the all-linear Tiny model (not the hybrid model), and the +0.50 margin is relative to RALA, not to the second-closest baseline.

Comments on the paper

1 comment

The authors have posted their responses to the reviews. Could I please encourage the reviewers to read these responses and reply to the authors.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.