acceptodds
Under review as a conference paper at ICLR 2027

HaLA: Householder-adapted Linear Attention for Vision Transformers

Abstract

Linear attention offers an efficient alternative to softmax attention, but its representational gap remains poorly understood. We show that part of this gap arises from a broken geometric symmetry. Softmax attention is coordinate-free: rotating queries and keys together preserves their inner products and leaves attention unchanged. ReLU-style linear attention, however, applies a non-negative gate along fixed feature axes, making attention sensitive to the coordinate system. This gate can passively truncate useful semantic directions before aggregation, leading to a failure mode we call coordinate-lazy truncation. To address this, we propose Householder-adapted Linear Attention (HaLA), a geometry-aware linear attention framework that turns passive truncation into active reorientation. By leveraging learnable Householder reflections, HaLA rotates the query-key coordinate system into the non-negative orthant before non-negative projection to maximize information retention. Built on this principle, HaLA introduces a cohesive multi-scale design: block-wise isometries enhance local discriminability, variance-aware modulation stabilizes long-context activation diversity, and cross-head reflections integrate dispersed subspaces through global covariance mixing. Across various vision tasks including long-sequence super-resolution and image generation, HaLA consistently strengthens linear attention and closes much of the gap to softmax attention under efficient linear-time computation. Source code can be found in the supplementary materials.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Linear attention, efficient transformer, vision transformer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_ZoEROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that part of the gap between linear and softmax attention comes from ReLU-style feature maps being tied to fixed coordinate axes: softmax attention is invariant to a shared orthogonal rotation of queries and keys, while ReLU(Q)ReLU(K)^T is not. The proposed fix, HaLA, applies learnable Householder reflections to queries and keys before the ReLU. It has three parts: block-wise 2D reflections inside each head, a reflection angle modulated by the per-block variance over tokens, and a single global reflection across heads before the head split. HaLA is evaluated as a backbone on ImageNet classification, COCO detection, ADE20K/Cityscapes segmentation, lightweight super-resolution, and DiT/SiT image generation, as well as a drop-in replacement in Swin-T and PVT-T. It reports consistent but mostly small improvements.

Critical strengths

The module is simple, cheap in parameters, and easy to plug into existing linear attention designs (ReLU, FLatten, 1+ELU).

The evaluation is broad, covering classification, detection, segmentation, super-resolution and diffusion models, and the improvements point in the same direction across tasks.

The controlled replacements on Swin-T and PVT-T, and the LRA results with two different feature maps, are the most convincing part of the paper.

Critical weaknesses
  1. The motivation doesn't seem to hold. The reflection is a fixed linear orthogonal map applied right after W_q/W_k, so it can be absorbed into the projection: ReLU(H W_q x) = ReLU(W'_q x). The baseline can already learn any basis. Only the variance-dependent angle is new, and its range is narrow (alpha_max/2 to alpha_max). Theorem 1 doesn't compare with vs. without modulation, and Tab. 16 removes H from a model trained with H, which is not a fair comparison.

  2. The gains are small and some comparisons are not matched. HaLA backbones have the same size as RAVLT and beat it by only about 0.2 points. In the controlled Swin-T/PVT-T setting, HaLA-ReLU is below Agent and PolaFormer. The detection text compares RAVLT-T with HaLA-S, two different sizes. The ~80% memory/latency savings are relative to softmax; against a plain ReLU map, HaLA is slower and uses more memory. Several appendix tables are "from the rebuttal" with unspecified units.

Questions for authors
  1. Does the gain survive if H is merged into W_q? What does the variance modulation alone give on top of plain ReLU?
  2. Can you compare retention between a model trained with plain ReLU and one trained with HaLA?
  3. Is HaLA-T/S/B/L the RAVLT architecture with only the feature map changed? If so, please report that comparison with matched seeds.
  4. Please report throughput and memory against the same linear attention without HaLA, with units.
  5. Table 26 lists 129G FLOPs for HaLA-T (3x) but 219G in Table 25 (1x). Is this a typo?
Reviewer_bVgBOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Householder-adapted Linear Attention (HaLA), which applies learnable geometric transformations before non-negative feature mapping. The method combines block-wise Householder reflections, variance-aware angle modulation, and cross-head mixing to reduce information loss associated with ReLU-based linear attention. Experiments cover image classification, detection, segmentation, super-resolution, and image generation.

Critical strengths
  1. Clear motivation and interpretable design. The paper's observation that ReLU-based feature maps depend on the coordinate system provides a reasonable motivation for learning feature orientations before activation. Householder reflections offer a novel way to implement this idea while maintaining linear complexity.
  2. Compatibility with existing methods. HaLA can improve an existing attention mechanism such as FLatten, suggesting that the approach can serve as a reusable component rather than requiring an entirely new architecture.
  3. Experiments demonstrate useful improvements. HaLA increases Swin-T accuracy from 81.8% to 82.4% over the ReLU linear-attention baseline. Adding HaLA to FLatten improves PVT-T accuracy from 77.8% to 79.2%. These comparisons provide evidence that the proposed module contributes to performance.
  4. Comprehensive experimental evaluation. The paper evaluates HaLA across classification, detection, segmentation, super-resolution, and image generation. It also includes component ablations, comparisons with alternative designs, and efficiency measurements, providing a thorough assessment of the method’s performance and practical costs.
Critical weaknesses
  1. The advantage over closely related methods is not consistent. Table 6 demonstrates that HaLA's comparison with PolaFormer depends on the configuration: HaLA-ReLU is slightly lower on Swin (82.4% vs. 82.6%), whereas HaLA-FLatten is higher on PVT (79.2% vs. 78.8%).
  2. Adding HaLA makes the model slower. Appendix Table 17 shows that, on PVT-T, processing speed drops from 2949 to 1903 for ReLU attention, a decrease of about 35%, and from 2823 to 1730 for FLatten, a decrease of about 39%. The accuracy improvements therefore come at the cost of slower inference.
Questions for authors

Do Q and K use the same reflection after the variance-based angle adjustment? Please clarify how the variances and angles are computed and which parameters are shared between Q and K.

Reviewer_ud8UOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Householder-adapted Linear Attention (HaLA), which applies reflections before the non-negative feature map. This design is expected to improve information retention and produce more diverse activation patterns. The method is evaluated on several vision tasks and additional sequence benchmarks.

Critical strengths
  1. The method is conceptually simple and can be used in combination with existing linear attention methods that adopt non-negative kernels.
  2. The evaluation includes several tasks, with more ablations and controlled experiments in the appendix.
Critical weaknesses
  1. The static reflections can be absorbed into the projection matrices. The paper's explaination for this weakness is weak and not convincing.
  2. The practical overhead is remarkable, although the FLOP increase is theoretically small. The computation for Household reflection is indeed lightweight, but it can waste the computational capacity of modern TensorCore. The paper's reported throughput supports this.
  3. The implementation of HaLA includes DWC and output gating, which are not all necessarily included in baselines. This setting underestimates much of the baseline's performance, and the relative improvement in the ablation study is trivial.
Questions for authors

See Weaknesses. The practical problem and experimental setting are very worrisome.

Reviewer_gPr6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines how the coordinate sensitivity of ReLU-based linear attention can affect feature representation. The authors argue that applying a fixed, axis-aligned non-negative mapping may discard useful information along certain feature directions. They propose Householder-adapted Linear Attention (HaLA), which uses learnable Householder reflections to adjust the orientation of query and key features before applying non-negative activation.

HaLA includes three components. Block-wise Householder reflections adjust feature orientations within local subspaces, variance-aware angle modulation adapts the reflections according to input statistics, and cross-head reflection allows information to be mixed across attention heads.

The authors evaluate HaLA on several vision tasks, including ImageNet-1K classification, COCO object detection and instance segmentation, semantic segmentation, image super-resolution, and diffusion-based image generation. The results generally show improvements over existing efficient-attention baselines. The paper also includes component ablations, feature-retention analyses, and efficiency measurements to examine the proposed design.

Critical strengths

Strength 1: Interesting geometric motivation

The paper identifies an important difference between dot-product attention and linear attention with non-negative feature mappings. In particular, feature orientation matters when ReLU is applied coordinate-wise, since different orientations can lead to different activation patterns. Using Householder reflections to study this issue is a reasonable and mathematically motivated choice. I find this perspective interesting, and it may also be relevant to other attention mechanisms that rely on non-negative feature mappings.

Strength 2: Broad experimental evaluation with promising results

The authors evaluate HaLA across a wide range of vision tasks, including classification, detection, segmentation, super-resolution, and image generation. The results are generally competitive and show improvements over several efficient-attention baselines. For example, the reported classification accuracy with PVT increases from 77.8% using FLatten to 79.2% using HaLA-FLatten. The ablation studies are also useful, as they provide evidence that the proposed components contribute to the overall performance. The feature-retention analyses offer further empirical support for the motivation.

Critical weaknesses

Weakness 1: The theoretical guarantee for activation diversification is not sufficiently justified

My main theoretical concern is with Theorem 1. The theorem appears to claim a strictly positive lower bound on the expected disagreement between activation masks, expressed in terms of pairwise feature distances, when the mean of the reflected features is sufficiently close to a ReLU activation boundary.

However, having the feature mean close to an activation boundary does not necessarily mean that different tokens will have different activation masks. The direction of the token perturbations also matters.

For example, consider two reflected feature vectors

y_2=(\varepsilon,1-\delta),$$ where $\varepsilon>0$ can be arbitrarily small and $0<\delta<1$. The mean of these two vectors is $(\varepsilon,1)$, which can be made arbitrarily close to the ReLU boundary in the first coordinate. However, both vectors have the same activation mask, even though their Euclidean distance is nonzero. This seems inconsistent with a strictly positive lower bound based on feature distance and boundary proximity alone. Additional assumptions about the distribution or directions of token perturbations may be needed. As a result, I do not think the current theorem adequately establishes the proposed activation-diversification mechanism. It would be helpful for the authors to revisit the proof and clarify the precise conditions under which the guarantee holds. **Weakness 2: The contribution of static Householder reflections is not clearly separated from standard learnable projections.** Another concern is whether static Householder reflections provide additional representational capacity beyond the existing query and key projection layers. If queries and keys are generated using unrestricted learnable linear projections, a fixed Householder transformation can be absorbed into the projection weights. For example, $$\operatorname{ReLU}(XW'_Q), \qquad W'_Q=W_QH.$$ Therefore, a fixed reflection does not necessarily expand the set of feature transformations that the model can represent. The performance improvements may come from the optimization properties or inductive bias introduced by the Householder parameterization. The input-dependent variance modulation may also provide additional expressive power, but its role needs to be distinguished more clearly from that of static reflection. The current ablations show that removing reflection-related components leads to worse performance. However, these comparisons do not fully address whether Householder reflections are more effective than simpler learnable transformations under comparable conditions. I think this distinction is important because the paper places considerable emphasis on the geometric role of the proposed reflections. More controlled comparisons would make the methodological contribution easier to assess.
Questions for authors

Q1. Theoretical validity of Theorem 1

Could the authors clarify the assumptions needed for the positive lower bound on activation-mask disagreement in Theorem 1? In particular, how does the proof handle cases where token perturbations are tangential to an activation boundary, so that the tokens have nonzero pairwise distances but identical activation masks?

I would appreciate a revised theorem or a more explicit statement of the required assumptions. This is an important concern for my assessment of the theoretical contribution.

Q2. Reparameterization and the source of performance gains

Since a fixed Householder transformation can be absorbed into unrestricted learnable query and key projections, what additional representational capability does the static reflection provide?

Could the authors compare HaLA with a matched learnable linear transformation or another projection parameterization? It would also be useful to isolate the gains from input-dependent variance modulation, since this component may offer a clearer source of additional expressivity.

Q3. Accuracy and efficiency trade-off

Appendix Table 17 shows that adding HaLA reduces throughput compared with the corresponding linear-attention baselines.

Could the authors provide a more direct comparison of accuracy and throughput under matched implementations? This would help clarify whether the accuracy improvements justify the additional computational cost, especially given that efficiency is one of the main motivations for linear attention.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.