- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that part of the gap between linear and softmax attention comes from ReLU-style feature maps being tied to fixed coordinate axes: softmax attention is invariant to a shared orthogonal rotation of queries and keys, while ReLU(Q)ReLU(K)^T is not. The proposed fix, HaLA, applies learnable Householder reflections to queries and keys before the ReLU. It has three parts: block-wise 2D reflections inside each head, a reflection angle modulated by the per-block variance over tokens, and a single global reflection across heads before the head split. HaLA is evaluated as a backbone on ImageNet classification, COCO detection, ADE20K/Cityscapes segmentation, lightweight super-resolution, and DiT/SiT image generation, as well as a drop-in replacement in Swin-T and PVT-T. It reports consistent but mostly small improvements.
Critical strengths
The module is simple, cheap in parameters, and easy to plug into existing linear attention designs (ReLU, FLatten, 1+ELU).
The evaluation is broad, covering classification, detection, segmentation, super-resolution and diffusion models, and the improvements point in the same direction across tasks.
The controlled replacements on Swin-T and PVT-T, and the LRA results with two different feature maps, are the most convincing part of the paper.
Critical weaknesses
-
The motivation doesn't seem to hold. The reflection is a fixed linear orthogonal map applied right after W_q/W_k, so it can be absorbed into the projection: ReLU(H W_q x) = ReLU(W'_q x). The baseline can already learn any basis. Only the variance-dependent angle is new, and its range is narrow (alpha_max/2 to alpha_max). Theorem 1 doesn't compare with vs. without modulation, and Tab. 16 removes H from a model trained with H, which is not a fair comparison.
-
The gains are small and some comparisons are not matched. HaLA backbones have the same size as RAVLT and beat it by only about 0.2 points. In the controlled Swin-T/PVT-T setting, HaLA-ReLU is below Agent and PolaFormer. The detection text compares RAVLT-T with HaLA-S, two different sizes. The ~80% memory/latency savings are relative to softmax; against a plain ReLU map, HaLA is slower and uses more memory. Several appendix tables are "from the rebuttal" with unspecified units.
Questions for authors
- Does the gain survive if H is merged into W_q? What does the variance modulation alone give on top of plain ReLU?
- Can you compare retention between a model trained with plain ReLU and one trained with HaLA?
- Is HaLA-T/S/B/L the RAVLT architecture with only the feature map changed? If so, please report that comparison with matched seeds.
- Please report throughput and memory against the same linear attention without HaLA, with units.
- Table 26 lists 129G FLOPs for HaLA-T (3x) but 219G in Table 25 (1x). Is this a typo?