acceptodds
Withdrawn from ICLR 2027

SAGA: Spatially-Aware Gated Attention for Separating Outlier Structure from Functional Effect in Vision Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision Transformers can develop a small number of unusually high-norm patch tokens, motivating methods that suppress, relocate or regularize this activity. Yet reducing an outlier count or producing a cleaner spatial map does not necessarily imply that the representation used by a downstream task has improved. We introduce SAGA (Spatially-Aware Gated Attention), a lightweight receiver-side mechanism that modulates patch-token attention outputs by layer, head and spatial position without adding extra tokens. Across three audited 300-epoch ImageNet-1K architecture-recipe cells with matched Baseline-SAGA checkpoints, SAGA reduces fixed-threshold high-norm-token counts by approximately – while yielding - percentage-point higher observed mean top-1 accuracy than the ViT baselines. Mechanistically, high-norm positions receive - more incoming attention per token than other patch positions and form highly reproducible spatial patterns. Matched frozen interventions, however, show that this spatial reproducibility does not translate into a reproducible coordinate-specific classifier response in the tested blocks. Functional relevance also depends on the readout: terminal patch-gate edits can alter patch representations while leaving CLS logits exactly unchanged, whereas, in a descriptive patch-level analysis, high-norm positions are associated with substantially poorer exact-transform correspondence. These results separate fixed-threshold outlier prevalence, spatial organization and readout-specific functional effect, motivating evaluation of ViT diagnostics against the downstream computation that consumes them.

Primary area:
interpretability and explainable AI
Keywords:
Vision Transformers, high-norm tokens, attention sinks, spatial attention, gated attention, representation analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cudpOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

SAGA adds an input-independent sigmoid gate per block, head and patch position to the attention output of ViTs. The aim is to suppress high-norm outlier tokens at negligible cost. On ImageNet-1K, it cuts high-norm token counts by 46-65% and improves top-1 by 0.3-0.5pp. Most of the paper is a mechanistic analysis of high-norm tokens. It argues that outlier prevalence, spatial pattern and functional effect should be evaluated separately.

Critical strengths
  1. The analysis protocol is careful. It uses separate discovery and evaluation splits, matched controls and replication across checkpoints. Counterexamples are reported openly.
  2. SAGA is very cheap, adding only 14-28K parameters and no extra tokens. It lowers high-norm token counts without the accuracy drop seen with registers.
Critical weaknesses
  1. The baselines appear undertrained. ViT-B reaches only 76.9%, below its own ViT-S and far below DeiT-B (81.8%). A 0.3-0.5pp gain on top of this says little. In addition, several negative transfer results appear only in the appendix.
  2. The spatial part of the gate is not shown to matter. Replacing each head's spatial gate by its mean barely changes the loss. A frozen 0.5 gate or a per-head scalar already gives most of the outlier reduction. The gate thus mostly rescales norms. Query-dependent gated attention (Qiu et al., 2025) and standard LayerScale are not compared experimentally, and the per-head scalar control is run only at 100 epochs.
  3. The mechanistic results are weak. CLS logits not changing under a patch-only final gate holds by construction. The frozen interventions give effects of at most a few 1e-4 nats per image, with no positive control. The paper admits these are not equivalence tests, yet its main thesis still leans on them.
Questions for authors
  1. Do the gains hold when both models are trained to published DeiT accuracy?
  2. Does the per-head scalar result hold at 300 epochs? How does SAGA compare with query-dependent gating, and can the intervention protocol detect an effect that is known to exist?
Reviewer_zWAaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes SAGA, a spatially-aware gating mechanism for mitigating high-norm tokens in Vision Transformers. The authors analyze outlier-token behavior and report modest improvements in image classification accuracy.

Critical strengths
  1. The paper addresses an important problem in Vision Transformers, as understanding and mitigating high-norm outlier tokens is relevant to improving visual representations.
  2. The authors provide empirical analyses of outlier-token behavior, including their spatial distribution, attention patterns, and functional effects on downstream predictions.
Critical weaknesses

1. Limited novelty and unclear contribution. Outlier tokens in Vision Transformers have been extensively studied, with numerous works proposing analyses and mitigation strategies. The paper does not provide novel insights or a compelling methodological contribution compared with existing work.

2. Unclear advantages over simpler methods. The proposed approach is unnecessarily complex and cumbersome, requiring architectural modifications and a specialized spatial gating mechanism. In contrast, existing methods such as register tokens are much simpler and more elegant. The paper does not convincingly justify this additional complexity or demonstrate why practitioners should adopt SAGA over simpler alternatives, especially given its modest performance improvements.

3. Limited downstream evaluation. The evaluation primarily focuses on classification. Although COCO detection and ADE20K segmentation are included, the improvements are marginal or absent. More extensive evaluations on segmentation (Cityscapes, PASCAL VOC), depth estimation (KITTI, NYUv2), and detection are needed to demonstrate the usefulness of the proposed approach

4. Insufficient baselines. The comparisons mainly focus on register-based approaches, despite extensive prior work on mitigating ViT artifacts. Relevant methods such as SINDER, Denoising Vision Transformers, and Vision Transformers Need More Than Registers should be considered.

5. Poor presentation. The paper is unnecessarily difficult to follow. The presentation should be simplified, particularly Figure 2, which combines too many concepts and obscures the main idea.


The paper addresses a well-studied problem but offers limited novelty and unconvincing practical benefits. The modest and inconsistent downstream improvements, insufficient baselines, and unclear presentation make the contribution relatively weak.

Questions for authors

see weaknesses

Reviewer_hBBjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The work attempts to treat an observed effect in ViT where some tokens have exceptionally high norm as compared to other tokens. This is a well known effect and prior works have attributed it as a cause of instances of poor task performance. To tackle the high-norm problem on these tokens, a spatially aware gated attention modification is proposed for the ViT based framework. The method applies a modulation gate on the standard dot-product attention, thus reducing the norm of these tokens. This gate has learnable spatial parameters, and thus applies a spatially dependent modulation to the token norms. In comparison with a few standard ViT architectures on image net, small accuracy performance improvements are observed and a large improvement is observed for the “high-norm”. Ablations studies were also performed to investigate the effect of the spatially dependent modulation via a few naive modulation schemes.

Critical strengths

Good attempts were made at studying the effectiveness of the proposed method with the proposal of different schemes that could potentially work in the same way. These are the different interventions as highlighted in figure 2, (b), (c), and (d). The inclusion of such experiments is important in showing the actual gains of the proposed method.

Critical weaknesses

Primary weaknesses Much of the work look at how the norm of token are being reduced. However, as evident from both the mathematical formulation of transformers and other empirical evidence in prior works, this is unavoidable as softmax have to sum to 1, and attention needs to go somewhere. In fact, the capping/scaling of these norm was also studied in the ICLR 2024 paper Vit needs registers, where they proposed alternative attention sinks (registers) and compared with scaling of norms. This work should be compare in far more depth with the registers method and other methods that aims to tackle the problem of the high-norm reducing spatial information of the token rather than just reducing the token norm. As such, it’s of this reviewer’s opinion that the goal of reducing token norm is a poor objective. The problem should be framed differently, perhaps answering questions like these - high-norm tokens are observed to be correlated with certain task performance issues, 1) high-norm tokens are bad unconditionally and reducing them helps task performance, 2) reducing them by redirecting them helps, 3) high-norms are necessary.

Other weaknesses The description of the gate isn't clear. It appears that the modulation is not input dependent. Correct? Evaluation in tables 1 and 2 are biased towards this work's results. As highlighted in previous parts of the review and also by the authors, several other works attempt to improve the performance of ViTs by treating the effect of high-norm tokens. Instead of just comparing with base ViT, these tables should compare with the results of all the other methods that was listed in prior works.

Questions for authors

From my reading, it appears that the gate does not depend on input. Thus modulation is fixed once a model is trained. Doesn’t this mean that the result of the gate is just a fixed spatial rescaling of the norms? The mechanism seems overly complicated for a fix, learned scaling. Clarify this if my reading was wrong.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.