- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
SAGA adds an input-independent sigmoid gate per block, head and patch position to the attention output of ViTs. The aim is to suppress high-norm outlier tokens at negligible cost. On ImageNet-1K, it cuts high-norm token counts by 46-65% and improves top-1 by 0.3-0.5pp. Most of the paper is a mechanistic analysis of high-norm tokens. It argues that outlier prevalence, spatial pattern and functional effect should be evaluated separately.
Critical strengths
- The analysis protocol is careful. It uses separate discovery and evaluation splits, matched controls and replication across checkpoints. Counterexamples are reported openly.
- SAGA is very cheap, adding only 14-28K parameters and no extra tokens. It lowers high-norm token counts without the accuracy drop seen with registers.
Critical weaknesses
- The baselines appear undertrained. ViT-B reaches only 76.9%, below its own ViT-S and far below DeiT-B (81.8%). A 0.3-0.5pp gain on top of this says little. In addition, several negative transfer results appear only in the appendix.
- The spatial part of the gate is not shown to matter. Replacing each head's spatial gate by its mean barely changes the loss. A frozen 0.5 gate or a per-head scalar already gives most of the outlier reduction. The gate thus mostly rescales norms. Query-dependent gated attention (Qiu et al., 2025) and standard LayerScale are not compared experimentally, and the per-head scalar control is run only at 100 epochs.
- The mechanistic results are weak. CLS logits not changing under a patch-only final gate holds by construction. The frozen interventions give effects of at most a few 1e-4 nats per image, with no positive control. The paper admits these are not equivalence tests, yet its main thesis still leans on them.
Questions for authors
- Do the gains hold when both models are trained to published DeiT accuracy?
- Does the per-head scalar result hold at 300 epochs? How does SAGA compare with query-dependent gating, and can the intervention protocol detect an effect that is known to exist?