- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces SSoG, a separable sum-of-Gaussians spatial operator that replaces dense self-attention with a learned mixture of axis-factorized Gaussian kernels, reducing the computational complexity of token mixing. The authors also introduce a content-aware variant that allows the Gaussian parameters to adapt to the input. They provide extensive ablation studies across standard discrete ViTs, weight-shared/looped Transformers, and continuous-depth Neural ODE Transformers. An important theoretical aspect of the paper is the Lipschitz continuity of the proposed operators, which provides stability guarantees by bounding how much the output can change in response to changes in the input. This property is particularly relevant for looped and continuous-depth architectures, where the same operator is repeatedly applied. The fixed-kernel SSoG is globally Lipschitz continuous, whereas the content-aware variant is Lipschitz continuous only on bounded sets of token inputs. The authors explicitly address these properties in their theoretical analysis. Experiments on CIFAR-100 and ImageNet-1k show that SSoG achieves improved or competitive accuracy compared with dense attention, while its throughput and memory advantages become increasingly substantial as the input resolution increases.
Critical strengths
A> The operator design is novel and it reduces the compute at every substep, across multiple settings. Importantly, SSoG can be incorporated into existing Transformer backbones without substantially modifying the overall architecture, while providing reduced runtime complexity.
B> The paper also considers the edge cases when it comes to Lipschitz continuity. The authors provide an extensive study on the same.
Critical weaknesses
A> Though the ablation studies are extensive for the hyperparameters, the proposed method needs to be compared to existing SOTA methods. Existing works like [1,2], use a subset of the pipeline to execute.
B> There is a possibility that the effectiveness of the algorithm is limited to classification-centric settings . The [CLS] token does not perform content-dependent patch selection as in standard self-attention; instead, its per-block update is based on the mean of the projected patch values. For tasks, which do not involve a single token, like generation or object detection; the model might not function as well.
[1] Meng, L., Li, H., Chen, B. C., Lan, S., Wu, Z., Jiang, Y. G., & Lim, S. N. (2022, June). Adavit: Adaptive vision transformers for efficient image recognition. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 12299-12308). IEEE. [2] Xu, G., Hao, J., Shen, L., Hu, H., Luo, Y., Lin, H., & Shen, J. (2023, October). Lgvit: Dynamic early exiting for accelerating vision transformer. In Proceedings of the 31st ACM International Conference on Multimedia (pp. 9103-9114).
Questions for authors
The paper is a nice read. Please explain how the proposed pipeline can be used on non-classification/ transformer based tasks, where the final decision does not depend on the [CLS] token.
Response ZauD
B and Q: Tasks without a [CLS] token. We agree with the reviewer that the class token only reads out the mean of the patch values. Patch-to-patch mixing never uses it. The backbone therefore produces ordinary per-patch features, and any dense head can use them. We fine-tuned the ImageNet backbones for segmentation without a class token to see if SSoG still handles in this task (Reviewer FjRV asked the same, W4: see answer to FjRV):
SSoG matches or exceeds dense attention, and at 1024 px its backbone needs 47% of the dense FLOPs. Generation or video tasks are added as a future research.
A: Comparison with AdaViT and LGViT. We see [1] and [2] as complementary methods as both reduces how much of the network runs for each input (AdaViT drops patches, heads and blocks; LGViT exits early), whereas SSoG reduces the cost of every block that runs. We test the LGViT-style exit heads on the frozen CIFAR-100 models, after blocks 2–10 or after as many loop iterations or Euler steps (Will be added as Appendix A.8). The table gives the first exit at which SSoG exceeds the accuracy of the full dense model:
At equal FLOPs, SSoG with exits is more accurate than dense attention with exits over most of the range in all three settings. The implementation is a simplification of LGViT (frozen backbone (for time)). For AdaViT's patch dropping we see it as a further analysis which cannot be directly applied because SSoG mixes over the full grid, so dropped patches would be zeroed, which saves the MLP but not the mixer.
We also compared SSoG with the two closest alternatives to attention, trained in the same block and recipe (discrete; new Appendix A.6):
SSoG is 3.4 points more accurate than linear attention. A freely learned separable filter matches it, but id goes on a different axis at it is a completely different architecture, not a ViT, so we believe is out of the scope of the paper. However the forced Gaussian structure adds a specified structure: a positive, normalized kernel with about 240 parameters per layer instead of about 41k params, and a Lipschitz constant that does not depend on the input (Proposition 1), on which the continuous-depth analysis relies.