acceptodds
Under review as a conference paper at ICLR 2027

Beyond Quadratic Attention: Separable Gaussian Operators for ODE Vision Transformers

Abstract

Self-attention carries a quadratic cost in token count, which limits how far Vision Transformers can scale with input resolution. We introduce SSoG, a separable sum-of-Gaussians spatial operator that replaces dense self-attention with a learned mixture of axis-factorized Gaussian kernels, reducing the per-layer cost from \(O(N^2)\) to subquadratic in \(N\). Unlike dot-product self-attention, whose Lipschitz constant is unbounded outside a compact input domain, SSoG with fixed kernels is globally Lipschitz continuous, giving the continuous-depth variant a well-posedness guarantee via the Picard–Lindel\"of theorem that dense-attention counterparts lack in general. These results establish SSoG as a practical, parameter-efficient, and theoretically grounded alternative to dense attention across discrete, weight-shared, and continuous-depth Transformer variants. On CIFAR-100 at \(224\times224\), in a matched discrete-Transformer setting and in the continuous setting, SSoG improves Top-1 accuracy by 2.22-4.12 points across three model widths while using 8.8-16.6% fewer parameters, with depth and training recipe held fixed. In a looped-Transformer setting, SSoG improves Top-1 accuracy by 5.4-8.14 points. We further probe inference-time scaling by evaluating on inputs up to \(1024\times1024\) at matched batch size: in the continuous-depth (Neural ODE) architecture, SSoG reaches up to 154% higher throughput and over 60% lower peak memory than dense attention, with the gap widening as resolution grows, consistent with the difference in asymptotic cost. In the discrete and looped settings, SSoG also achieves over 30% higher throughput than dense attention at this resolution.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Gaussian mixture models, neural ODEs, continuous-depth networks, parameter efficiency, Lipschitz continuity, attention mechanism, vision transformers

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZauDOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces SSoG, a separable sum-of-Gaussians spatial operator that replaces dense self-attention with a learned mixture of axis-factorized Gaussian kernels, reducing the computational complexity of token mixing. The authors also introduce a content-aware variant that allows the Gaussian parameters to adapt to the input. They provide extensive ablation studies across standard discrete ViTs, weight-shared/looped Transformers, and continuous-depth Neural ODE Transformers. An important theoretical aspect of the paper is the Lipschitz continuity of the proposed operators, which provides stability guarantees by bounding how much the output can change in response to changes in the input. This property is particularly relevant for looped and continuous-depth architectures, where the same operator is repeatedly applied. The fixed-kernel SSoG is globally Lipschitz continuous, whereas the content-aware variant is Lipschitz continuous only on bounded sets of token inputs. The authors explicitly address these properties in their theoretical analysis. Experiments on CIFAR-100 and ImageNet-1k show that SSoG achieves improved or competitive accuracy compared with dense attention, while its throughput and memory advantages become increasingly substantial as the input resolution increases.

Critical strengths

A> The operator design is novel and it reduces the compute at every substep, across multiple settings. Importantly, SSoG can be incorporated into existing Transformer backbones without substantially modifying the overall architecture, while providing reduced runtime complexity.

B> The paper also considers the edge cases when it comes to Lipschitz continuity. The authors provide an extensive study on the same.

Critical weaknesses

A> Though the ablation studies are extensive for the hyperparameters, the proposed method needs to be compared to existing SOTA methods. Existing works like [1,2], use a subset of the pipeline to execute.

B> There is a possibility that the effectiveness of the algorithm is limited to classification-centric settings . The [CLS] token does not perform content-dependent patch selection as in standard self-attention; instead, its per-block update is based on the mean of the projected patch values. For tasks, which do not involve a single token, like generation or object detection; the model might not function as well.

[1] Meng, L., Li, H., Chen, B. C., Lan, S., Wu, Z., Jiang, Y. G., & Lim, S. N. (2022, June). Adavit: Adaptive vision transformers for efficient image recognition. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 12299-12308). IEEE. [2] Xu, G., Hao, J., Shen, L., Hu, H., Luo, Y., Lin, H., & Shen, J. (2023, October). Lgvit: Dynamic early exiting for accelerating vision transformer. In Proceedings of the 31st ACM International Conference on Multimedia (pp. 9103-9114).

Questions for authors

The paper is a nice read. Please explain how the proposed pipeline can be used on non-classification/ transformer based tasks, where the final decision does not depend on the [CLS] token.

1 comment
AuthorsOpenReview ↗

Response ZauD

B and Q: Tasks without a [CLS] token. We agree with the reviewer that the class token only reads out the mean of the patch values. Patch-to-patch mixing never uses it. The backbone therefore produces ordinary per-patch features, and any dense head can use them. We fine-tuned the ImageNet backbones for segmentation without a class token to see if SSoG still handles in this task (Reviewer FjRV asked the same, W4: see answer to FjRV):

SSoG matches or exceeds dense attention, and at 1024 px its backbone needs 47% of the dense FLOPs. Generation or video tasks are added as a future research.

A: Comparison with AdaViT and LGViT. We see [1] and [2] as complementary methods as both reduces how much of the network runs for each input (AdaViT drops patches, heads and blocks; LGViT exits early), whereas SSoG reduces the cost of every block that runs. We test the LGViT-style exit heads on the frozen CIFAR-100 models, after blocks 2–10 or after as many loop iterations or Euler steps (Will be added as Appendix A.8). The table gives the first exit at which SSoG exceeds the accuracy of the full dense model:

SettingFull dense: top-1 / GFLOPsSSoG with exitsFLOPs saved
Discrete75.09 / 35.176.80 / 22.237%
Looped72.30 / 35.176.11 / 16.553%
Continuous-depth69.43 / 35.270.78 / 20.542%

At equal FLOPs, SSoG with exits is more accurate than dense attention with exits over most of the range in all three settings. The implementation is a simplification of LGViT (frozen backbone (for time)). For AdaViT's patch dropping we see it as a further analysis which cannot be directly applied because SSoG mixes over the full grid, so dropped patches would be zeroed, which saves the MLP but not the mixer.

We also compared SSoG with the two closest alternatives to attention, trained in the same block and recipe (discrete; new Appendix A.6):

Mixer (CIFAR-100)Top-1GFLOPs at 1024 pximg/s at 1024 px, compiled
Dense attention71.2170.958
Linear attention (Katharopoulos et al., 2020)77.6620.184
Separable depthwise convolution81.0610.5102
Fixed SSoG81.0512.974

SSoG is 3.4 points more accurate than linear attention. A freely learned separable filter matches it, but id goes on a different axis at it is a completely different architecture, not a ViT, so we believe is out of the scope of the paper. However the forced Gaussian structure adds a specified structure: a positive, normalized kernel with about 240 parameters per layer instead of about 41k params, and a Lipschitz constant that does not depend on the input (Proposition 1), on which the continuous-depth analysis relies.

Reviewer_mcUYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work tests whether using a separable sum-of-Gaussians spatial operator can replace standard dot-product attention in vision Transformers. It uses spatial kernels and query-dependent steering, provides a theoretical analysis, and evaluates classification accuracy and computational efficiency. The fixed operator is subquadratic and improves CIFAR-100 accuracy across the standard, looped, and continuous-depth settings, while the gains on ImageNet are smaller and setting dependent.

Critical strengths
  1. The comparisons across three architectural setups, by testing SSoG across discrete, looped, and continuous-depth models, strengthen the claim more than a comparison between unrelated architectures. Testing on ImageNet shows that SSoG remains competitive, though it may not scale to the same performance gap seen on smaller datasets, like CIFAR-100.

  2. Reporting the negative results openly clarifies the practical use cases where SSoG may be beneficial over standard multi-head attention (MHA), and where MHA may be preferable.

Critical weaknesses
  1. The ODE-ViT baseline does not use the fused-attention implementation used by the other dense-attention baselines. This makes it difficult to separate the benefit of SSoG from the difference in implementation. In fact, the isolated comparison in Table 3 shows that SSoG uses more memory than fused MHA. The abstract should avoid implying that SSoG’s lower computational cost alone explains the largest efficiency gains.

  2. Some of the reported accuracy changes do not match the values in the tables. In the patch-size robustness experiment (Table 13), a drop in continuous-depth MHA accuracy is reported as an improvement, while an increase for discrete MHA is reported as a decline. The continuous-depth ImageNet gain stated in line 374 ("+0.30-point margin for continuous-depth") also does not match Table 2. This makes the comparisons harder to interpret, so I would suggest re-checking the accuracy values and reported changes, then clarifying the discrepancies. Overall, the results support SSoG as a competitive alternative to MHA, though its advantages depend on the setting. The comparisons across architectures are useful, but the largest efficiency gains cannot be attributed to the operator alone. I would suggest narrowing those claims to what the comparisons show and resolving the inconsistencies in the reported accuracy changes.

  3. There are no big dataset experiments. How does the method perform on ImageNet? Without results on ImageNet, just based on CIFAR-100 experiments, it is impossible to judge the resilience of the method.

Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Response mcUY

W1: ODE-ViT baseline and the source of the efficiency gains. We agree with the reviewer that the largest continuous-depth gains could not be attributed to the operator. The cause, however, was not the attention kernel. Profiling shows that the ODE-ViT baseline, like the other two dense baselines, runs PyTorch's fused FlashAttention kernel at inference. We fixed a memory leak in the official ODE-ViT baseline, caused by retaining 12 intermediate states, and re-ran all efficiency benchmarks. Even after this fix, SSoG retains its throughput advantage at higher sequence lengths, achieving a 20–35% higher peak throughput than fused FlashAttention at 1024×1024 resolution (25–30% when both are compiled), while compiled dense attention remains faster below 512–640 px. Peak memory consumption stays within 20% of the fused kernel across settings. Beyond this empirical validation, our theoretical complexity derivation (Appendix A.1) confirms that SSoG has a lower cost than MHA. While FlashAttention benefits from extensive hardware-level engineering optimizations that SSoG currently lacks, SSoG already demonstrates superior throughput as token counts scale, leaving substantial room for further performance gains through dedicated implementation tuning.

W2: Reported changes that did not match the tables. Thank you for this catch. Both mismatches had the same root cause: checkpoint-selection conventions had been mixed across the tables and we missed it when we proofread the paper. We have checked again the reported figures under this convention to rule out further inconsistencies, and the text now matches the tables. Under this convention, the reported changes match the table values: in the patch-size table, continuous MHA 69.43 → 70.23 (+0.80) and discrete MHA 75.09 → 74.91 (−0.18); on ImageNet, continuous 64.02 vs. 63.04 (+0.98) and discrete 77.99 vs. 76.60 (+1.39).

W3: ImageNet. We are a little bit confused about this comment, as the reviewer pointed out in their summary of the paper that results on ImageNet were reported, although gains on that dataset are “smaller and setting dependent”. We agree, of course, that more experiments of bigger datasets will always provide more insights into the resilience of the method; we believe that the current results on ImageNet provide sufficient evidence to judge the effectiveness and value of our proposed method within the scope of this work.

Reviewer_FjRVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the Sum of Separable Gaussians (SSoG) operator as an alternative to dense self-attention in discrete, looped, and continuous-depth Vision Transformers. SSoG replaces pairwise query-key interactions with a learned mixture of separable Gaussian spatial kernels, reducing the asymptotic spatial-mixing complexity from quadratic to subquadratic in the number of tokens. The paper further establishes global Lipschitz continuity for fixed SSoG and Lipschitz continuity on bounded token-norm domains for its content-steered variant, LookAt. Experiments on CIFAR-100 and ImageNet-1K show competitive or improved classification accuracy, while high-resolution evaluations demonstrate throughput advantages in several settings. The work provides a theoretically motivated study of efficient spatial mixing, particularly for looped and continuous-depth vision architectures.

Critical strengths

The paper connects the Lipschitz properties of the token-mixing operator to the requirements of continuous-depth and repeatedly applied Transformer architectures, providing a more principled motivation than computational efficiency alone.

The separable Gaussian formulation avoids constructing a dense token-to-token attention matrix and achieves subquadratic spatial-mixing complexity for a fixed number of Gaussian components.

The method is evaluated in discrete, looped, and continuous-depth Transformers, offering useful evidence about its applicability beyond a single architecture.

Critical weaknesses
  1. Limited novelty relative to prior SSoG work: The paper builds on an existing SSoG operator and its fixed and content-steered variants. The main contribution is its theoretical analysis and application to looped and continuous-depth Transformers. The paper should more clearly establish the conceptual and technical advances beyond the original proposal.

  2. Insufficient evidence linking theory to empirical benefits: The Lipschitz analysis is relevant to continuous-depth models, but the experiments do not fully establish that the theoretical guarantee itself leads to improved optimization stability or robustness. More direct empirical evidence would strengthen this connection.

  3. Implementation-dependent efficiency gains: Although SSoG has a better asymptotic complexity, its practical speed and memory advantages vary across architectural settings and resolutions. The comparison is also affected by the use of optimized fused attention kernels for the dense-attention baselines and less optimized operations for SSoG. This makes it difficult to isolate the benefits of the operator itself from implementation effects.

  4. Limited task diversity: The evaluation focuses primarily on image classification. It remains unclear whether the proposed operator preserves the benefits of content-dependent global interactions on more demanding tasks, such as dense prediction or video modeling.

  5. Trade-offs of content steering: LookAt introduces additional input-dependent flexibility but also increases computational and training costs, and it does not consistently outperform fixed SSoG across all settings. The conditions under which content steering is beneficial deserve further investigation.

Questions for authors
  1. Novelty: Could the authors clarify the main technical contributions beyond the original SSoG proposal, particularly regarding the theoretical analysis and its integration into looped and continuous-depth Transformers?

  2. Theoretical implications: Can the authors provide direct evidence that the Lipschitz guarantees improve optimization stability, sensitivity to initialization, or robustness in continuous-depth models, beyond the observed classification results?

  3. Efficiency and implementation: How much of the reported speedup comes from the asymptotic complexity of SSoG rather than differences in kernel optimization? Could the authors provide comparisons using optimized implementations of both SSoG and dense attention, including end-to-end latency and peak memory?

  4. Content steering: Under what conditions does LookAt consistently outperform fixed SSoG? Can the authors explain why content steering improves some settings but provides limited or negative gains in others, particularly in the continuous-depth setting?

  5. Generalization to other tasks: Have the authors evaluated SSoG on dense prediction or video-related tasks, where spatial interactions and content-dependent relationships may be more complex than in image classification?

2 comments
AuthorsOpenReview ↗

Response FjRV

W1/Q1: Novelty. We agree that the operator and its two variants are due to Pisoni et al. (2026). However, the citing is over a blog-page. We considered the idea to have potential and room for improvement and we settled the theoretical basis and baselines to further build on top of that. This paper adds, first, the theory (Appendix A.9, not studied before): fixed SSoG is globally Lipschitz (Proposition 1, via Lemma 1); steered SSoG is Lipschitz on balls of radius ρ (Proposition 2, via Lemma 2), against ρ² for dot-product attention (Kim et al., 2021). Second, its first use as a looped mixer and as the vector field of a NeuralOde, where this property matters. Third, an efficiency study against fused and compiled kernels (W3), plus new comparisons with other mixers, dense prediction and early exiting (see response to ZauD), as a proof to show that building on top is worth for the community.

W2/Q2: Theory and empirical benefits. We added training-free probes (future Appendix A.10). We test this directly by scaling the real inputs of each mixer by c ∈ [0.25, 16] and estimating the spectral norm of its Jacobian. The measured growth follows the predicted order: the norm is constant for fixed SSoG (fitted exponent 0.00), grows with a local slope of 0.5–1.1 for steered SSoG, and grows with an exponent of 1.4–1.9 for dense attention.

We further compare the solution obtained with the training solver (Euler) with a different solver (RK4) same NFE. For fixed SSoG the two final states differ by 0.072 (relative norm), against 0.305 for dense, and the accuracy for SSoG drops 4.2, against 10.5 for dense, which we see as another proof of robustness complementary to patch size experiment Tab.14. Along the trajectory, the average token norm barely changes with fixed SSoG (from 194 to 202), whereas with dense attention it triples (from 192 to 579). Finally, Fixed SSoG also cannot form attention sinks (Appendix A.2).

Togheter, these results are consistent with the theory.

W3/Q3: Operator or implementation? We agree; another reviewer raised the same point (see our reply to Reviewer mcUY). All dense baselines run fused FlashAttention, while SSoG runs as eager PyTorch. With both operators compiled (Appendix A.1), SSoG keeps 25–30% higher peak throughput at 1024×1024 in all settings, while compiled dense attention is faster below 512–640 px. Independently of the implementation, SSoG needs 53% fewer attention FLOPs at 224×224 and 82% fewer at 1024×1024.

W4/Q5: Other tasks. See also our reply to Reviewer ZauD (B). Spatial mixing never uses the class token, so the backbone gives ordinary per-patch features. We fine-tuned the ImageNet backbones of all three settings for semantic segmentation on bigger images without any class token to also show the computational gains (will be the Appendix A.7). Each cell gives mIoU / training hours:

Dataset, inputSettingDenseFixed SSoGSteered SSoG
ADE20K, 512 pxDiscrete41.1 / 7.538.5 / 7.940.8 / 11.4
DocLayNet, 512 pxDiscrete77.6 / 6.177.3 / 5.678.6 / 10.2
DocLayNet, 512 pxLooped78.2 / 5.677.1 / 5.077.2 / 10.0
DocLayNet, 512 pxContinuous73.1276.3 / 6.272.8 / 11.2
Beyond Words, 1024 pxDiscrete70.5 / 3.071.6 / 2.470.3 / 5.8
Beyond Words, 1024 pxLooped70.4 / 2.871.4 / 2.169.7 / 5.7
Beyond Words, 1024 pxContinuous65.766.2 / 2.666.4 / 6.2

Cost of the backbone (discrete; the three settings differ by less than 0.3 GFLOPs):

DenseFixed SSoGSteered SSoG
GFLOPs per image, 512 px214.1151.1 (−29%)155.6 (−27%)
GFLOPs per image, 1024 px1319.6623.6 (−53%)641.7 (−51%)
Parameters, whole model93.2M78.9M81.2M

W5/Q4: When does steering help? When one spatial pattern per head is not enough. Fixed SSoG applies the same window at every query. That suits classification, where features are pooled anyway. The LookAt was an extension of the fixed, and we added the expetimentation to validate it, however we agree that is unclear where it specifically helps. We hypothesize that in dense prediction where the objects have irregula shape, the LookAt should work better. Fine-tuned on ADE20K, fixed SSoG trails dense attention by 2.6 mIoU, and steering recovers 2.2 of these points (W4 table). On documents, steering is more varied (−3.5 to +1.3 mIoU), plausibly because page layouts are axis-aligned regions that the fixed kernel already fits, while natural shapes are irregular. We therefore recommend steering for dense prediction on natural images in the scope of this paper.

I think that the additional analysis and experiments provided by the authors have addressed my concerns. Thanks.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Paper Updated with Experiments and Revisions

Dear reviewers,

Thank you for your comments and for the concerns you raised about our paper. We have just uploaded a revised version that includes the changes and new experiments addressing these points. We are glad to see that these revisions and experiments confirm and further strengthen the claims of the paper.

We hope the revision resolves your concerns, and we thank you again for your time and feedback throughout the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.