acceptodds
Under review as a conference paper at ICLR 2027

Flash EQ-Linear: Accelerating Equivariant Linear Layers via Group-wise Discrete Fourier Transform

Abstract

Equivariant networks embed geometric symmetries as structural priors through weight sharing, achieving remarkable parameter efficiency across vision tasks. However, this parameter efficiency does not translate into compute efficiency: existing implementations unroll the structured weights into dense matrices and dispatch them to generic dense kernels, so an equivariant layer costs no fewer MACs than its non-equivariant counterpart. In this paper, we observe that the equivariant linear (EQ-Linear) layer—the most fundamental and frequently used module in modern equivariant architectures—is essentially a circular convolution along the group dimension composed with a linear transform along the channel dimension. Building on this observation, we propose Flash EQ-Linear, an exact acceleration algorithm that reduces the cost to of the original dense formulation ( is the equivariant group size) by combining the Fourier convolution theorem along the group dimension with the conjugate symmetry of the real DFT. To translate these computational savings into wall-clock speedups, we further develop dedicated CUDA kernels covering forward and backward passes in FP32 and FP16. At the operator level, Flash EQ-Linear achieves up to forward speedup over PyTorch's highly optimized F.linear; at the network level, Flash EQ-ViT achieves up to end-to-end speedup over both equivariant and non-equivariant baselines. As an operator-level acceleration algorithm, Flash EQ-Linear provides plug-and-play acceleration for diverse pretrained equivariant models, including EQ-ViT, EQ-Swin, EQ-VMamba, and EQ-INR, without retraining or architectural changes. Code is available at https://anonymous.4open.science/r/FlashEQLinear-46D1.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Equivariant network acceleration, Equivariant linear layers, Low-level vision

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FgJzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Flash EQ-Linear, an efficient implementation of equivariant linear layers for cyclic-group regular representations. It uses Fourier transforms and conjugate symmetry to simplify equivariant computation, while fused CUDA kernels further improve runtime efficiency. Experiments on equivariant ViTs and other architectures demonstrate substantial latency and throughput improvements without retraining or accuracy degradation.

Critical strengths
  1. The paper is clearly written and well organized。

  2. The CUDA implementation is carefully optimized and translates the theoretical reduction in computation into substantial practical speedups.

Critical weaknesses
  1. The proposed acceleration is limited to cyclic-group regular representations, and its extension to non-abelian groups such as DND_N is not established. Such groups require matrix-valued Fourier blocks rather than the simple scalar-frequency decomposition exploited by Flash EQ-Linear.

  2. The theoretical contribution of this paper is limited, as Quick ViT already introduced a framework for accelerating equivariant linear layers based on Fourier transformation. Flash EQ-Linear can be viewed as its specialization to cyclic groups, and reinterpret through Fourier analysis and convolution.

  3. For the experiments comparing against Quick ViTs, the paper does not report the training recipes for either the Quick ViT baselines or the Naive EQ-ViT checkpoints used by Flash EQ-ViT. Moreover, Table 5 does not provide a compute-matched comparison, as the Quick ViTs use substantially fewer FLOPs and parameters, which may partly explain their lower accuracy.

Questions for authors

see weakness

10 comments

We sincerely thank the reviewer for appreciating the clear presentation and carefully optimized CUDA implementation, and for recognizing the practical speedups. In the revision, we have extended Flash EQ-Linear to the non-abelian group D8D_8, clarified our contributions relative to Quick ViTs, and added comparisons at matched parameter and FLOP budgets.

The revised manuscript has been uploaded (with all changes highlighted in red), and our point-by-point responses are detailed below.

Weaknesses 1: Limited to cyclic-group regular representations, the extension to non-abelian groups such as DND_N is not established.

Response 1: We sincerely thank the reviewer for this insightful comment regarding the scope of our group formulations. Before introducing our extension to dihedral groups, we first wish to respectfully emphasize the standalone practical value of our initial cyclic-group formulation. Finite cyclic groups (especially p4p_4 and p8p_8) form the foundation of the most widely deployed regular-representation equivariant networks in 2D vision, such as G-CNN, GSA-Net, EQ-ViT, and EQ-VMamba. These architectures have already demonstrated state-of-the-art performance across diverse real-world applications. Therefore, providing a training-free, operator-level acceleration specifically tailored for cyclic groups inherently delivers substantial, immediate practical value to the computer vision community.

Nevertheless, we fully agree with the reviewer that extending our framework to non-abelian groups (such as DND_N) is a crucial and highly meaningful generalization. We are pleased to report that our methodology is indeed fully extensible to these cases. In the revised manuscript, we have successfully extended our derivation to general dihedral groups—utilizing matrix-valued Fourier blocks as you perceptively noted—and implemented the corresponding fused CUDA inference kernels for the non-abelian group D8D_8.

The measured end-to-end speedups for D8D_8 are highly significant:

ModelFP32 speedupFP16 speedup
Flash EQ-Linear-D84.6x4.3x
Flash EQ-ViT-D8-H2.2x1.6x

We also extend the derivation to general dihedral groups, following Algorithm 1's transform–multiply–inverse-transform structure with matrix-valued Fourier blocks, as detailed below.

1) Setup. Denote the dihedral group DND_N as D2nD_{2n} of order 2n2n, with nn rotations and nn reflections ordered as (e,r,…,rn−1,s,rs,…,rn−1s),(e,r,\ldots,r^{n-1},s,rs,\ldots,r^{n-1}s), where ee is the identity, rr rotates by 2π/n2\pi/n , and ss is a reflection. For one token and input–output channel pair, collect the real inputs, outputs, and weights into x,y,w∈R2nx,y,w\in\mathbb R^{2n} . The EQ-Linear layer computes yg=∑h∈D2nWg−1hxh,y_g=\sum_{h\in D_{2n}}W_{g^{-1}h}x_h, Split the input and output into rotation and reflection halves: xR=(x0,…,xn−1)⊤, xS=(xn,…,x2n−1)⊤.x_R=(x_0,\ldots,x_{n-1})^\top,~x_S=(x_n,\ldots,x_{2n-1})^\top.Thus, the layer becomes

\quad A,B\in\mathbb{R}^{n\times n}\ \text{circulant},$$ with $A_{t,u}=W_{(u-t)\bmod n}, ~~B_{t,u}=W_{n+(u-t)\bmod n}, ~~t,u=0,\ldots,n-1$. **2) Algorithm R3: Flash EQ-Linear for $D_{2n}$** **Step 1 — Real group Fourier transform.** Apply an $n$-point DFT to each half, $U_k=\sum_{t<n}x_t e^{-2\pi ikt/n}$, $V_k=\sum_{t<n}x_{n+t}e^{-2\pi ikt/n}$, with $U_{n-k}=\overline{U_k}, ~V_{n-k}=\overline{V_k}$. The independent components include $q$ real scalars ($q=4$ even $n$, $2$ odd) plus $m=\lfloor(n-1)/2\rfloor$ real $2\times2$ matrices, giving $q+4m=2n$ coordinates. For $k=1,\dots,m$, $$\hat X^{(k)}=\begin{pmatrix}Re(U_k)+Re(V_k) & Im(U_k)-Im(V_k)\\\\ -Im(U_k)-Im(V_k) & Re(U_k)-Re(V_k)\end{pmatrix}.$$ Stacking scalars then matrix entries gives $\hat{x}=\mathbf{T}x, \mathbf{T}\in\mathbb{R}^{2n\times2n}$, which uses only additions. Since $\mathbf T$ block-diagonalizes $M$, the multiplication in Step 2 is block-wise. **Step 2 — Multiply the Fourier blocks.** Scalars are independent, $\hat y_j=\hat x_j\hat w_j$; matrices multiply as $\hat Y^{(k)}=\hat X^{(k)}\hat W^{(k)}$. For $D_8$, $$\begin{pmatrix}\hat y_4&\hat y_5\\\\ \hat y_6&\hat y_7\end{pmatrix} =\begin{pmatrix}\hat x_4&\hat x_5\\\\ \hat x_6&\hat x_7\end{pmatrix} \begin{pmatrix}\hat w_4&\hat w_5\\\\ \hat w_6&\hat w_7\end{pmatrix},$$ so the four scalar products plus one $2\times2$ matrix product cost $4+4\times2=12$ real multiplications. This computation follows directly from the identity $\mathbf{T}M\mathbf{T}^{-1}=diag\big(\hat w_0,\hat w_1,\hat w_2,\hat w_3,(\hat W^{(1)})^{\top},(\hat W^{(1)})^{\top}\big)$. **Step 3 — Inverse real group Fourier transform.** Because the rows of $\mathbf T$ are orthogonal, the inverse is $$y=\mathbf T^\top D^{-1}\hat{y}, ~~D=\mathbf T\mathbf T^\top=diag(2nI_q,nI_{4m}),$$ where $I_d$ is the $d\times d$ identity matrix. Add the original shared bias afterward. These steps recover $Mx$ in exact arithmetic. **3) Theoretical MAC reduction.** Per token and input–output channel pair, the core multiplication count is $$q+8m=\begin{cases}4n-4,&n\text{ even},\\\\ 4n-2,&n\text{ odd},\end{cases}$$ versus $(2n)^2$ for the dense layer. For $D_8$ the reduction is $64/12\approx5.33\times$. **Revision.** We have added the dihedral-group derivation, explicit $D_8$ transforms, and the operator-level benchmarks and end-to-end Flash EQ-ViT-$D_8$ evaluations in Appendices B and E.

Weaknesses 2: The theoretical contribution beyond Quick ViT is limited.

Response 2: We sincerely thank the reviewer for pointing this out. We fully agree that the representation-theoretic foundation (Fourier block-diagonalization) was elegantly established in Quick ViTs. Accordingly, we have revised the manuscript to explicitly attribute this foundational observation to them and narrowed our novelty claims.

However, we would like to kindly clarify that, as insightfully noted by Reviewer 5aQV, our contribution lies in translating the theoretical observation into a highly practical acceleration paradigm, which enables training-free operator replacement and unlocks practical speedups for small models. Specifically, beyond the theoretical observation in Quick ViTs, our contributions are fourfold:

1) Operator-level, training-free replacement across equivariant architectures. Unlike Quick ViTs which keep features in the Fourier domain throughout the network (requiring architectural redesign and retraining), Flash EQ-Linear strictly preserves the regular-representation input and output interfaces of each layer, changing bases only internally. Thus, a pretrained model can be accelerated entirely out-of-the-box by replacing its EQ-Linear layers, with no retraining, no architectural change, and exactly zero accuracy degradation. This operator-level design provides a general and easy-to-use acceleration paradigm for the community.

2) Fused CUDA kernels that make the savings practical at small sizes. Turning theoretical FLOP reduction into measured wall-clock speedup matters immensely in practice. Quick ViTs leave the practical speedup in the small-model regime open: Bökman et al., 2025 [1] report that "for small models, such as ViT-S and ConvNeXt-S, the equivariant versions have a worse throughput" and state that "custom GPU-code is likely required"; Nordström et al., 2026 [2] attribute this gap to insufficient arithmetic intensity at low channel widths. We completely close this gap on small models with dedicated fused CUDA kernels, successfully yielding end-to-end speedups at every model scale, in both precisions.

3) Explicit arithmetic and efficient implementation for finite cyclic groups. While Quick ViTs focus on reflection and dihedral groups (e.g., D2D_2 and D8D_8), whose irreducible representations are entirely of real type, we derive the explicit arithmetic framework for finite cyclic groups (e.g., p4,p8p_4, p_8). The real Fourier decomposition of these cyclic groups inherently contains complex-type components. We explicitly model these carefully coupled real and imaginary computations. Establishing this explicit arithmetic for complex-type representations is mathematically distinct from the real-type block diagonalizations used in [1, 2].

For complex-type blocks, Gauss's three-product identity reduces each complex matrix product from 4 real matrix multiplications to 3:

\Updownarrow&\\\\ T_1=Re(\hat X)Re(\hat W)^\top, ~~T_2=Im(\hat X)Im(\hat W)^\top, ~~&T_3=\bigl(Re(\hat X)+Im(\hat X)\bigr)\bigl(Re(\hat W)+Im(\hat W)\bigr)^\top,\\\\ Re(\hat Y)=T_1-T_2, ~~&Im(\hat Y)=T_3-T_1-T_2.\end{aligned}$$ This would push the theoretical FLOPs reduction bound even higher—from $16/6$ to $16/5=3.20\times$ for $p_4$, and from $64/14$ to $64/11 \approx 5.82\times$ for $p_8$. **Beyond cyclic groups, our operator-level framework seamlessly accommodates the dihedral groups.** We provide explicit cyclic-group arithmetic and a fused operator implementation for complex-type components. Our plug-and-play formulation is fully compatible with the real-type irreducible representations of non-abelian groups. We have explicitly extended our derivation to general dihedral groups and implemented the corresponding fused CUDA kernels for $D_8$. **4) Comprehensive empirical validation demonstrating the universal viability of equivariant networks.** By conducting plug-and-play experiments across diverse architectures, ranging from standard Transformers (EQ-ViT, EQ-Swin) to State Space Models (EQ-VMamba) and Implicit Neural Representations (EQ-INR), we prove that regular-representation equivariant methods can simultaneously deliver superior accuracy and faster inference across modern vision architectures. This breaks the stereotype that equivariance inevitably trades speed for robustness, making a significant contribution to the broader adoption of equivariant networks in latency-critical, real-time image processing scenarios, such as remote sensing (which heavily relies on $p_4$ and $p_8$ equivariance) and autonomous driving (which utilizes $D_2$ equivariance). **Revision.** We have clarified the contribution statements and attribution to Quick ViTs, expanded the arithmetic analysis, and added the dihedral-group derivation and corresponding $D_8$ experiments.
Reviewer_5aQVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Flash EQ-Linear, an exact acceleration algorithm for Linear layers in equivariant networks. It relies on the fact that, in the frequency domain, group convolution become pointwise multiplications. Whie many equivariant networks have stayed only in the spatial domain, giving no compute savings from the symmetry-induced weight sharing, this paper achieves 2(T−1)/T22(T −1)/T^2 fewer FLOPs than a dense Linear layer. To translate FLOP savings to wall-clock speed up, the authors make various engineering efforts to ensure fast GPU kernels. The authors validate this speedup on various model configurations and show that existing EQ-ViT models can use this method to speed them up, without any retraining.

Critical strengths
  • The paper proposes a practically useful method that has rigorous theoretic backing as well as extensive engineering
  • It is very nice to be able convert existing equivariant networks using this method and thus speeding them up without having to retrain
  • The speedups seem to work well even for smaller models, which is not the case without efficient kernels
  • The paper is quite well-written and makes the point of the paper clear reasonably well I think
Critical weaknesses
  • Novelty: The paper frames itself in the Introduction and the Abstract as being the first to come up with the idea that most equivariant networks are doing it wrong and not leveraging the compute savings that are possible. For example,

In this paper, we observe that the equivariant linear (EQ-Linear) layer—the most fundamental and frequently used module in modern equivariant architectures—is essentially a circular convolution along the group dimension composed with a linear transform along the channel dimension.

However, as the authors briefly note in the paper, this has already been observed by Bökman et. al [1] and Nordström et. al [2]. While this does not invalidate the paper's contributions, it requires a different framing to the reader. Now instead, the novelty of the paper, in my estimation, is that you realize that this observation can be used for speeding up existing models, without retraining, by only converting to and from the Fourier domain in the linear layer (instead of being in the Fourier domain all of the time, except in the non-linearities, as is done in [1, 2]). Furthermore, the paper makes this work good for small models by efficient kernels, which [1, 2] does not do.

  • Experimental rigor: In [1, 2 ] they explicitly state that small models don't work very well with the block diagonal matrices on GPUs and make much of the argument about scaling the models. Yet, your comparison is only at the smallest scale.

  • Only considering p4: While not critical, I think the paper would benefit from additional groups. For example, having also reflections would allow for a better direct comparison to [1, 2].

[1] G. Bökman, D. Nordström, F. Kahl. Flopping for FLOPs: Leveraging equivariance for computational efficiency, in ICML 2025

[2] D. Nordström, J. Edstedt, F. Kahl, Georg Bökman. Quick ViTs: Speeding up Vision Transformers through Equivariance, in ECCV 2026

Questions for authors
  • How do your models, in terms of speed and FLOPs, compare to [1] and [2] when scaled to hundreds of millions or even billions of parameters? I advice you to look at both FP16 and FP32.
  • Can you enlighten me about the relation between you formula, 2(T−1)/T22(T −1)/T^2, and the one used in [2] that calculates that the Linear layers for the D8D_8 group save 16/316/3 times the FLOPs? I am curious about the connection. I find your section A.2 quite enlightening. Maybe some of this can be included in the main manuscript?
  • What makes your implementation so quick for small models and why do the time savings not seem to increase as the channel dimension increases?

Overall, I think the paper has potential. I implore the authors to frame their contributions in a more transparent light and provide additional comparisons. The main contributions I see, as I mentioned earlier are: (i) realizing that this can be done only for the linear layers to avoid retraining and (ii) making it work for small models through clever engineering. I ask the authors to please clarify if there are any additional novelties that I am missing. I remain open minded in-terms of scoring of the paper.

[1] G. Bökman, D. Nordström, F. Kahl. Flopping for FLOPs: Leveraging equivariance for computational efficiency, in ICML 2025

[2] D. Nordström, J. Edstedt, F. Kahl, Georg Bökman. Quick ViTs: Speeding up Vision Transformers through Equivariance, in ECCV 2026

5 comments

We sincerely thank the reviewer for recognizing our method as "practically useful," with "rigorous theoretic backing as well as extensive engineering," and for highlighting its ability to accelerate existing equivariant networks "without having to retrain" and "even for smaller models." Following the suggestion, we have narrowed our novelty claims and clarified our contributions in relation to prior work.

The revised manuscript has been uploaded (with all changes highlighted in red), and our point-by-point responses are detailed below.

Weaknesses 1: The novelty should be framed more clearly relative to prior work [1, 2].

Response 1: We sincerely thank the reviewer for this constructive feedback and for accurately summarizing our core practical contributions. We fully agree that the representation-theoretic foundation (Fourier block-diagonalization) was elegantly established in [1, 2]. Accordingly, we have revised the Abstract and Introduction to explicitly attribute this foundational observation to [1, 2], completely removing the "our key insight" framing.

Following your insightful suggestions, we have reframed our main text to explicitly emphasize the following contributions:

1) Operator-level, training-free replacement across equivariant architectures. Unlike [1, 2] which keep features in the Fourier domain throughout the network, Flash EQ-Linear strictly preserves the regular-representation input and output interfaces of each layer, changing bases only internally. Consequently, a pretrained model can be accelerated entirely out-of-the-box by replacing its EQ-Linear layers, with no retraining, no architectural change, and exactly zero accuracy degradation. This operator-level design provides a general and easy-to-use acceleration paradigm for the community.

2) Fused CUDA kernels that make the savings practical at small sizes. Turning theoretical FLOP reduction into measured wall-clock speedup matters immensely in practice. Both [1] and [2] leave the practical speedup in the small-model regime open: [1] reports that "for small models, such as ViT-S and ConvNeXt-S, the equivariant versions have a worse throughput" and states that "custom GPU-code is likely required"; [2] attributes this gap to insufficient arithmetic intensity at low channel widths. We completely close this gap on small models with dedicated fused CUDA kernels, successfully yielding end-to-end speedups at every model scale, in both precisions.

3) Explicit arithmetic and efficient implementation for finite cyclic groups. While [1, 2] focus on reflection and dihedral groups (e.g., D2D_2 and D8D_8), whose irreducible representations are entirely of real type, we derive the explicit arithmetic framework for finite cyclic groups (e.g., p4,p8p_4, p_8). The real Fourier decomposition of these cyclic groups inherently contains complex-type components. We explicitly model these carefully coupled real and imaginary computations. Establishing this explicit arithmetic for complex-type representations is mathematically distinct from the real-type block diagonalizations used in [1, 2].

In the revision, we have further shown that this complex-type formulation opens a promising direction for future optimization. Specifically, by applying Gauss's three-multiplication trick for complex numbers, the cost per complex product can theoretically be reduced from 4 real multiplications to 3:

\Updownarrow&\\\\ T_1=Re(\hat X)Re(\hat W)^\top, ~~T_2=Im(\hat X)Im(\hat W)^\top, ~~&T_3=\bigl(Re(\hat X)+Im(\hat X)\bigr)\bigl(Re(\hat W)+Im(\hat W)\bigr)^\top,\\\\ Re(\hat Y)=T_1-T_2, ~~&Im(\hat Y)=T_3-T_1-T_2.\end{aligned}$$ This would push the theoretical FLOPs reduction bound even higher—from $16/6$ to $16/5=3.20\times$ for $p_4$, and from $64/14$ to $64/11 \approx 5.82\times$ for $p_8$. While our current main benchmarks retain the standard 4-multiplication approach, this demonstrates the significant remaining potential of our arithmetic framework. **Beyond cyclic groups, our operator-level framework seamlessly accommodates the dihedral groups established by [1, 2].** We have explicitly extended our derivation to general dihedral groups and implemented the corresponding fused CUDA kernels for $D_8$ (please refer to our response to Reviewer FgJz for details). This structural alignment with [1, 2] demonstrates the broad versatility of our acceleration framework. **4) Comprehensive empirical validation demonstrating the universal viability of equivariant networks.** By conducting plug-and-play experiments across diverse vision architectures, ranging from Transformers (EQ-ViT, EQ-Swin) to State Space Models (EQ-VMamba) and Implicit Neural Representations (EQ-INR), we prove that regular-representation equivariant methods can simultaneously deliver superior accuracy and faster inference. This breaks the stereotype that equivariance inevitably trades speed for robustness, making a significant contribution to the broader adoption of equivariant networks in latency-critical, real-time image processing scenarios, such as remote sensing (which heavily relies on $p_4$ and $p_8$ equivariance) and autonomous driving (which utilizes $D_2$ equivariance). **Revision.** We have rewritten the Abstract and Introduction to attribute the observation to [1, 2] and to state the contributions above.

Weaknesses 2: The comparison lacks larger-scale models.

Response 2: As suggested, we have extended the comparison with Quick ViTs to all five scales (Tiny to Huge) in both FP32 and FP16. Furthermore, to ensure an absolutely fair and direct comparison, we explicitly implemented Flash EQ-ViT-D8D_8. This perfectly matches Quick ViT-D8D_8 in parameter count and FLOPs.

MethodPrecisionThroughput (Tiny)Throughput (Small)Throughput (Base)Throughput (Large)Throughput (Huge)
Quick ViT-D8FP323294 (1.2×)2249 (1.4×)1389 (1.6×)504 (1.9×)265 (1.9×)
Flash EQ-ViT-D8FP324444 (1.6×)3030 (1.8×)1743 (2.0×)546 (2.0×)308 (2.2×)
Quick ViT-D8FP163229 (0.4×)3449 (0.7×)3388 (1.3×)1255 (1.6×)685 (1.6×)
Flash EQ-ViT-D8FP169297 (1.2×)6029 (1.2×)3847 (1.4×)1319 (1.7×)688 (1.6×)

1) Small scales. Consistent with the insightful scaling observations discussed in [1, 2] (i.e., hardware utilization drops at low arithmetic intensity), the baseline Quick ViT-D8D_8 exhibits limited throughput at smaller scales in FP16 (0.4×0.4\times and 0.7×0.7\times). Flash EQ-ViT-D8D_8 stays faster than the standard ViT at both scales (1.2×1.2\times), and is 2.9×2.9\times and 1.7×1.7\times faster than Quick ViT-D8D_8.

2) Large scales. Both implementations achieve greater speedups at larger scales, consistent with the scaling behavior discussed in [1, 2]. At Large and Huge scales, Flash EQ-ViT-D8D_8 (2.0×2.0\times–2.2×2.2\times) slightly outperforms Quick ViT-D8D_8 (1.9×1.9\times) in FP32, while both models achieve comparable speedups in FP16.

Importantly, we would like to clarify that our formulation does not claim algorithmic superiority over [1, 2] on the D8D_8 group. Theoretically, our matrix-valued Fourier block implementation for D8D_8 shares the exact same computational complexity as the framework proposed in [1, 2] (both achieving a theoretical FLOP reduction of 16/3×16/3 \times). Therefore, the empirical gains shown above—particularly the significant throughput advantages at smaller scales and the highly comparable results at larger scales—stem primarily from our fused CUDA kernel optimizations. These results simply demonstrate that our operator-level, plug-and-play design achieves the same theoretical optimality as previous work on D8D_8, while our hardware-level engineering ensures this theoretical ceiling is fully realized into wall-clock speedups across all model scales.

Revision. We have expanded Table 6 to cover all five model scales in both FP32 and FP16, including the D8D_8 variant of Flash EQ-ViT.

Weaknesses 3: Evaluation is limited to p4p_4, with no experiments on D8D_8.

Response 3: We have extended the CUDA kernels for D2D_2, p4p_4, p8p_8, D8D_8, and implemented Flash EQ-ViT-D8D_8, so the comparison with [1, 2] can now be made for the same group and the same arithmetic budget. In FP32, Flash EQ-ViT-D8D_8 is faster than Quick ViT-D8D_8 at every one of the five scales (44444444 vs. 32943294 imgs/s at Tiny, 308308 vs. 265265 at Huge), and in FP16 the two are on par at Large and Huge while ours stays faster at Tiny and Small (Tables 19 and 20). The comparison results are summarized below.

ModelParams (M)FLOPs (G)Top-1 (%)FP32 throughput (speedup)FP16 throughput (speedup)
ViT-T21.79.276.52796 (1.0×1.0\times)7794 (1.0×1.0\times)
Quick ViT-D8-T2.81.771.63294 (1.2×1.2\times)3229 (0.4×0.4\times)
Flash EQ-ViT-D8-T2.81.771.84444 (1.6×1.6\times)9297 (1.2×1.2\times)
ViT-S33.814.176.51658 (1.0×1.0\times)5030 (1.0×1.0\times)
Quick ViT-D8-S4.32.676.32249 (1.4×1.4\times)3449 (0.7×0.7\times)
Flash EQ-ViT-D8-S4.32.676.33030 (1.8×1.8\times)6029 (1.2×1.2\times)
ViT-B85.935.177.1881 (1.0×1.0\times)2688 (1.0×1.0\times)
Quick ViT-D8-B10.86.574.71389 (1.6×1.6\times)3388 (1.3×1.3\times)
Flash EQ-ViT-D8-B10.86.577.41743 (2.0×2.0\times)3847 (1.4×1.4\times)
ViT-L303.4123.178.4269 (1.0×1.0\times)795 (1.0×1.0\times)
Quick ViT-D8-L38.0622.677.1504 (1.9×1.9\times)1255 (1.6×1.6\times)
Flash EQ-ViT-D8-L38.0622.678.5546 (2.0×2.0\times)1319 (1.7×1.7\times)
ViT-H631.0254.781.3139 (1.0×1.0\times)420 (1.0×1.0\times)
Quick ViT-D8-H79.0746.869.3265 (1.9×1.9\times)685 (1.6×1.6\times)
Flash EQ-ViT-D8-H79.0746.878.7308 (2.2×2.2\times)688 (1.6×1.6\times)

In FP32, Flash EQ-ViT-D8D_8 is faster than Quick ViT-D8D_8 at every one of the five scales (44444444 vs. 32943294 imgs/s at Tiny, 308308 vs. 265265 at Huge), and in FP16 the two are on par at Large and Huge while ours stays faster at Tiny and Small.

Revision. We have (i) added CUDA kernels for D2D_2, p4p_4, p8p_8 and D8D_8, and integrated them into Flash EQ-ViT; (ii) added the operator-level scaling study over T∈{2,4,8}T\in\{2,4,8\} and D8D_8 (Tables 16 and 17), together with the matched-budget comparison against Quick ViTs in FP32 and FP16 (Tables 19 and 20).

Reviewer_ryQDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper considers the problem of the efficiency of existing linear equivariant layers. The authors highlight a significant problem in the equivariant neural network domain: although many works focus on the parameter efficiency of equivariant architectures by modifying the architectures themselves, this usually does not result in faster performance because of naive implementations. The authors aim to modify the implementation of the basic EQ-Linear layers (which they identify as the most fundamental and frequently used) so that the resulting models outperform their non-equivariant counterparts in task metrics, parameter efficiency (number of parameters), and inference speed. Their speed-up solution is based on the fact that EQ-Linear is a circular convolution along the group dimension composed with a linear transform along the channel dimension. Using this fact, the authors propose an algorithm (which works for any finite group) that applies the Fourier convolution theorem, and develop CUDA kernels for the implementation (in the particular case of the group p4).

Critical strengths
  1. I like that the acceleration is obtained not by heuristics or modifications to numerous architectures, but from the algebraic structure of the weight matrices. The approach can be plugged into existing architectures that use EQ-Linear layers for the p4 group, and these architectures can benefit immediately.
  2. The theoretical result about the form of the weight matrices (Section 3.1) and the accelerated algorithm are obtained for any finite group.
Critical weaknesses
  1. The proposed insight about the structure of the equivariant weight matrices works only for EQ-Linear layers and only for finite groups. I see some limitations in the scope of the work in this area. I identified these limitations while reading the main sections, and then noticed that the authors partially mention them in the Limitations section. In my opinion, this somewhat limits the value of the results. The paper does not address whether the same ideas can be exploited for gated or norm nonlinearities, which are the standard equivariant activations. EQ-Linear layers are important without any doubt, but, from my experience, nonlinear layers are at least as important and valuable for equivariant architectures. As far as I can see now, the results cannot be easily generalized to nonlinear equivariant layers.
  2. The practical implementation of CUDA kernels based on the obtained insight about the structure of the weight matrices is currently done only for the p4 group. For other groups, the kernels would need to be derived case-by-case and implemented by practitioners, which limits applicability. This limitation is also mentioned by the authors, but I still regard it as significant.
Questions for authors

My questions are mostly about my significance concerns. Although I liked the proposed approach, I doubt that its impact is broad enough for the community:

  1. Can a similar approach be applied to nonlinear equivariant layers? What are the main difficulties?
  2. Can a general approach for deriving CUDA kernels for an arbitrary equivariance group be suggested?
  3. Can the approach be modified for the case of infinite dimensional groups (e.g. SO(3), O(3), which are very important for applications)?
6 comments

We sincerely thank the reviewer for appreciating the "lossless acceleration principle and plug-and-play capability" of our algorithm. We understand that the presentation in our initial manuscript might have caused concerns regarding the breadth of our method's impact. In short, we want to clarify that our method covers a wide range of existing 2D equivariant models and is fully compatible with their existing non-linear layers. Furthermore, by turning theoretical FLOP reductions into actual wall-clock speedups, our method makes the computational-efficiency advantage of equivariant architectures hold in practice. Specifically, this plug-and-play method delivers consistent speedups across diverse model scales and architectures (e.g., Transformers, Mamba, INRs).

The revised manuscript has been uploaded (with all changes highlighted in red), and our point-by-point responses are detailed below.

Weaknesses 1: The method is limited to EQ-Linear layers (it does not cover nonlinear layers) and to finite groups.

Response 1: We thank the reviewer for pointing this out and appreciate the opportunity to clarify the scope of our method. Equivariant networks generally follow two main lines of research, distinguished by how features are stored: regular-representation features (e.g., G-CNN, GSA-Net, EQ-ViT, and EQ-VMamba) and irreducible-representation features (e.g., 3D Steerable CNNs, SE(3)-Transformer, and Quick ViTs). Flash EQ-Linear specifically targets the former, regular-representation equivariant models. This class of models includes many architectures that have demonstrated broad effectiveness in practical 2D vision applications, such as image restoration, remote sensing, and medical imaging, even when the equivariance is restricted to finite groups like p4p_4. Crucially, in regular-representation, standard pointwise non-linearities are already inherently equivariant, eliminating the need to design specialized equivariant activation functions. Specifically, we would like to clarify the following points:

1) Regular-representation models support nonlinearities without destroying equivariance. The gated or norm-based nonlinearities are indeed essential for higher-dimensional irreducible representations, where applying arbitrary elementwise activations would destroy the equivariance. Our target models—EQ-ViT, EQ-Swin, EQ-VMamba, and EQ-INR—operate on regular-representation features, on which the group acts strictly by coordinate permutation. Thus, standard pointwise activations (e.g., GELU, SiLU, ReLU) perfectly preserve equivariance when applied identically across the permuted coordinates.

In fact, even for models built upon an irreducible/isotypic basis, recent equivariant acceleration works (Bökman et al., 2025; Nordström et al., 2026) handle nonlinearities by transforming features back to a regular basis and applying standard pointwise activations. Since our targeted architectures inherently operate in the regular basis, they naturally enjoy these compatible nonlinearities for free. Thus, replacing the dense EQ-Linear bottleneck with Flash EQ-Linear seamlessly accelerates the network end-to-end, requiring no modifications to the nonlinear operators.

2) Linear layers are the primary computational bottleneck in 2D vision architectures, making their acceleration of substantial practical value. As shown in the table below, linear layers account for 93–97% of FLOPs and 59–74% of inference latency. In contrast, the other layers, including the nonlinear operators, take a smaller share (3–7% of FLOPs). Thus, accelerating the linear layers unlocks the most significant end-to-end performance gains.

ScaleLinear FLOPs ratioLinear latency ratioLinear-layer speedup (vs. naive)End-to-end speedup (vs. naive)
Small92.9%58.7%1.9x1.41x
Base95.4%67.1%1.89x1.48x
Large96.7%73.9%2.07x1.59x

3) Finite groups are sufficient for most 2D vision tasks. We acknowledge that our current formulation applies only to finite groups. Nevertheless, finite groups (such as p4p_4 and D8D_8) are exactly the foundation of the most widely used equivariant networks for 2D images, including G-CNN (Cohen & Welling, 2016), GSA-Net (Romero & Cordonnier, 2021), EQ-ViT (Fu et al., 2026), and EQ-VMamba (Zhao et al., 2026). These models have demonstrated state-of-the-art effectiveness across a wide range of practical 2D vision tasks, such as image restoration, remote sensing, and medical imaging. Thus, providing acceleration for finite-group equivariant layers inherently unlocks immense practical value for the 2D vision community.

4) Practical impact: broad applicability and plug-and-play acceleration without retraining. The computational overhead of equivariant architectures is a widely recognized obstacle. While recent works (Bökman et al., and Nordström et al.) address this by designing new irrep-based architectures that must be trained from scratch, Flash EQ-Linear provides a highly practical alternative. It accelerates existing regular-representation models (e.g., EQ-ViT, EQ-Swin, EQ-VMamba, and EQ-INR) directly at the operator level, requiring zero retraining. Crucially, our heavily optimized fused CUDA kernels overcome the memory bottleneck, delivering actual wall-clock speedups even for small models where prior implementations offer limited gains. As a result, Flash EQ-ViT/Swin/VMamba are strictly faster than their standard, non-equivariant baselines. These results strengthen the practical case for equivariance: practitioners no longer need to trade inference speed for accuracy and robustness.

Revision. In the revision, we have (i) made clear that our acceleration method targets regular-representation models and clarified the compatibility with equivariant nonlinearities, and (ii) stated the finite cyclic group assumption in Sec. 3.

Weaknesses 2: CUDA kernels only support the p4 group.

Response 2: We agree with the reviewer's suggestion to broaden our hardware support. While our original submission focused on p4p_4 to demonstrate the core principle, our algorithmic framework is broadly extensible. In the revised manuscript, we have developed heavily optimized CUDA kernels for D2D_2, p4p_4, p8p_8, and the non-abelian octic group D8D_8 (Tables 16–20), comprehensively covering the most widely used discrete symmetry groups in 2D image equivariant networks. Specifically:

  • Finite cyclic groups (p4p_4, p8p_8): As Algorithm 1 natively applies to any finite cyclic group of even order TT, we have implemented the corresponding highly efficient kernels for p4p_4 and p8p_8.
  • Dihedral groups (D2D_2, D8D_8): Going beyond cyclic groups, we have extended our mathematical derivation to handle the matrix-valued Fourier blocks of general dihedral groups (detailed in the newly added Appendix B), and implemented the corresponding kernels for D2D_2 and D8D_8.

This extension represents a substantial engineering effort (yielding over 10,000 lines of custom CUDA code that support both forward and backward passes in FP32 and FP16) to ensure these theoretical savings translate into out-of-the-box acceleration. Some representative FP32 inference speedups are summarized below:

GroupOrder TTRepresentative symmetrySpeedup over F.linear (C/T=2048C/T=2048)Speedup over standard ViT-H
D2D_22Reflection1.8×1.8\times1.4×1.4\times
p4p_4490∘90^\circ Rotations2.1×2.1\times1.7×1.7\times
p8p_8845∘45^\circ Rotations4.3×4.3\times2.2×2.2\times
D8D_8890∘90^\circ Rotations and reflections4.6×4.6\times2.2×2.2\times

Beyond its engineering value, this comprehensive implementation serves a critical validation purpose. In low-level operator design, theoretical FLOP reductions often fail to translate into actual wall-clock speedups due to memory bandwidth and kernel-launch overheads. Our kernels bridge this gap, empirically proving that the theoretical efficiency of equivariant methods can indeed be converted into real-world inference acceleration, ultimately demonstrating their potential to strictly outperform traditional non-equivariant baselines.

Revision. We have added operator-level and end-to-end benchmarks for the additional groups in Appendix E (Tables 16–20).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Final Remarks

Dear Area Chair and Reviewers,

We sincerely thank the AC and all three reviewers for their time and effort, appreciation of our work, and constructive suggestions. The revised manuscript has been uploaded with all changes highlighted in red, and the key revisions are summarized as follows.

1) Novelty Framing and Attribution to Prior Work:

Reviewer 5aQV (W1) and Reviewer FgJz (W2) raised concerns about the novelty framing relative to Quick ViTs [1, 2]. We have narrowed our claims as suggested and reframed our contributions around four explicit points: (i) operator-level, training-free replacement; (ii) fused CUDA kernels that turn the theoretical savings into practical speedups at small scales; (iii) explicit arithmetic for complex-type irreps in finite cyclic groups; and (iv) comprehensive empirical validation across Transformers, SSMs, and INRs.

2) Extension to the Non-Abelian Group D8D_8:

Reviewer FgJz (W1) and Reviewer ryQD (W2) requested extensions beyond cyclic groups. We have derived a full framework (Algorithm R3) for general dihedral groups D2nD_{2n}, implemented fused CUDA kernels for D8D_8, and trained Flash EQ-ViT-D8D_8. Measured FP32 operator-level speedups over F.linear at C/T=2048C/T=2048 are 1.8×1.8\times (D2D_2), 2.1×2.1\times (p4p_4), 4.3×4.3\times (p8p_8), and 4.6×4.6\times (D8D_8). The derivation and the operator-level and end-to-end benchmarks are given in Appendices B and E (Tables 16–20).

3) Matched-Budget Comparison with Quick ViTs:

Reviewer 5aQV (W2, W3, Q1) and Reviewer FgJz (W3) requested larger-scale and matched-budget evaluation. Table 6 now covers all five scales (Tiny to Huge) in FP32 and FP16, with Flash EQ-ViT-D8D_8 exactly matching Quick ViT-D8D_8 in reported parameters and FLOPs. New ImageNet-100 results (throughput in imgs/s; speedup over the standard ViT at the same scale):

ScaleQuick FP32Flash FP32Quick FP16Flash FP16
Tiny3294 (1.2×1.2\times)4444 (1.6×1.6\times)3229 (0.4×0.4\times)9297 (1.2×1.2\times)
Small2249 (1.4×1.4\times)3030 (1.8×1.8\times)3449 (0.7×0.7\times)6029 (1.2×1.2\times)
Base1389 (1.6×1.6\times)1743 (2.0×2.0\times)3388 (1.3×1.3\times)3847 (1.4×1.4\times)
Large504 (1.9×1.9\times)546 (2.0×2.0\times)1255 (1.6×1.6\times)1319 (1.7×1.7\times)
Huge265 (1.9×1.9\times)308 (2.2×2.2\times)685 (1.6×1.6\times)688 (1.6×1.6\times)

Flash EQ-ViT-D8D_8 is faster than Quick ViT-D8D_8 in FP32 at every scale, and in FP16 it is 2.9×2.9\times and 1.7×1.7\times faster at Tiny and Small respectively, while the two are on par at Large and Huge. Training recipes for all five scales have been added to Appendix E, as requested by Reviewer FgJz (W3).

4) Scope and Nonlinear Compatibility:

Reviewer ryQD (W1) raised concerns about nonlinear-layer compatibility and the finite-group restriction. Flash EQ-Linear targets regular-representation models (EQ-ViT, EQ-Swin, EQ-VMamba, EQ-INR), in which the group acts on features by coordinate permutation, so standard pointwise activations (GELU, SiLU, ReLU) are already exactly equivariant. No modification to the nonlinear operators is required, and the end-to-end integration is verified to be conflict-free (Table 5). Finite groups such as p4p_4 and D8D_8 underlie the most widely deployed 2D equivariant vision models, so this scope delivers substantial practical value.

5) Impact Concerns:

The computational overhead of equivariant architectures is a widely recognized obstacle. Comparing to prior architecture-level approaches, Flash EQ-Linear provides a highly practical alternative: it accelerates existing regular-representation models (EQ-ViT, EQ-Swin, EQ-VMamba, EQ-INR) directly at the operator level, requiring zero retraining. Crucially, our heavily optimized fused CUDA kernels overcome the memory bottleneck, delivering actual wall-clock speedups even for small models, where prior implementations offer limited gains. As a result, Flash EQ-ViT/Swin/VMamba are strictly faster than their standard, non-equivariant baselines. These results strengthen the practical case for equivariance: practitioners no longer need to trade inference speed for accuracy and robustness.

We would like to reiterate our core contributions: a training-free, plug-and-play operator requiring zero architectural change; fused CUDA kernels covering D2D_2, p4p_4, p8p_8, and D8D_8 (over 10,000 lines of custom code, forward and backward, FP32 and FP16) that deliver end-to-end speedups at every model scale; explicit arithmetic for complex-type irreps, together with a Gauss-identity analysis that lowers the theoretical FLOP bound from 16/616/6 to 16/516/5 for p4p_4; and end-to-end validation showing that equivariant networks can simultaneously outperform standard ViTs in both accuracy and throughput.

We hope these revisions address all reviewers' concerns and would be deeply grateful for your reconsideration of the scores.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.