acceptodds
Under review as a conference paper at ICLR 2027

TrimSO(2): Fast and Memory-Efficient Training of Equivariant Interatomic Potentials

Abstract

Equivariant interatomic potentials enable accurate and efficient simulations of molecules and materials. Among them, models based on SO(2) convolutions reduce the cost of tensor products and achieve state-of-the-art accuracy, but their training remains time-consuming and memory-intensive. Existing training implementations materialize large edge tensors for representation transformations surrounding dense matrix multiplications, increasing memory usage and data movement. We observe that these transformations can be integrated into the model's existing dataflow, avoiding separate intermediate materialization. We introduce TrimSO(2), a fast and memory-efficient algorithm for SO(2)-based equivariant models. It realizes this principle through rotation on gather, recombination with activation, and inverse rotation on reduction, with support for the higher-order differentiation required for training. By avoiding intermediate materialization, TrimSO(2) delivers 2.71–16.73 operator speedups across forward, backward, and double backward and reduces model-level high-bandwidth memory traffic by 3.81–9.88. Finally, TrimSO(2) achieves 2.07–2.94 training speedups and 1.53–3.16 reductions in peak training memory over PyTorch implementations for official eSEN, UMA, and EquiformerV3 workloads on A100 and H100 GPUs.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Equivariant neural networks, Machine learning interatomic potentials, SO(2) convolutions, GPU acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_NZQPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets the training cost of SO(2)\mathrm{SO}(2) based interatomic potentials. The dense matrix products in these models are already efficient, but current implementations write large intermediate edge tensors to memory for the steps around them, namely gathering, rotation, recombination, layout changes and inverse rotation. TrimSO(2) folds these steps into neighbouring operations. Rotation happens while gathering node features, recombination happens inside the activation, and inverse rotation happens during the reduction onto nodes. The kernels support the backward and double backward passes needed for force training, and a similar idea is applied to the node feed forward networks. On eSEN, UMA and EquiformerV3 trained on OMat24 in FP32, TrimSO(2) gives 2.07×2.07\times to 2.94×2.94\times faster training, 1.53×1.53\times to 3.16×3.16\times lower peak memory and 3.81×3.81\times to 9.88×9.88\times less HBM traffic than the official PyTorch code, while matching its outputs and gradients to about 10−710^{-7}.

Critical strengths
  1. The gains are consistent across three recent models and two GPUs, and the numerical checks are thorough. Outputs, gradients and training states match the reference closely, and the architectures are unchanged, so the method can be dropped into existing training code.

  2. The paper handles the double backward pass needed to train on forces, including the rotation derivatives. This is the hard part, and it is often what stops existing fusion or compilation tools from being used for these models.

Critical weaknesses
  1. The only baseline is the official PyTorch code in eager FP32. This is a relevant baseline, since torch.compile does not support the double backward pass and the official training setups run eagerly. Still, a 2×2\times to 3×3\times gain from fusing gather, scatter and elementwise operations over eager PyTorch is roughly what one would expect, and the paper does not test a stronger option such as a traced and compiled path, as the authors themselves note. All runs also disable TF32 and mixed precision, while models like UMA are pretrained in BF16. With faster low precision matrix products the balance of the step changes, so the reported speedups may not carry over to how these models are actually trained.

  2. The technical idea is close to existing GNN systems work. Avoiding intermediate edge tensors by fusing gather, transform and scatter is used in Graphiler, FlashTP, cuEquivariance and OpenEquivariance. The new part is carrying this through the SO(2)\mathrm{SO}(2) layout changes and the double backward, but the paper does not explain what made this hard beyond existing designs. Also, much of the EquiformerV3 gain comes from the node feed forward reordering, which is not specific to SO(2)\mathrm{SO}(2) (Table 7 goes from 1.82×1.82\times to 2.83×2.83\times when it is added).

  3. HBM traffic drops 9.88×9.88\times for UMA but the step time only drops 2.94×2.94\times. A breakdown of the remaining time would show where the limit now is.

  4. On H100, the full TrimSO(2) step for UMA (244.86 ms) is slower than the previous ablation stage (238.37 ms) in Table 7.

Questions for authors
  1. How does the forward pass of TrimSO(2) compare with fairchem's Triton kernels for UMA inference?
  2. Why is the full method slower than the previous stage for UMA on H100?
  3. Need to show training speedups with TF32 or BF16 enabled.
  4. Need to show validation errors after the 40,000 step eSEN run, and results on graphs with uneven neighbour counts.
4 comments

We thank Reviewer NZQP for the detailed and professional feedback. The revision adds each of the measurements you listed. Our responses to your questions are provided below.

1. Comparison with FAIRChem UMA inference kernels. We compared TrimSO(2) against two optimized inference settings from FAIRChem, umas_fast_gpu and Nitro. For the 1,000-atom copper system, the median end-to-end latencies are as follows.

GPUumas_fast_gpu (ms)TrimSO(2) (ms)Speedup
A10069.1635.171.97×
H10032.6525.921.26×

In the H100 size sweep, Nitro is fastest for 10–500 atoms, and TrimSO(2) is 1.30× and 1.28× faster than Nitro at 1,000 and 5,000 atoms. The full settings and results are in Appendix C.7, Table 23 and Figure 14.

2. UMA ablation timing on H100. Thank you for pointing this out. The original three runs varied considerably across these stages due to server fluctuations: the run-to-run standard deviation of Edge → Node is about 19% of its mean, and the difference between 238.37 ms and 244.86 ms falls within measurement variation. We re-measured both stages three times each on the same H100 with the same workload (OMat24 batches, at most 700 atoms per batch). Edge → Node averaged 303.66 ms/step and full TrimSO(2) averaged 286.87 ms/step, so enabling the node-side optimization reduces time by 5.53%, and the reversal did not recur. The revised Table 8 uses the re-measured values. We re-checked the remaining results and found no comparable variation.

3. UMA training performance at different precisions. We compared UMA-S-1.2 training with FP32, TF32, and BF16 autocast on the same A100. Both implementations use the same random initial state and OMat24 batch sequence, with at most 700 atoms per batch.

PrecisionPyTorch (ms/step)TrimSO(2) (ms/step)SpeedupPyTorch peak memory (GiB)TrimSO(2) peak memory (GiB)
FP321076.84367.282.93×27.0417.66
TF32945.21248.523.80×27.0417.66
BF16 autocast670.25357.181.88×17.9015.41

With TF32 and BF16 autocast, official PyTorch is 1.14× and 1.61× faster than its own FP32 setting. At matched precision, TrimSO(2) is 2.93×, 3.80×, and 1.88× faster than official PyTorch in FP32, TF32, and BF16 respectively. Even in FP32, TrimSO(2) is 1.82× faster than the fastest of the three official settings, BF16 autocast.

4. eSEN validation error after 40,000 training steps on OMat24. We trained eSEN 30M for 40,000 steps with PyTorch and with TrimSO(2), and evaluated the resulting checkpoints on the OMat24 validation set (Appendix C.6.4, Table 20).

Validation metricPyTorchTrimSO(2)
Energy MAE (meV/atom)37.2036.49
Force MAE (meV/Å)123.19122.69
Stress MAE (meV/ų)4.614.54

The per-atom energy, force, and stress MAE are close between the two implementations, with relative differences of 1.9%, 0.4%, and 1.5% respectively.

5. Training performance on graphs with uneven neighbor counts. On A100 we compared four neighbor-count distributions with EquiformerV3. Each graph has 256 atoms and 16,384 directed edges, averaging 64 neighbors per atom. Each setting ran independently three times, timing 100 steps per run. The table reports the mean time and the highest peak memory over the three runs.

Neighbor-count SDPyTorch (ms/step)TrimSO(2) (ms/step)SpeedupPyTorch peak memory (GiB)TrimSO(2) peak memory (GiB)
01548.60530.462.92×50.6116.03
161553.69537.272.89×50.6116.03
321553.21533.962.91×50.6116.03
481553.72536.282.90×50.6116.03

With the total edge count fixed, TrimSO(2) achieves 2.89–2.92× training speedup as the neighbor-count standard deviation ranges from 0 to 48, and peak memory is about 32% of official PyTorch. Peak memory is the same in all four settings (50.61 and 16.03 GiB), as expected. With the total edge count fixed, the peak is set by the tensors allocated per edge, independent of how those edges are distributed across nodes. Time is also essentially unaffected, so uneven neighbor counts introduce no pronounced load imbalance.

6. Comparison with a compiled baseline. We used NequIP-style tracing to place the energy, force, and stress computations in a single graph, then compiled that graph and the parameter-gradient computation it requires with PyTorch Inductor. We measured full training steps for eSEN, UMA, and EquiformerV3 on A100 and H100 (Appendix C.2, Table 6). The compiled baseline was faster than eager in all six comparisons. Over this baseline, TrimSO(2) still achieves 1.16–2.62× end-to-end speedup and 1.73–2.75× lower peak allocated memory.

7. UMA HBM traffic and kernel time. We broke down HBM traffic and GPU kernel time during UMA training on A100.

Kernel categoryPyTorch HBM read+write (GiB)TrimSO(2) HBM read+write (GiB)PyTorch kernel time (ms)TrimSO(2) kernel time (ms)
Elementwise kernels979.4715.60704.9616.64
Matrix multiplication54.0236.83309.66241.92
Other kernels37.5355.9446.5760.85
Total1071.02108.371061.19319.41

Before the optimization, elementwise kernels accounted for 91.5% of total traffic and 66.4% of total kernel time, and these operators were bandwidth-bound. After the optimization, their HBM read+write falls from 979.47 to 15.60 GiB and their time from 704.96 to 16.64 ms. Matrix multiplication now accounts for 75.7% of total kernel time, and the bottleneck shifts from bandwidth to compute, which is why bandwidth falls by a larger factor than time. Reducing matrix-multiplication cost further, for example with mixed precision, could improve efficiency further.

8. SO(2) dataflow design and the contribution of the node-side optimization. The difficulty in optimizing SO(2) convolution is that the dense matrix products and the surrounding operations use different coordinate frames and feature layouts. The matrix products organize their inputs by local angular order. The activation must obtain both the positive and negative angular-order components of the product output. Node aggregation first has to rotate messages from different edges back into a common global frame. TrimSO(2) uses these dependencies: it generates the angular-order layout the matrix products require while gathering, reads and recombines the paired components in the activation, and rotates back and accumulates edge messages in the reduction. This preserves the efficient dense matrix products and avoids separate writes and re-reads of entire edge tensors for these transformations. We extend the same dataflow design to backward and double backward, accelerating the full training computation.

The node-side equivariant feedforward optimization does not depend on the angular-order structure specific to SO(2), but also follows the same principle. Channel mixing that can be done per tile stays on chip, avoiding a write-back of intermediates. The three edge-side optimizations alone give 1.82× end-to-end speedup, and adding the node-side optimization brings it to 2.83× (A100, 686.16 → 441.90 ms/step). The three edge-side optimizations apply to all three models, and the node-side optimization provides further improvements for models where the feedforward network is an appreciable cost.

We again thank Reviewer NZQP for the thorough comments. Your comments have helped us improve the clarity of the paper. Please let us know if any part of our response remains unclear, and we would be glad to explain further.

The authors have provided additional experiments that addressed my concerns. The overall performance and acceleration remain consistent in the new experiments. I wil therefore keep my score.

We are glad that our response addressed your concerns! Thank you again for your time and constructive feedback.

Reviewer_5pgpOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce TrimSO(2), a fast kernel specifically designed for SO(2) style convolutions introduced in the eSCN paper and utilized in the eSEN, UMA, and EquiformerV3 MLIPs. The kernels leads to 2.71-16.7x speedups in forward, backward, and double backward operations while achieving 2.07-2.94x end-to-end training speedup and 1.53-3.16x reduction in peak training memory over standard PyTorch implementations.

Critical strengths

TrimSO(2) appears to be a performant kernel at accelerating and reducing the peak memory consumption of the training of SO(2) style MLIPs such as UMA, EquiFormerV3, and eSEN. In particular, UMA is an important model used by many computational materials and molecular scientists.

Critical weaknesses

One big limitation of this work is its novelty and research impact. The introduction of a novel kernel is very much welcomed to the community and useful for its practitioners; however, the fundamental kernel-level tricks introduced are more akin to kernel engineering techniques as opposed to methods that may be of interest to the MLIP kernel research community. There have been many prior works investigating equivariant kernels as well as techniques involving the minimization of HBM accesses [1, 2, 3, 4]. Although these prior works primarily involve SO(3) convolutions, the reduction of operations from SO(3) to SO(2) is straightforward and mathematically simpler than SO(3), as shown in the paper.

I do not believe the benchmark comparisons, especially on the UMA model, are fair. The UMA repository has an officially-supported flash-UMA setting that uses custom triton kernels that also speed up the SO(2) convolutions [5, 6]. The authors should benchmark TrimSO(2) against the entire repertoire of state of the art inference settings instead of just choosing native pytorch.

The training trajectories in Figure 6b are not sufficient to show long training stability. For example, eSEN-30M-OMat was trained on 4 epochs of OMAT24's training split at batch size 512, which would amount to almost 800k training steps [7]. Even the smallest UMA was trained on a total of 2.68 million training steps [8]. Figure 6b, on the other hand, shows non-trivial divergences in the training dynamics comapred to reference training only at around 40k training steps. From just figure 6b, it is not possible to confidently extrapolate overall stable training dynamics throughout the entire length of a 800k-2.68 million step training run. Because training on the entire training sets from the original eSEN and UMA papers is challenging, I would recommend performing a head-to-head comparison on energy/forces and stable simulations of common systems from training on something like the OMol25 4M training split -- a much more tractable but still realistic training target -- in order to be able to claim overall training stability of TrimSO(2).

Also, I find it somewhat odd that the authors report speedup in forward and backward passes but do not report inference results. Inference results would contextualize the significance of the TrimSO(2) results more. Why did the authors not benchmark inference if forward and backward kernels are better than native Pytorch?

[1] https://arxiv.org/abs/2205.14135 [2] https://docs.nvidia.com/cuda/cuequivariance/ [3] https://openreview.net/forum?id=wiQe95BPaB [4] https://github.com/PASSIONLab/OpenEquivariance [5] https://facebookresearch.github.io/fairchem/ase-calculator/ [6] https://github.com/ArcelliEmanuele/flash-uma [7] https://arxiv.org/html/2502.12147v2 [8] https://arxiv.org/html/2506.23971v2

Questions for authors

See above.

5 comments

We thank Reviewer 5pgp for the constructive and detailed feedback. The revision adds a long training run on OMol25 4M and a comparison with both UMA inference backends. Below are our responses to your points.

1. Technical novelty and research value. The kernels in cuEquivariance, FlashTP, and OpenEquivariance that relate to this work mainly optimize Clebsch–Gordan coupling in SO(3) tensor products. In SO(2) models, edge alignment turns the Clebsch–Gordan tensor product into SO(2) matrix products that are independent by angular order. The tensor product itself becomes much cheaper but introduces rotation and inverse rotation. The matrix products are already efficient, yet the rotation, recombination, and inverse rotation around them still produce large per-edge tensors. TrimSO(2) addresses the overhead that remains after this simplification, using the structure of the SO(2) convolution to fold these transformations into adjacent operations. Training conservative-force models also requires differentiating the force computation once more, and TrimSO(2) extends the same dataflow design to backward and double backward.

2. Training stability over long runs. We trained for 794,480 steps on OMol25 4M, close to the roughly 800,000 steps you cite for eSEN-30M on OMat24. Both sides use the same conservative-force eSEN model with 4 layers, 48 channels, and 923,281 trainable parameters, with identical initialization, data, and optimization settings, and we evaluate on the same validation structures each epoch (Appendix C.6.5).

Validation metric after 794,480 stepsPyTorchTrimSO(2)
Energy MAE (meV/atom)3.3743.386
Force MAE (meV/Å)20.88220.798

The relative differences in energy and force error are 0.36% and 0.40%, and the validation curves stay close over the ten epochs.

We also ran 100 ps NVE simulations on MD22 Ac-Ala3-NHMe and DHA with these two checkpoints, at an initial temperature of 500 K and a 1 fs time step, with three initial states per system (Appendix C.6.5, Table 22). For each initial state the two checkpoints use the same coordinates and velocities, and forces are evaluated with PyTorch throughout, to isolate the effect of the training implementation.

SystemTraining implementationCompletedAbsolute energy drift (meV/atom/ps)Max. energy excursion (meV/atom)Max. bond-length change (Å)
Ac-Ala3-NHMePyTorch3/30.00045 ± 0.000231.3780.345
Ac-Ala3-NHMeTrimSO(2)3/30.00117 ± 0.000151.8670.361
DHAPyTorch3/30.00148 ± 0.000781.5410.294
DHATrimSO(2)3/30.00058 ± 0.000811.2630.307

The drift rate is the absolute slope of a linear fit to per-atom total energy over time, reported as mean ± sample standard deviation over three trajectories. Maximum energy excursion and maximum bond-length change are measured relative to each trajectory's initial state and then maximized over the three trajectories. Both checkpoints completed all 100 ps, and the maximum bond-length change stayed below 0.5 Å in all 12 trajectories. Integrated over 100 ps, the energy drift of the four cases stays below 0.2 meV/atom, far below the 1.26–1.87 meV/atom maximum energy excursion of the same trajectories.

We also ran an additional 10 ps NVE simulation of DHA from the same three initial states and compared the vibrational power spectra of the two checkpoints (Appendix C.6.5, Figure 13). The two area-normalized mean spectra overlap by 94.2%. Two training implementations give similar vibrational spectra.

3. Comparison with the UMA inference optimizations. Our reply to Reviewer GnMA reports the end-to-end inference comparison against umas_fast_gpu in fairchem-core 2.23.0. The FlashUMA work you point to has made a valuable contribution to UMA inference acceleration. FAIRChem acknowledges it in the UMA Nitro acknowledgments, and the implementation has been incorporated into the official backend. We therefore did not benchmark that repository separately, and instead compared directly against the two optimized inference backends that FAIRChem provides, which covers the acceleration from FlashUMA. Nitro was released with fairchem-core 2.24.0 on October 22, 2026, later than our submission deadline, so the original submission did not compare against it.

We used fairchem-core 2.25.0 on a single H100 80G to compare multigpu, Nitro, and TrimSO(2) with CUDA Graph enabled, on UMA-S-1.2.1. multigpu (formerly turbo) runs the umas_fast_gpu backend here, with TF32 and dynamic compilation. The test uses synthetic face-centered-cubic carbon with a lattice constant of 3.8 Å in a non-orthogonal periodic cell, covering 10 to 20,000 atoms (Appendix C.7).

Speedups are relative to FAIRChem multigpu.

AtomsFAIRChem multigpu (ms/step)FAIRChem Nitro (ms/step)TrimSO(2) (ms/step)Nitro speedupTrimSO(2) speedup
1018.412.082.998.87×6.16×
5015.362.553.276.03×4.69×
10018.743.173.905.92×4.80×
50029.0910.0810.442.88×2.79×
1,00031.2317.6313.581.77×2.30×
5,00082.1369.6154.431.18×1.51×
10,000169.68OOM117.17—1.45×
15,000261.82OOM178.09—1.47×
20,000OOMOOM225.26——

Nitro is fastest for 10–500 atoms. At 1,000 and 5,000 atoms, TrimSO(2) is 1.30× and 1.28× faster than Nitro. At 10,000 and 15,000 atoms Nitro runs out of memory, and TrimSO(2) is 1.45× and 1.47× faster than multigpu. At 20,000 atoms, the largest scale in this test, TrimSO(2) is the only one of the three paths to complete inference, at 225.26 ms/step. TrimSO(2)'s advantage on large graphs comes from two additional optimization details: Opposite-edge reuse assigns the two directions of an atom pair to the same compute block, sharing node feature reads and radial weights. Node-side projection moves the second linear transformation to the node side, so gated results accumulate directly during reduction, avoiding a write and a subsequent read. Nitro and umas_fast_gpu target inference, whereas TrimSO(2) targets the computation that training requires, such as double backward for forces, and also covers the inference path.

We again thank Reviewer 5pgp for the helpful and constructive comments. We believe the longer training run and the broader inference comparison that you suggested will make our paper stronger. Please let us know if you would like us to go further into any of these results, and we would be glad to respond.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer 5pgp,

As the author-reviewer discussion period draws to a close, we wanted to gently check whether our response and the added experiments in the revision have addressed your concerns. Please let us know if any points remain unclear.

Sincerely,

The authors

Reviewer_GnMAOpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TrimSO(2), a GPU optimization framework for SO(2)-based equivariant interatomic potentials. It reduces intermediate tensor materialization through fused rotation, recombination, and aggregation operations, with support for forward, backward, and double-backward computations. Evaluations on MLIPs like eSEN, UMA, and EquiformerV3 using A100 and H100 GPUs report 2.07–2.94× training speedups and 1.53–3.16× peak memory reductions over official PyTorch implementations.

Critical strengths

1. This is a technically well motivated SO(2) speciffic optimization work for SO(2)-MLIPs. This paper identifies that SO(2)-equivariant convolutions incur significant intermediate memory overhead and data movement costs. To address this, the authors propose three synergistic optimization strategies: rotation during the gather phase, recombination integrated with activation functions, and inverse rotation during the reduction phase. A key advantage of this design is that it preserves the structure of dense matrix multiplication while leveraging compact rotational blocks, seamlessly integrating these transformations into existing computational data flows. Furthermore, the authors extend these optimizations to double-backward computations, which is critical for training interatomic potentials.

2. Validation on multiple architectures. The evaluation covers three SO(2) based MLIPs (eSEN,UMA, and EquiformerV3) on both A100 and H100 GPUs. This paper reported improvements are subtantial, with 2.07-2.94x end-to-end training speedups and 1.53-3.16x reductions in peak allocated memory relative to official Pytorch ones. Overall, the breadth of the evaluation and the attention to higher-order derivative correctness support the practical relevance of the proposed system.

Critical weaknesses

1. Substantial overlap with an earlier publicly available technical implementation raises concerns about the novelty. I identified an earlier publicly available GitHub Gist, titled "Fused wigner↔SO2-conv edgewise ops (umas_fast_gpu)" (https://gist.github.com/aditvenk/98551ea637848bfc186a913bc6d01033). And this technique alreay has been used in UMA Commit 8a570eb. First, it fuses node-feature gathering, Wigner rotation, angular-order reordering (L-to-M), radial scaling, and GEMM input packing into a producer kernel, which is highly similar to the proposed Node-to-Edge optimization. Second, it fuses angular-order unpacking (M-to-L) with inverse Wigner rotation in a consumer kernel, closely resembling the proposed Edge-to-Node optimization. This is more substantial than a general conceptual similarity: the two implementations target closely corresponding computational boundaries and intermediate representations in SO(2) convolutions. This concern affects my assessment of the paper's novelty.

2. The distinct technical novelty needs to be established more clearly. The main components of TrimSO(2), including operator fusion, tiling, intermediate recomputation, and reduction fusion, are established systems optimization techniques. Previous works like FlashTP and Sobek already employed some of those techniques in the optimization for SO(3) MLIPs. To me, TrimSO(2) feels more like an application of operator optimization techniques to SO(2) convolutions rather than an innovative technical invention. However, the paper does not sufficiently isolate which aspects of the design specifically exploit SO(2) structure and which largely follow generic fusion and streaming principles.

3. The lack of a sufficiently optimized training baseline limits the strength of the performance claims. The main comparsion is against the official PyTorch implementations without compilation. The reported speedups therefore establish clear advantages over the selected eager PyTorch implementations, but do not establish how much of the improvement remains against a stronger compiled or otherwise optimized training pipeline. This makes it difficult to assess the practical performance advantage of TrimSO(2) relative to the current optimization frontier.


Lee, S.Y., Kim, H., Park, Y., Jeong, D., Han, S., Park, Y. & Lee, J.W.. (2025). FlashTP: Fused, Sparsity-Aware Tensor Product for Machine Learning Interatomic Potentials. Proceedings of the 42nd International Conference on Machine Learning.

Vladimir Chorošajev and Cédric Bény. Sobek: Streaming Equivariant Tensor Product Convolutions. (2026). arXiv. 2607.18074

Questions for authors
  1. Could the authors clarify the relationship between TrimSO(2) and the publicly available GitHub Gist? I am particularly interested in understanding the incremental technical contributions beyond the two dataflow optimizations already described in the Gist.

  2. The paper mentions that “FAIRChem also provides Triton kernels for UMA inference,” but it does not specify which operations these kernels cover? Could the authors provide further clarification on this with TrimSO(2)?

  3. I am concerned that the reported speedup is based on benchmarks that were not compiled. The practical advantage over a carefully compiled official implementation remains unclear. If the authors could include a comparison of "NequIP-style tracing + compile" for even a single model in their rebuttal—even if limited to a subset of operators or a simplified version, it would strengthen their argument.

5 comments

We thank Reviewer GnMA for the professional and insightful comments. The revision adds a comparison with the UMA inference optimizations and a comparison with a compiled baseline. Below are our responses to your questions.

1. Relation to the UMA inference optimizations. Thank you for identifying the Gist and 8a570eb specifically. The Gist merges node feature gathering and Wigner rotation into the input preparation of the first matrix product, and in backward it re-reads node features and rotates them again instead of retaining the per-edge intermediate. On the output side it combines order restoration and inverse rotation into one step. 8a570eb (PR #2154) adds a fused gated activation and combines inverse rotation with accumulation at receiving nodes into one operation. These works identify the main overheads at the same computational boundaries as TrimSO(2), which we acknowledge. The Gist and 8a570eb appeared shortly before the submission deadline (8a570eb was merged only one month prior), so we did not have this comparison ready in time for the original submission. The revision adds it in Appendix C.7.

The two efforts differ in the computational scope they support. umas_fast_gpu targets the UMA inference path and is specialized for L=M=2L=M=2, choices that match its inference use. TrimSO(2) targets force training, which requires double backward in addition to forward and first backward, and extends the edge-side design to the node-side equivariant feedforward network. The table below lists the scope each covers by computation stage.

ImplementationUMAeSENEquiformerV3Energy forwardFirst backwardDouble backwardNode-side FFN
FAIRChem 2.23.0 umas_fast_gpuYes——YesYesPyTorch path—
TrimSO(2)YesYesYesYesYesYesYes

Each of these extensions required separate design. Double backward differentiates once more after forward and first backward are complete, so for each of the three dataflows we had to determine when to recompute and which intermediates to retain. Models with L≠ML\neq M retain only part of the orders (eSEN has L=3,M=2L=3, M=2 and EquiformerV3 has L=4,M=2L=4, M=2), and the size of the rotation matrices and the layout conversions change accordingly. The node-side equivariant feedforward network is a separate bottleneck outside the edge computation: on EquiformerV3 the three edge-side optimizations give 1.82× end-to-end speedup, and adding the node-side optimization brings it to 2.83× (A100, 686.16 → 441.90 ms/step).

We compared TrimSO(2) end-to-end against umas_fast_gpu in fairchem-core 2.23.0, which includes the optimizations above. Both used the same UMA-S-1.2 checkpoint and a 1,000-atom periodic copper system, with TF32 and dynamic compilation enabled.

MetricA100 umas_fast_gpu → TrimSO(2)A100 changeH100 umas_fast_gpu → TrimSO(2)H100 change
CUDA-event median latency / QPS69.04 → 35.04 ms / 14.48 → 28.54QPS +97.0%32.39 → 25.88 ms / 30.88 → 38.64QPS +25.1%
CUDA-event minimum latency67.65 → 34.27 ms−49.3%29.14 → 20.61 ms−29.3%
End-to-end median latency / QPS69.16 → 35.17 ms / 14.46 → 28.43QPS +96.6%32.65 → 25.92 ms / 30.63 → 38.58QPS +25.9%
End-to-end minimum latency67.77 → 34.39 ms−49.3%29.35 → 20.63 ms−29.7%
Peak allocated over the whole run6.65 → 3.94 GiB−40.8%6.70 → 5.09 GiB−24.0%
Peak allocated in the timed interval6.65 → 3.94 GiB−40.8%6.70 → 5.09 GiB−24.0%
Reserved after warmup7.69 → 4.67 GiB−39.3%7.72 → 6.06 GiB−21.5%

TrimSO(2) further reduces per-edge work relative to umas_fast_gpu. This improvement comes from two optimization details: Opposite-edge reuse assigns the two directions of an atom pair to the same compute block, sharing node feature reads and radial weights. Node-side projection moves the second linear transformation to the node side, so gated results accumulate directly during reduction, avoiding a write and a subsequent read. On A100 these two optimizations give an additional 1.97× inference speedup and a 40.8% reduction in peak allocated memory. On H100 the median latencies of umas_fast_gpu and TrimSO(2) are 32.39 and 25.88 ms, and TrimSO(2) is still 1.26× faster.

2. Comparison with a compiled baseline. We added a comparison against NequIP-style tracing and compilation for eSEN, UMA, and EquiformerV3. We place the energy, force, and stress computations in a single graph, then compile that graph and the parameter-gradient computation it requires with PyTorch Inductor. We measured the time and peak memory of full training steps on A100 and H100 (Appendix C.2, Table 6). The compiled baseline was faster than eager in all six comparisons. Over this compiled baseline, TrimSO(2) still achieves 1.16–2.62× end-to-end training speedups and 1.73–2.75× lower peak memory. The largest gap is UMA on A100 (2.62×), where compilation brings little benefit (1077.19 → 959.44 ms) and per-edge intermediate tensors dominate. The smallest gap is eSEN on H100 (1.16×), the comparison where compilation helps most (370.71 → 199.80 ms).

GPUModelPyTorch ((ms/step) / GiB)Compiled ((ms/step) / GiB)TrimSO(2) ((ms/step) / GiB)Speedup vs. compiledPeak memory reduction vs. compiled
A100eSEN653.55 / 34.62399.14 / 30.92315.05 / 17.661.27×1.75×
A100UMA1077.19 / 27.04959.44 / 32.81366.49 / 17.672.62×1.86×
A100EquiformerV31249.30 / 70.98810.10 / 61.88441.90 / 22.481.83×2.75×
H100eSEN370.71 / 34.66199.80 / 30.62172.17 / 17.711.16×1.73×
H100UMA629.05 / 27.09607.59 / 32.81286.87 / 17.712.12×1.85×
H100EquiformerV3606.86 / 71.02347.52 / 61.88246.38 / 22.521.41×2.75×

The three columns use the same measurement protocol (Appendix C.2). Compilation raises UMA peak allocated memory from 27.04 and 27.09 GiB to 32.81 GiB on A100 and H100, because Inductor allocates additional intermediate buffers and a static workspace for the compiled graph. Under the same protocol TrimSO(2) uses 17.67 and 17.71 GiB.

3. Exploiting SO(2) structure. Earlier SO(3) work such as FlashTP and Sobek uses fusion, tiling, and recomputation to optimize computation built on Clebsch–Gordan coupling. Unlike SO(3), an edge-aligned SO(2) convolution can mix channels independently by angular order and therefore uses efficient dense matrix products. TrimSO(2) preserves these products. Its optimization target is the representation change that connects the matrix products with the graph operations. Fusion and recomputation are general techniques. Which boundaries can be fused, and what each step reads and writes, follows from the data dependencies of the SO(2) convolution:

  • SO(2) rotation acts independently on each degree, and the rotation matrix of each degree retains only the orders actually used. Different degrees can therefore be processed separately and written directly into the input arrangement the matrix product requires. Clebsch–Gordan coupling in SO(3) mixes across several input and output degrees and does not have this independence.
  • Recombination reads the positive and negative components of each angular order together and then applies the gate. This pairing requirement determines how the activation is grouped, so it cannot be handled as elementwise fusion.
  • Double backward must decide, across the three dataflows, which intermediates to retain for recomputation. This step depends on the gather structure of the SO(2) convolution.

4. Additional requirements of the training workload. The loss for force training differentiates the force computation once more, so double backward is a necessary part of the training pipeline. The inference path of umas_fast_gpu does not need this. All three fusions in TrimSO(2) provide corresponding double-backward kernels: on A100, double backward is 7.24×, 6.53×, and 5.62× faster than PyTorch for Node→Edge, Edge→Edge, and Edge→Node respectively (Appendix C.4, Table 10).

We note that UMA released a further optimized version, Nitro, after the submission deadline. We added that comparison to the revision and describe it in our reply to Reviewer 5pgp. We again thank Reviewer GnMA for the professional and detailed comments. We believe these comments greatly contributed to the clarity of the paper, particularly in the discussion of scope. Please let us know if you have any further questions, and we would be glad to explain further.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer GnMA,

As the author-reviewer discussion period nears its end, we wanted to briefly follow up to see whether our response and the revisions regarding related work and baselines address your questions. Please let us know if anything would benefit from further clarification.

Sincerely,

The authors

Reviewer_hEXWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Training SO(2)-based equivariant interatomic potentials (like eSEN, UMA, and EquiformerV3) is notoriously slow and memory-hungry. Existing implementations materialize massive intermediate edge tensors in HBM just to handle coordinate frame rotations and feature recombinations, causing massive bandwidth waste. This gets even worse because training these potentials requires computing forces (first derivatives of energy) and stresses, which forces the system to keep intermediate variables alive for double backward passes, blowing up memory pressure. To fix this, the authors introduce TrimSO(2). The core idea is straightforward but highly effective dataflow fusion: they merge "gather + rotation", "GEMM output + recombination + activation", and "inverse rotation + aggregation" directly into on-chip GPU computations using custom Triton kernels, completely bypassing HBM writes for intermediate results. Crucially, they successfully extend this exact logic to both backward and double backward passes. Their experiments on A100/H100 GPUs show roughly 3x memory reduction and over 2x speedup across three mainstream models, while a 40,000-step long training run proves the numerical precision remains virtually identical to the original PyTorch implementations.

Critical strengths
  • Fusing the double backward pass for SO(2) models via custom Triton kernels is a great engineering feat that directly solves the memory and time bottlenecks for force gradient calculations.
  • The 40k-step loss curve perfectly matching the baseline proves there's no accuracy-speed tradeoff.
Critical weaknesses
  • Unfair baseline comparison. Comparing hand-written Triton kernels against uncompiled PyTorch eager mode muddies the waters. It's hard to isolate how much speedup comes from the specific dataflow fusion versus just beating uncompiled PyTorch.
  • Narrow scope. The method is heavily tied to SO(2) local-frame architectures. It doesn't easily translate to pure SO(3) tensor products (like MACE), making it feel more like a custom patch than a general systems insight.
Questions for authors
  • If we use the NequIP-style tracing trick to enable torch.compile on the PyTorch baseline, does TrimSO(2) still keep that 2-3x speedup, or is the Triton kernel just crushing compiled eager ops?
  • Cramming all these ops into one Triton kernel must spike register pressure. Did you see a drop in warp occupancy, and at what graph size does this register-bound bottleneck actually hit?
  • How do the memory and speed benefits scale with graph density? Eliminating O(E) intermediate tensors helps a lot for dense bulk metals, but what about sparse small molecules?
3 comments

We thank Reviewer hEXW for the careful and constructive comments. The revision adds a compiled baseline for all three models, together with measurements of register use, occupancy, and performance across graph densities. Below are our responses to your questions.

1. Comparison with a compiled baseline. We used NequIP-style tracing to place the energy, force, and stress computations in a single graph, then used PyTorch Inductor to compile that graph and the parameter-gradient computation it requires. We measured the time and peak memory of full training steps on A100 and H100 (Appendix C.2, Table 6). The compiled baseline was faster than eager in all six comparisons, and TrimSO(2) still achieved 1.16–2.62× speedups and 1.73–2.75× lower peak memory over it. The smallest gap is eSEN on H100 (1.16×), which is also the comparison where compilation helps most (370.71 → 199.80 ms).

GPUModelPyTorch ((ms/step) / GiB)Compiled ((ms/step) / GiB)TrimSO(2) ((ms/step) / GiB)Speedup vs. compiledPeak memory reduction vs. compiled
A100eSEN653.55 / 34.62399.14 / 30.92315.05 / 17.661.27×1.75×
A100UMA1077.19 / 27.04959.44 / 32.81366.49 / 17.672.62×1.86×
A100EquiformerV31249.30 / 70.98810.10 / 61.88441.90 / 22.481.83×2.75×
H100eSEN370.71 / 34.66199.80 / 30.62172.17 / 17.711.16×1.73×
H100UMA629.05 / 27.09607.59 / 32.81286.87 / 17.712.12×1.85×
H100EquiformerV3606.86 / 71.02347.52 / 61.88246.38 / 22.521.41×2.75×

2. Register pressure and warp occupancy. TrimSO(2) performs fused computation in fixed-size tiles. A larger graph only means more tiles to process, and the per-thread register allocation of a given kernel stays the same. On A100 we profiled the three dataflow operators of EquiformerV3, covering forward, backward, and double backward, and aggregated register counts and occupancy weighted by kernel execution time (Appendix C.4.3, Figure 10).

AtomsDirected edgesRegisters per thread (PyTorch / TrimSO(2))Warp occupancy (PyTorch / TrimSO(2))Total kernel time (ms) (PyTorch / TrimSO(2))
482,04863.0 / 89.748.45% / 35.82%16.65 / 3.09
964,09664.2 / 85.851.42% / 41.08%31.89 / 5.26
1928,19264.2 / 84.053.51% / 44.66%62.99 / 9.43
38416,38464.6 / 83.054.38% / 46.52%124.50 / 18.02
76832,76864.9 / 82.454.63% / 47.59%247.60 / 34.91
1,53665,53665.0 / 82.354.73% / 48.11%493.83 / 69.30

At the same graph size, TrimSO(2) uses more registers per thread than PyTorch (about 1.3–1.4×) and has correspondingly lower warp occupancy. Lower occupancy does not imply lower performance: the fused kernel performs more arithmetic per thread, so each thread does more useful work than PyTorch's elementwise kernels. As the graph grows, the average register count falls from 89.7 to 82.3 and occupancy rises from 35.82% to 48.11%. This change reflects the different shares of forward, backward, and double backward at different sizes, not a change in the register allocation of a single kernel. TrimSO(2) has lower total kernel time at all six sizes, with speedups of 5.39–7.13×.

3. Effect of graph density on training performance. On an A100 80 GB we compared PyTorch and TrimSO(2) with eSEN-30M on ethane, NaCl, silicon, and copper, four systems of different density, at a fixed 512 atoms per batch. The four systems have 7, 32, 46, and 78 neighbors per atom (Appendix C.5, Table 13).

SystemNeighbors/atomDirected edgesTime (ms/step) (PyTorch / TrimSO(2))SpeedupPeak memory (GiB) (PyTorch / TrimSO(2))Reduction
Ethane73,584557.21 / 323.201.72×8.71 / 4.721.84×
Rock-salt NaCl3216,3841,408.45 / 711.991.98×34.92 / 17.851.96×
Diamond-cubic silicon4623,5521,935.53 / 988.561.96×49.60 / 25.231.97×
FCC copper7839,936OOM / 1,635.55—OOM / 42.09—

Compared with dense systems, sparse systems show somewhat lower speedups and memory reductions. Even on the sparsest ethane batch, TrimSO(2) was 1.72× faster and reduced peak memory by 46% (1.84×). On the denser NaCl and silicon batches, the speedup and the memory reduction are both close to 2×. More edges mean larger per-edge intermediate tensors, so removing them saves more memory. On the densest copper system, PyTorch ran out of memory while computing forces, and TrimSO(2) completed the training step with a peak of 42.09 GiB.

We also compared real training batches. Using the OMat24 and OMol25 datasets on A100, we timed 100 consecutive training batches for each with eSEN-30M (Appendix C.5, Table 15).

DatasetMean atomsMean directed edgesMean neighbors/atomTime (ms/step) (PyTorch / TrimSO(2))SpeedupPeak memory (GiB) (PyTorch / TrimSO(2))Reduction
OMat24263.6310,352.5439.27974.62 / 506.281.93×35.13 / 17.921.96×
OMol25250.047,875.8831.50772.83 / 379.982.03×34.65 / 17.681.96×

Across the 100 randomly sampled training batches, OMat24 and OMol25 averaged 39.27 and 31.50 neighbors per atom respectively. TrimSO(2) achieved about 2× training speedup and peak memory reduction on both.

4. Scope. TrimSO(2) targets the convolution structure shared by SO(2)-based equivariant interatomic potentials. Rewriting the SO(3) tensor product as an SO(2) convolution in edge-local frames reduces the complexity from O(L6)O(L^6) to O(L3)O(L^3) [1], and this structure has been used in materials property prediction, molecular modeling, and catalysis [2–8]. The structure has the same form in many models (e.g., eSEN, UMA, EquiformerV3): the dense matrix products organize their inputs by local order, and on either side are edge-local rotation, recombination, and inverse rotation. TrimSO(2) targets exactly these three boundaries, so it applies without architectural change to both convolution blocks (eSEN, UMA) and attention blocks (EquiformerV3), and to the L/M=3/2L/M=3/2, 2/22/2, and 4/24/2 order configurations. Reducing the per-edge intermediate reads and writes at these boundaries yields the training speedups and memory reductions.

[1] S. Passaro and C. L. Zitnick. Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNs. ICML, 2023.

[2] Y.-L. Liao et al. EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations. ICLR, 2024.

[3] X. Fu et al. Learning Smooth and Expressive Interatomic Potentials for Physical Property Prediction. ICML, 2025.

[4] B. M. Wood et al. UMA: A Family of Universal Models for Atoms. NeurIPS, 2025.

[5] L. Barros-Luque et al. The Open Materials 2024 (OMat24) Inorganic Materials Dataset and Models. Nat. Comput. Sci., 2026.

[6] Y.-L. Liao et al. EquiformerV3: Scaling Efficient, Expressive, and General SE(3)-Equivariant Graph Attention Transformers. arXiv:2604.09130, 2026.

[7] T. Li et al. DPA4: Pushing the Accuracy–Cost Frontier of Interatomic Potentials with EMFA SO(2) Convolution. arXiv:2606.02419, 2026.

[8] Z. Xu, W. Xie, and P. Hu. Edge Cluster Expansion with Radial Rotary Attention for Interatomic Potentials. arXiv:2607.10664, 2026.

We again thank Reviewer hEXW for the detailed feedback. Your comments on the fairness of the baseline and on graph density helped us make the evaluation in the paper clearer and more complete. Please let us know if any part of our response remains unclear, and we would be glad to clarify.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer hEXW,

With the discussion period closing soon, we wanted to gently check whether our response and the additional evaluations in the revision address your questions. We would be glad to clarify anything further if needed.

Sincerely,

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers for their thoughtful and constructive feedback. To provide a clear overview of our revisions and responses, we summarize the main updates below, with all manuscript changes highlighted in blue.

  • Comparison with UMA inference optimizations (Reviewers GnMA, 5pgp, NZQP): Expanded Related Work and Appendix C.7 to clarify the relation to FAIRChem's UMA inference backends (umas_fast_gpu and Nitro). While those backends target the L=M=2L=M=2 inference path, TrimSO(2) covers conservative-force training (including double backward), truncated-order models (L≠ML\neq M), and node-side equivariant FFNs, and further reduces per-edge work via opposite-edge reuse and node-side projection. Added end-to-end inference benchmarks against umas_fast_gpu on A100 and H100 (Appendix C.7, Table 23) and against multigpu and Nitro up to 20,000 atoms on H100 (Appendix C.7, Figure 14).
  • Scope and SO(2) technical positioning (Reviewers hEXW, GnMA, 5pgp, NZQP): Clarified how TrimSO(2) exploits the data dependencies of edge-aligned SO(2) convolutions—degree-wise rotation, paired-order recombination, and layout conversion around dense matrix products—across both convolution and attention blocks (eSEN, UMA, and EquiformerV3), distinguishing our design from SO(3) Clebsch–Gordan kernels.
  • Training baselines and profiling (Reviewers hEXW, GnMA, NZQP): Added NequIP-style compiled baselines across all three models on A100 and H100 (Appendix C.2, Table 6), mixed-precision UMA training comparisons across FP32, TF32, and BF16 (Appendix C.2, Table 7), kernel profiling and HBM/time breakdown analyses (Appendix C.3.2 and C.4.3), and graph-density and uneven-neighbor evaluations (Appendix C.5).
  • Accuracy and long-horizon stability (Reviewers 5pgp, NZQP): Added 40,000-step OMat24 validation errors (Appendix C.6.4, Table 20), a 794,480-step OMol25 4M training run (Appendix C.6.5, Table 21 and Figure 11), and 100 ps MD22 NVE trajectories with DHA vibrational power spectra (Appendix C.6.5, Table 22 and Figures 12–13).

Sincerely,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.