acceptodds
Under review as a conference paper at ICLR 2027

Accelerating SO(2)-Equivariant Materials Models through Low-Precision Emulation

Abstract

Rotationally equivariant architectures improve the physical fidelity of machine-learned (ML) universal materials models based on graph neural networks (GNNs). However, their scalability is limited by the use of non-trivial, high-precision tensor operations during both training and inference. Adopting the reduced-precision formats of matrix multiplication units (MMUs), e.g., FP8 instead of FP32, provides massive increases in throughput, up to 30× (60×) on NVIDIA Hopper (Blackwell) GPUs, but may lead to drastic loss of accuracy. Here, we introduce tailored ‘emulated’ low-precision operations under the integer Ozaki scheme to accelerate the ‘SO(2)-equivariant’ processing layers of GNNs, which dominate the runtime of (1) many state-of-the-art ML interatomic potentials (MLIPs) and (2) most universal ML Hamiltonian (MLH) models. Our framework, FlashSO(2), outperforms TF32, the most accurate native low-precision MMU format, in throughput, accuracy, and memory footprint at once. At the SO(2) layer level, it reaches 40× (7×) the throughput of MALOQ’s (UMA’s) reference implementation, an MLH (MLIP) model, sustaining up to 236 (232) effective Tflop/s on an NVIDIA GH200, 3.5× the FP32 peak. At the model level, FlashSO(2) helps train MALOQ 3.9× faster than its pure-FP32 version while reducing peak memory by 58% at no loss in accuracy. In molecular dynamics simulations with UMA, the speedup factor is 3.3×, and the peak memory is reduced by 51%. FlashSO(2) thus pushes the speed-accuracy Pareto frontier of equivariant materials models beyond the native low-precision formats and substantially lowers the resources required to exploit their scaling laws.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
precision emulation, mixed-precision training, tensor cores, GPU kernels, kernel fusion, equivariant graph neural networks, machine learning Hamiltonians, machine learning interatomic potentials

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ub3aOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FlashSO(2), which accelerates the SO(2) convolution introduced by Passaro & Zitnick (eSCN).

The authors provide three fused kernels for rotation and gated activation, and apply the Ozaki scheme for precision emulation to the two SO(2) convolution blocks.

Their results show that the proposed implementation achieves a rotation equivariance error similar to that of the original FP32 implementation and accelerates MALOQ training and UMA-M inference by 3.9× and 3.3×, respectively.

Critical strengths

Clear results. The throughput gain is significant, and the paper tackles an important problem with current SO(2) convolution layers.

Critical weaknesses
  1. While the paper measures equivariance error, training loss, and Hamiltonian MAE, important metrics are missing. For example, material properties obtained from the Hamiltonian matrix for MALOQ, as well as energy, force, and stress errors from the MLIP model, are not reported.

  2. Limited accuracy comparison between FlashSO(2) and the original FP32 implementation using FP32-trained models. This is important because we can use pretrained checkpoints for faster inference while preserving accuracy.

  3. No training loss results for UMA-M. Conservative force-field training requires double backward, and although training throughput is reported, training loss and accuracy comparisons with FP32 are missing.

Questions for authors
  1. Can the authors provide metrics for energy, forces, and/or other material properties with FlashSO(2), in addition to training loss and rotation equivariance error?
  2. Can the authors separately report the performance contribution of each of the three fused kernels (S1, S2, and S3)?
  3. Can the authors provide error metrics for UMA-M and MALOQ using existing FP32-trained checkpoints with FlashSO(2), to verify whether using this method leads to any accuracy degradation?
  4. Can the authors provide training loss curves and accuracy metrics for conservative MLIP models trained with FlashSO(2) and compare them with FP32 models?
4 comments

Thank you for the careful review of our paper. We have run new experiments to address all three weaknesses (see point 1 for the first, point 3 for the second, and point 4 for the third). All your questions are answered in order below.

Notation. FP32-REF is the original PyTorch implementation of the SO(2) layer in FP32. FP32-FL denotes our fused kernels with the convolution GEMMs in FP32, i.e., without emulation. Emu2 is the emulated setting of the paper with two INT8 slices, in which our fused kernels run the GEMMs with integer tensor cores. Everything outside the SO(2) layer remains in FP32 in all our settings. All timings are on one GH200.

1. Can the authors provide metrics for energy, forces, and/or other material properties with FlashSO(2), in addition to training loss and rotation equivariance error?

For UMA, we trained the UMA-S architecture (without MoLE) from scratch on MPtrj in FP32 and with Emu2 under identical settings (the same recipe, seed, initial weights and batches). Against DFT on 80,105 held-out frames, Emu2 exhibits an MAE change of +0.5%, +0.1% and −0.3% for the energy, force and stress with respect to FP32, respectively. All these values remain within the statistical noise (Table R1). Relaxations of 2,000 materials do not separate the two models either. For MALOQ, we derived orbital energies from the predicted Hamiltonians. With Emu2, the median errors of the HOMO, LUMO and gap against DFT stay within 0.3 meV of FP32 (Table R2). The revised paper will include these orbital energies as well.

Table R1. Energy, force and stress errors against DFT on 80,105 frames held out from training, with the final weights of each run evaluated in FP32. Brackets give 95% bootstrap intervals over materials.

trainingE MAE (meV/atom)F MAE (meV/Å)ΔE vs FP32ΔF vs FP32ΔS vs FP32
FP32 (fairchem)14.4642.3–––
Emu214.5442.3+0.5% [−0.7, +1.8]+0.1% [−0.7, +1.0]−0.3% [−1.7, +1.1]

Table R2. Median absolute errors of orbital energies against DFT over the 32 held-out conformers, in meV, with the weights trained with Emu2.

inference precisionHOMOLUMOgapoccupied levels
FP324.09.712.04.5
Emu24.39.912.34.5

2. Can the authors separately report the performance contribution of each of the three fused kernels (S1, S2, and S3)?

Table R3 gives the time of each operation in one SO(2) layer of UMA-M, including the three fused kernels. Fusion cuts S1 from 18.8 to 5.6 ms, S2 from 4.4 to 1.2 ms and S3 from 6.2 to 0.9 ms. With Emu2, S1, S2 and S3 drop further to 1.7, 0.6, and 0.5 ms, because they pass INT8 slices instead of FP32 data. The largest gain comes from the convolution GEMMs between the kernels, which emulation runs 6.8× faster (45.0 → 6.6 ms). The revised paper will include the more detailed breakdown of Table R3.

Table R3. GPU time in ms of each operation in one SO(2) layer of UMA-M with 2,304 atoms and 179,712 edges. The percentages correspond to the shares of the summed time of the five operations.

FP32-REFFP32-FLEmu2
S1: gather, rotation, radial weights, l→ml \to m18.8 (21%)5.6 (10%)1.7 (16%)
radial network5.4 (6%)4.8 (8%)1.3 (12%)
convolution GEMMs54.1 (61%)45.0 (78%)6.6 (62%)
S2: gate4.4 (5%)1.2 (2%)0.6 (6%)
S3: m→lm \to l, inverse rotation, aggregation6.2 (7%)0.9 (2%)0.5 (5%)
layer forward (wall clock)88.758.111.5

3. Can the authors provide error metrics for UMA-M and MALOQ using existing FP32-trained checkpoints with FlashSO(2), to verify whether using this method leads to any accuracy degradation?

Yes. FlashSO(2) keeps the weights in FP32, so FP32-trained checkpoints load directly. Running the FP32-trained weights of the UMA-S architecture from point 1 with Emu2 does not change any held-out MAE by more than 0.02%. For the released UMA-M 1.1 checkpoint, Emu2 changes the energy and force MAEs against DFT on 20,000 frames of the OMat24 validation set by less than 0.01% (7.59 meV/atom and 46.4 meV/Å with both FP32 and Emu2).

For MALOQ, Emu2 inference of the FP32-trained weights changes the MAE of the Hamiltonian entries by 0.5% (median over the 32 conformers of Table R2). The revised paper will include these errors for MALOQ and UMA-M 1.1.

4. Can the authors provide training loss curves and accuracy metrics for conservative MLIP models trained with FlashSO(2) and compare them with FP32 models?

Training UMA requires on the order of a hundred H200 GPUs for weeks (28 days for UMA-M and 10 days for UMA-S) [1], so we trained the UMA-S architecture of point 1 on the much smaller MPtrj dataset instead. Forces and stresses come from autograd, so each of the 319,084 training steps runs double backward through the SO(2) layer. Emu2 matches FP32 within the statistical noise (Table R1), and its training loss tracks the FP32 curve over the whole run, ending only 0.3% above it. Table R4 gives the training loss of both runs at different steps along the run.

Table R4. Training loss of the UMA-S architecture with FP32 and with Emu2 (same recipe, seed, initial weights and batches), as the median over the 20 epochs before each step.

training stepFP32Emu2Emu2 vs FP32
80,0001.21601.2178+0.15%
160,0001.00451.0057+0.12%
240,0000.78070.7823+0.20%
319,084 (end)0.65670.6586+0.28%

We hope that these additional experiments address your concerns, and we would be glad to hear whether anything remains unclear.

References

[1] B. M. Wood et al., UMA: A family of universal models for atoms, NeurIPS (2025).

Thank you for the detailed response and additional experiments. The new results address my main concerns regarding accuracy preservation, the performance breakdown of the fused kernels, and conservative MLIP training.

I appreciate the authors' efforts during the rebuttal period. I am satisfied with the responses and will raise my score.

We are glad that our response addressed your concerns. And thank you for the careful review and the constructive suggestions, and for raising your score.

Reviewer_o3JAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper speeds up the SO(2)\mathrm{SO}(2) convolution layer used in MLIPs such as UMA and in Hamiltonian models such as MALOQ. It does two things. First, it fuses the rotation, the two SO(2)\mathrm{SO}(2) linear layers, the gate and the inverse rotation into three kernels. Second, it runs the matrix products on INT8 tensor cores with the integer Ozaki scheme, where pp INT8 slices with block scaling keep about 7p7p bits. With two slices (Emu2) the layer stays within a 10−410^{-4} equivariance tolerance that TF32 just misses. On one GH200 the layer runs 40×40\times faster than the MALOQ reference and 7×7\times faster than the UMA reference. End to end, MALOQ training is 3.9×3.9\times faster with 58% less memory, and UMA-M molecular dynamics is 3.3×3.3\times faster with 51% less memory.

Critical strengths
  1. The speedups are large and target the real bottleneck, since the SO(2)\mathrm{SO}(2) layer takes 79 to 91% of a UMA-M training step. The memory savings also allow much larger systems on a single GPU.
  2. Using precision emulation to meet an equivariance budget is a new idea for this area, where low precision has been seen as unsafe. The roofline analysis that separates the gain from fusion and from emulation is clear.
Critical weaknesses
  1. Most of the headline gain comes from kernel fusion, and the end to end baselines are not fused. On the MALOQ layer, fusion alone gives 7.0×7.0\times and emulation adds 5.6×5.6\times. At the model level, Table 2 shows a fused FP32 training step of 275 ms against 152 ms for Emu2, so emulation itself gives about 1.8×1.8\times, and the conclusion puts Emu2 only 1.1×1.1\times ahead of TF32. The 40×40\times and 3.9×3.9\times numbers in the abstract should be put next to fused FP32 and fused TF32 results.

  2. The accuracy evidence does not fully support the claim that TF32 is unsuitable while Emu2 is safe. In training, TF32 and Emu2 reach the same validation loss (1.9×10−41.9 \times 10^{-4} in Table 2). The 21.6% inference penalty for TF32 comes from PyTorch's global TF32 mode, which also rounds the Wigner rotations, so it is not a fair comparison with Emu2. Beyond that, MALOQ accuracy is measured on only 32 held-out conformers, UMA has no accuracy check against DFT, and the NVE drift is only described in words. The 10−410^{-4} tolerance is also not linked to any physical quantity.

  3. The equivariance test uses random weights for MALOQ, a single water molecule, rotations about the zz axis only, and a smoothed median. Trained weights, random O(3)\mathrm{O}(3) transformations and the maximum error would be more convincing.

Questions for authors
  1. What are the end to end speedups of Emu2 over fused FP32 and fused TF32?
  2. What is the inference error when TF32 is used only inside the FlashSO(2) convolutions?
  3. What is energy and force errors of UMA-M with Emu2 against DFT, and actual numbers for the NVE drift.
  4. Can you do an equivariance test with trained weights and random O(3)\mathrm{O}(3) transformations.
4 comments

Thank you for the careful review of our paper. We have run new experiments to address all three weaknesses (see point 1 for the first, points 2 and 3 for the second, and point 4 for the third), and we answer your questions in order below.

Notation. FP32-REF is the original PyTorch implementation of the SO(2) layer in FP32. FP32-FL and TF32-FL are our fused kernels with the convolution GEMMs in FP32 and in TF32, respectively, i.e., without emulation. TF32-FL also uses TF32 for the radial network and the products in S3. Everything outside the SO(2) layer remains in FP32 in all our settings. Emu2 and Emu3 are the two emulated settings of the paper, in which our fused kernels run the GEMMs with integer tensor cores. All timings are on one GH200.

1. What are the end to end speedups of Emu2 over fused FP32 and fused TF32?

End to end, Emu2 is 2.46× faster than our fused FP32 kernels (FP32-FL) in MALOQ training and 3.95× faster per UMA-M MD step, while the speedups are 1.06× and 1.23× with respect to our fused TF32 kernels (TF32-FL). Table R1 details the benefits brought by compilation, fusion, and emulation, one at a time.

Table R1. Step time in ms. Each row adds one optimization to the row above, with the speedup over that row in brackets.

MALOQ training (lmax⁡=4l_{\max}=4, 2,137 atoms)UMA-M MD step (lmax⁡=4l_{\max}=4, 1,008 atoms)
FP32-REF, eager1,204896
+ torch.compile824 (1.46×)761 (1.18×)
+ fusion (FP32-FL)631 (1.31×)660 (1.15×)
+ emulation (Emu2)256 (2.46×)167 (3.95×)
fused TF32 (TF32-FL)271206

The revised paper will report the headline speedups next to fused FP32 and fused TF32 for all models and values of lmax⁡l_{\max}.

2. What is the inference error when TF32 is used only inside the FlashSO(2) convolutions?

TF32-FL is exactly this setting, as it uses TF32 only inside our FlashSO(2) layer (for the convolution GEMMs, the radial network, and the products in S3) and keeps everything else in FP32. It increases the MAE of the Hamiltonian entries by 9.8% over FP32, against 0.18% with Emu2 (Table R2). This value (9.8%) corresponds to roughly half of the MAE increase measured with PyTorch's global TF32 mode in Fig. 5(b) (21.8%), which also rounds the Wigner rotations. Crucially, it is still 54× the increase with Emu2 (0.18%). Emu2 is also faster than TF32-FL (point 1) and halves the peak memory (Section 4.3). The revised paper will show both TF32 scopes.

Table R2. Increase of the MAE of the Hamiltonian matrix entries, measured against DFT, relative to FP32 inference, with the weights trained with Emu2; median over 32 conformers.

inference precisionMAE increase over FP32
TF32, global+21.8%
TF32-FL+9.8%
Emu2+0.18%

To link the tolerance to a physical quantity, we also derived orbital energies from the predicted Hamiltonians. With Emu2, the median errors of the HOMO, LUMO and gap against DFT stay within 0.3 meV of FP32, e.g., 12.3 error in the predicted band gap with Emu2 versus 12.0 with FP32 (Table R3).

Table R3. Median absolute errors of orbital energies against DFT over the 32 held-out conformers, in meV, with the weights trained with Emu2.

inference precisionHOMOLUMOgapoccupied levels
FP324.09.712.04.5
Emu24.39.912.34.5

3. What are the energy and force errors of UMA-M with Emu2 against DFT, and actual numbers for the NVE drift?

For the released UMA-M 1.1 checkpoint, Emu2 changes the energy and force MAEs against DFT on 20,000 frames of the OMat24 validation set by less than 0.01% (7.59 meV/atom and 46.4 meV/Å with both FP32 and Emu2). We also trained the UMA-S architecture (without MoLE) from scratch on MPtrj in FP32 and with Emu2 under identical settings (the same recipe, seed, initial weights and batches). Against DFT on 80,105 held-out frames, Emu2 matches FP32 within the statistical noise (Table R4), and running the FP32-trained weights with Emu2 changes no held-out MAE by more than 0.02%.

Table R4. Energy and force errors against DFT on 80,105 frames held out from training, with the final weights of each run evaluated in FP32. Brackets give 95% bootstrap intervals over materials.

trainingE MAE (meV/atom)F MAE (meV/Å)ΔE vs FP32ΔF vs FP32
FP32 (fairchem)14.4642.3––
Emu214.5442.3+0.5% [−0.7, +1.8]+0.1% [−0.7, +1.0]
TF32-FL14.5042.1+0.2% [−1.1, +1.5]−0.3% [−1.2, +0.5]
BF16 (fairchem, autocast)15.5153.9+7.3%+27.6%

For the NVE drift, we ran 56 simulations of 100 ps each (200,000 velocity Verlet steps of 0.5 fs), starting from velocities at 600 K, for 8 materials randomly drawn from MPtrj, which were not seen in training. The revised paper will show the curves of energy against time. Results are summarized in Table R5: Over 100 ps, the drift of Emu2 adds up to 0.9–2.0 μeV/atom. That is about twice the drift of FP32 (1.7–2.1×), and the total energy fluctuates by 1.4–1.6× as much. On UMA-M, TF32-FL drifts 86× more than FP32 and fluctuates 10.8× more.

Table R5. Drift rate (slope of a linear fit of the total energy against time) and RMS fluctuation around that fit: median over the 8 materials, range over the models (the UMA-S architecture trained in FP32 or with Emu2, and UMA-M 1.1).

routedrift rate, absolute (10−510^{-5} meV/atom/ps)RMS fluctuation (10−310^{-3} meV/atom)
FP320.5–1.00.7–0.9
Emu20.9–2.01.0–1.3
TF32-FL (UMA-M only)458.7

4. Can you do an equivariance test with trained weights and random O(3) transformations.

We repeated the equivariance test with trained weights, for MALOQ and for the UMA-S architecture of point 3, both trained with Emu2. We used 8 structures per model (8 of the 32 conformers of Table R2, and the final frames of the FP32 NVE runs of the 8 materials of point 3) and applied to each of them the same 100 random rotations, drawn uniformly from SO(3). The error is the largest entry of the residual over the largest entry of the output. Table R6 presents the resulting equivariance error.

Table R6. Equivariance error under rotation over 8 structures × 100 rotations.

settingMALOQ, medianMALOQ, maxUMA-S forces, medianUMA-S forces, max
FP32-REF5.9e-79.0e-68.6e-65.5e-4
TF32-FL3.4e-46.2e-42.9e-41.5e-3
Emu22.5e-55.7e-54.5e-54.8e-4
Emu35.8e-79.0e-68.4e-65.5e-4

Under rotation, Emu3 matches FP32-REF exactly, and the median error of TF32-FL is 14× (MALOQ) and 6× (UMA-S forces) that of Emu2. For the UMA-S forces, the maximum of FP32-REF comes from a single structure and rotation, which is also the worst case in FP64, and the next largest is 4.5e-5. The revised paper will include the related test.

We hope that these additional experiments address your concerns, and we would be glad to hear whether anything remains unclear.

I appreciate the authors' efforts in performing the additional experiments and providing detailed responses, which have addressed my questions. I will therefore keep my original score.

Thank you for confirming that our responses addressed your concerns. We appreciate your positive assessment and your helpful questions.

Reviewer_PCc2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents FlashSO(2), an implementation that accelerates SO(2)-equivariant computations in materials models. Building on the integer Ozaki scheme, it combines INT8 slice-based computation and block scaling with kernel fusion across the SO(2) pipeline. It reduces memory use through compressed intermediate activations and a recomputation mode called MiniMem. The authors evaluate numerical errors, task accuracy, and computational performance using MALOQ, a Hamiltonian prediction model, and UMA, an interatomic potential model. They report up to approximately 3.9× faster training steps for MALOQ and 3.3× faster molecular dynamics inference steps for UMA-M.

Critical strengths

The paper addresses computational and memory bottlenecks in SO(2)-equivariant materials models by combining integer precision emulation with kernel fusion. It demonstrates end-to-end performance gains on MALOQ and UMA. The training and validation results for MALOQ also support task accuracy close to FP32 under the tested settings with Emu2. These results indicate practical value in reducing the cost of training and deploying equivariant models.

Critical weaknesses
  1. The model-level comparisons do not isolate the benefits of precision emulation. The headline speedups combine kernel fusion, compilation, and emulation. End-to-end comparisons against fused FP32 and TF32 implementations remain incomplete. The accuracy comparison uses global TF32 but restricts Emu2 to SO(2) layers. The accuracy gap therefore cannot be attributed to numerical format alone. These limitations weaken the evidence for emulation’s incremental benefits.

  2. The evidence that the reported speedups preserve accuracy in MLIPs is insufficient. The accuracy validation focuses on MALOQ and does not adequately cover UMA’s training and MD applications. UMA training with a force loss requires mixed second derivatives. The paper reports training speed but provides no training accuracy results. It also claims NVE energy drift comparable to FP32 without reporting quantitative results or the simulation duration.

Questions for authors
  1. Figure 5(b) uses global TF32, while Emu2 is applied only within SO(2) layers. Attributing the accuracy gap to numerical format despite these different scopes is misleading. Meanwhile, Figure 9(a) and Table 2 show that TF32 restricted to SO(2) convolutions achieves training and validation results close to Emu2. How does the MAE of TF32 compare with Emu2 when TF32 is restricted to SO(2) convolutions? Would this comparison still support the claim that TF32 will generally fail?
  2. The paper reports that SO(2) layers account for over 80% of runtime, but this includes gathering, rotations, gating, and aggregation, not just GEMMs. Figure 6(a) shows that these layers still dominate after optimization. Please provide operation-level timings before and after optimization and separate the gains from compilation, kernel fusion, and precision emulation.
  3. During UMA’s mixed second-derivative computation, errors from the forward pass and first derivatives may propagate or be amplified. Could the authors evaluate UMA training, comparing Emu2 and FP32 under identical settings for convergence and final energy and force validation errors?
  4. Numerical errors from low-precision computation may accumulate over integration steps, so short runs may not establish long-term stability. Please provide energy-versus-time curves for the Emu2–FP32 comparison in Appendix C and report the integration timestep, total simulation duration, and energy drift rate.
  5. The first two methods subsections mainly describe existing techniques. What is the specific technical contribution beyond combining integer Ozaki emulation, block scaling, and kernel fusion?
  6. The models evaluated in the paper are relatively computationally expensive and do not fully reflect the model sizes commonly used in the community, where smaller variants such as UMA-S are more widely adopted. Can Emu2/Emu3 provide similar performance benefits on smaller models such as UMA-S?
  7. The models evaluated in the paper are essentially all based on eSCN-style. This raises concerns about whether the proposed method generalizes to other SO(2)-equivariant models, such as EquiformerV2/V3 and DPA4.
6 comments

Thank you for the careful review of our paper. We have run new experiments to address both weaknesses (see points 1 and 2 for the first and points 3 and 4 for the second), and we answer your questions in order below.

Notation. FP32-REF is the original PyTorch implementation of the SO(2) layer in FP32. FP32-FL and TF32-FL are our fused kernels with the convolution GEMMs in FP32 and in TF32, respectively, i.e., without emulation. TF32-FL also uses TF32 for the radial network and the products in S3. Everything outside the SO(2) layer remains in FP32 in all our settings. Emu2 and Emu3 are the two emulated settings of the paper, in which our fused kernels run the GEMMs with integer tensor cores. All timings are on one GH200.

1. How does the MAE of TF32 compare with Emu2 when TF32 is restricted to SO(2) convolutions?

We added TF32-FL, which uses TF32 only inside our SO(2) layer. The rest remains as in Emu2. In case of Hamiltonian prediction, TF32-FL roughly halves the MAE increase of the global TF32 mode of Fig. 5(b) (+9.8% instead of +21.8% over FP32), but this value is still 54× the increase with Emu2 (+0.18%, Table R1). The revised paper will show both TF32 scopes.

Table R1. Increase of the MAE of the Hamiltonian matrix entries, measured against DFT, relative to FP32 inference, with the weights trained with Emu2; median over 32 conformers.

inference precisionMAE increase over FP32
TF32, global+21.8%
TF32-FL+9.8%
Emu2+0.18%

2. Please provide operation-level timings before and after optimization and separate the gains from compilation, kernel fusion, and precision emulation.

The operation-level timings requested are below:

Table R2. GPU time in ms of each operation in one SO(2) layer of UMA-M with 2,304 atoms and 179,712 edges. Percentages are shares of the summed time of the five operations.

FP32-REFFP32-FLEmu2
S1: gather, rotation, radial weights, l→ml \to m18.8 (21%)5.6 (10%)1.7 (14%)
radial network5.4 (6%)4.8 (8%)1.3 (11%)
convolution GEMMs54.1 (61%)45.0 (78%)6.6 (55%)
S2: gate4.4 (5%)1.2 (2%)0.6 (5%)
S3: m→lm \to l, inverse rotation, aggregation6.2 (7%)0.9 (2%)1.8 (15%)
layer forward (wall clock)88.758.112.8

Fusion (FP32-FL column) removes most of the non-GEMM work (34.8 → 12.5 ms) but the GEMMs are still bound by the FP32 units (54.1 → 45.0 ms). Emulation then runs them 6.8× faster on the INT8 tensor cores (45.0 → 6.6 ms). At the same time, S1, S2, and the radial network get faster because the kernels pass INT8 slices and run the radial network on the tensor cores.

The gains from compilation, kernel fusion, and precision emulation are separated below. Emulation gives the largest gain, and Emu2 is also faster than our fused TF32 kernels (TF32-FL), 1.06× for MALOQ, 1.23× for UMA-M:

Table R3. Step time in ms. Each row adds one optimization to the row above, with the speedup over that row in brackets.

MALOQ training (lmax⁡=4l_{\max}=4, 2,137 atoms)UMA-M MD step (lmax⁡=4l_{\max}=4, 1,008 atoms)
FP32-REF, eager1,204896
+ torch.compile824 (1.46×)761 (1.18×)
+ fusion (FP32-FL)631 (1.31×)660 (1.15×)
+ emulation (Emu2)256 (2.46×)167 (3.95×)
fused TF32 (TF32-FL)271206

3. Could the authors evaluate UMA training, comparing Emu2 and FP32 under identical settings for convergence and final energy and force validation errors?

Training UMA requires using on the order of a hundred H200 GPUs for weeks (28 days for UMA-M and 10 days for UMA-S) [1]. To provide the required information without incurring a tremendous computational cost, we trained the UMA-S architecture (without MoLE) from scratch on the MPtrj datasets with Emu2 and with FP32 under identical settings (the same recipe, seed, initial weights and batches). The errors in the final weights are reported below:

Table R4. Errors in the final weights, all evaluated in FP32 on 80,105 frames held out from training. Brackets give 95% bootstrap intervals over materials.

trainingE MAE (meV/atom)F MAE (meV/Å)ΔE vs FP32ΔF vs FP32
FP32 (fairchem)14.4642.3––
Emu214.5442.3+0.5% [−0.7, +1.8]+0.1% [−0.7, +1.0]
TF32-FL14.5042.1+0.2% [−1.1, +1.5]−0.3% [−1.2, +0.5]
BF16 (fairchem, autocast)15.5153.9+7.3%+27.6%

Emu2 matches FP32 within the statistical noise, and its training loss tracks the FP32 curve over the whole run, ending only 0.3% above it. Therefore, we see no sign of errors being amplified through the second derivatives.

Note that we cannot include this information in the paper, because UMA is defined in the community as the collection of datasets trained on through the MoLE approach. Hence, calling a model 'UMA' while training on different or smaller datasets might mislead readers.

4. Please provide energy-versus-time curves for the Emu2–FP32 comparison in Appendix C and report the integration timestep, total simulation duration, and energy drift rate.

As requested, we will provide ‘energy vs time’ curves to show long term stability in the Appendix. We ran 56 NVE simulations of 100 ps each (200,000 velocity Verlet steps of 0.5 fs), starting from velocities at 600 K, for 8 materials randomly drawn from MPtrj, which were not seen in training.

Table R5. Drift rate (slope of a linear fit of the total energy against time) and RMS fluctuation around that fit: median over the 8 materials, range over the models (the UMA-S architecture trained in FP32 or with Emu2, and UMA-M 1.1).

routedrift rate, absolute (10−510^{-5} meV/atom/ps)RMS fluctuation (10−310^{-3} meV/atom)
FP320.5–1.00.7–0.9
Emu20.9–2.01.0–1.3
TF32-FL (UMA-M only)458.7

Over 100 ps, the drift of Emu2 adds up to 0.9–2.0 μeV/atom. That is about twice the drift of FP32 (1.7–2.1×), and its energy fluctuates 1.4–1.6× as much. On UMA-M, TF32-FL drifts 86× as much as FP32 and fluctuates 10.8× as much.

5. What is the specific technical contribution beyond combining integer Ozaki emulation, block scaling, and kernel fusion?

Our implementation is specialized to the series of operations contained within the SO(2) blocks, as well as the relative numerical range of each operation. Our approach was to first use kernel fusion to slowly remove overhead and push the runtime into GEMMs. Second, we introduced integer emulation via the Ozaki scheme to accelerate the raw GEMM operations by running them on the tensor core units of GPUs. Finally, we tuned the scaling of these emulated GEMMs by taking advantage of the known numerical ranges of the different tensors present in SO(2) convolutions (e.g., the weights are evenly distributed, while the activations spread over orders of magnitude within a row). We summarize the technical contributions as follows:

First, we show that integer Ozaki emulation can replace FP32 when training equivariant models, including the double backward pass required by the training of forces. With Emu2, the training loss of MALOQ matches that of FP32 (Fig. 5(a)) and the model of point 3 above reaches the accuracy of FP32, while BF16 falls short in both cases.

Second, we designed new kernels for emulation with block scales, which is not a simple combination of both techniques. For each input block of 128 entries, Emu2 needs three times the tensor core instructions of a plain INT8 GEMM, plus a rescaling in FP32 that existing emulated GEMMs [2] do only once at the end, and that here takes about as long as the products. Our kernels hide this cost through warp specialization, with TMA prefetching and two warp groups that run out of step, in the forward, backward, and double backward passes.

Third, we use the INT8 slices as the storage format. FlashSO(2) saves the activations for the backward and double backward passes as INT8 slices instead of FP32 (Fig. 3). Two slices take about two bytes per entry, like BF16, but are accurate enough to match FP32 in training, so the peak memory is halved (Section 4.3), and since the slicing is fused into S1 to S3, it adds little time.

All together, that makes FlashSO(2).

6. Can Emu2/Emu3 provide similar performance benefits on smaller models such as UMA-S?

Yes, FlashSO(2) provides similar performance benefits on UMA-S: specifically, 3.61× speedups in inference compared to FP32 and 2.20× speedups compared to TF32:

Table R6. Speedup over fairchem FP32 for UMA-S, both sides compiled: training with forces on 2,304 atoms and inference of energy and forces on 5,808 atoms.

UMA-Sfairchem FP32fairchem TF32TF32-FLEmu2Emu3
training1.00×1.72×2.14×2.31×1.32×
inference1.00×1.64×3.03×3.61×2.24×

In our paper, we originally chose to demonstrate our speedups on UMA-M because it (1) shows greater accuracy overall, (2) has been found especially accurate for challenging material systems involving disorder where smaller models still fail [3], and (3) is only relatively less accessible to the materials modeling community due to the computational cost mentioned by the reviewer.

7. Does the proposed method generalize to other SO(2)-equivariant models, such as EquiformerV2/V3 and DPA4?

Yes, our method is applicable to both the Equiformer series of models and DPA4 because they rely on eSCN-style operations too. eSCN and "SO(2)-equivariant models" denote the same operation, eSCN was simply the name under which it was first introduced. Both models mentioned use the same SO(2)-style convolution block, in the same format as us. As a consequence, the emulated GEMMs of FlashSO(2) can be directly ported, while the fused kernels around them may need to be adapted to each model; we see no fundamental obstacle to this.

To summarize our responses, we (1) isolated the benefits of emulation to show more clearly that it is the majority contribution towards all headline speedups, (2) proved the numerical stability of FlashSO(2) through long-runtime MD simulations with UMA, and (3) showed that training accuracy is exactly maintained with Emu2. We also clarified that it is applicable across all SO(2) equivariant models available today, and that speedups are also significant with smaller models like UMA-S. We hope that these additional experiments and clarifications will strengthen the reviewer's level of confidence in the impact of FlashSO(2) towards accelerating materials simulations.

References

[1] B. M. Wood et al., UMA: A family of universal models for atoms, NeurIPS (2025).

[2] D. Lu, A. Maeder, M. Luisier and A. N. Ziogas, EmuGEMM: Fused Tensor Core kernels for precision emulation in matrix multiplication, arXiv:2606.25453 (2026).

[3] N. L. Fragapane and V. L. Deringer, Amorphous materials as a frontier challenge for universal interatomic potentials, arXiv:2607.11384 (2026).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion and the revision

We thank the reviewers and the AC for the constructive discussion. In short, Emu2 stays close to FP32 in accuracy while being faster than TF32 and using less memory, so the fair reference for FlashSO(2) is FP32. The reviews raised four main points, all addressed with new experiments.

  1. Gain of emulation alone (PCc2, o3JA). We separated compilation, fusion and emulation, and emulation gives the largest share. Emu2 is 2.5 to 4× faster than our fused FP32 kernels (Table 6), and Tables 8 and 9 now list fused FP32 and fused TF32 for every model and lmaxl_{max}.

  2. Accuracy of MLIPs, including the double backward (PCc2, ub3a, o3JA). The UMA-S architecture trained from scratch with Emu2 matches FP32 within statistical noise (Table R4 in our reply to PCc2), and the released UMA-M 1.1 keeps its accuracy with Emu2. We also added orbital energies, 100 ps NVE runs with energy curves, and an equivariance test with trained weights (Appendix D.2).

  3. Timings per operation and smaller models (PCc2, ub3a). Appendix D.3 gives the time of each operation, and Table 9 adds UMA-S and UMA-L.

  4. TF32 restricted to the SO(2) layer (PCc2, o3JA). TF32-FL trains about as well as FP32, but it is not accurate enough at inference. It exceeds the equivariance tolerance, raises the Hamiltonian error far more than Emu2, and has 86× the NVE energy drift of FP32 (Appendix D.2).

The revised PDF contains all results promised in our responses, and all text that is new relative to the submission is shown in blue.

Reviewers o3JA and ub3a confirmed that our responses addressed their concerns, and Reviewer ub3a raised their score. Reviewer PCc2 found that some concerns were addressed, and we replied to their remaining point on TF32 in their thread.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.