- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
LunaTP introduces fused GPU kernels for Clebsch–Gordan tensor-product convolutions while preserving the original formulation and coupling paths. Its node-centric execution accumulates contributions from incoming edges on chip before writing the results to global memory. Register-aware path grouping balances input reuse against accumulator storage, while task ordering accounts for differences in node degree and path cost. The generated kernels exploit CG sparsity and reuse shared inputs, products, and coefficients.
The implementation extends these optimizations to backward, double-backward, and triple-backward computations. Evaluation covers kernel performance, numerical accuracy, end-to-end training, and molecular dynamics across multiple models, GPUs, and numerical precisions.
Critical strengths
Broad applicability and practical derivative support. LunaTP accelerates existing CGTP-based models without restricting their coupling paths or requiring retraining. The optimization addresses both computation and graph reduction, rather than accelerating isolated tensor products alone. Higher-order derivative support extends its usefulness from inference to force and Hessian training. Numerical comparisons against e3nn cover the forward pass and derivatives through third order in FP32 and FP64. The NequIP-XL training experiment additionally uses the same initialization and training settings for both backends, obtaining similar convergence and validation errors differing by less than 0.3% across energy, force, stress, and Hessian targets. These checks provide meaningful evidence of correctness beyond the performance measurements.
Extensive evaluation on contemporary SOTA models and optimized baselines. Training benchmarks cover eight architectures: MACE-L, NequIP-XL, TACE-L, SevenNet-Omni, EquFlashV2, NequIX, Prophet, and BAM. Comparisons include OpenEquivariance, cuEquivariance, and FlashTP, rather than relying only on gains over e3nn. Additional experiments examine different GPUs, precisions, system sizes, graph densities, angular degrees, and channel widths. Matched-batch and maximum-batch results are clearly distinguished, helping separate execution improvements from larger-workload benefits. The MD benchmarks further demonstrate practical gains on TACE-L and NequIP-XL, reporting higher speed than OpenEquivariance at the reported large-system workloads.
Critical weaknesses
Further memory optimization remains an opportunity. The paper reports that memory consumption is comparable to FlashTP and OpenEquivariance from forward through double-backward, although LunaTP provides further savings for triple-backward. The convolution still consumes path weights generated by the radial MLP. Given the demonstrated speed improvements, extending fusion to radial-weight generation could offer additional memory savings by avoiding materialized edge weights. I would welcome discussion of combining LunaTP with streaming ideas from Sobek. This is a potential extension rather than a demonstrated flaw or a result established by the current paper.
Questions for authors
Beyond avoiding materialized radial weights, what similarities and differences exist between Sobek and LunaTP in graph reduction, on-chip accumulation, computational reuse, and higher-order differentiation? Which aspects of Sobek’s streaming design could complement LunaTP’s current optimizations?
4 comments
Q1: Similarities and differences between Sobek and LunaTP
Both Sobek and LunaTP use node-centric reduction and on-chip accumulation, but differ in work partitioning, computational reuse, and higher-order differentiation, as discussed below.
Graph reduction: Both implementations use node-centric reduction. Sobek partitions forward tasks by node and channel block, while LunaTP additionally partitions work across path groups, allowing different path groups for the same node to execute in parallel.
On-chip accumulation: Both implementations keep output accumulators live during neighborhood traversal and accumulate neighbor contributions on chip. LunaTP constrains path-group size using a per-channel output component budget. This preserves input reuse within each group while controlling register usage to maintain sufficient parallelism.
Computational reuse: Sobek processes pairs of paths that share inputs during double backward to reduce repeated loads. LunaTP reuses inputs through path grouping and further reuses intermediate products within each path. It also combines product terms that share coefficients to reduce redundant computation.
Higher-order differentiation: Both implementations organize reductions according to the differentiation target and use dedicated double-backward kernels. LunaTP extends path grouping and within-path reuse to derivative computation. It fuses derivative branches that share inputs and assigns independent branches to different CUDA streams to overlap their execution. These mechanisms are further extended to dedicated triple-backward kernels.
In terms of feature support, Sobek supports only FP32 and requires all input and output node irreps to have the same multiplicity, with an edge irrep multiplicity of 1. In contrast, LunaTP does not have these restrictions.
A detailed comparison of these implementations has been added to Appendix A.
Please let us know if you have any feedback to our explanation or other questions. We will be very happy to respond.
Thank you for the detailed explanation and clarification. I look forward to the release of LunaTP and its broader adoption by the community. In particular, we look forward to releasing a general-purpose interface for Sobek-style fused operations, as Sobek's existing interface is not sufficiently general, limiting its widespread adoption by the community. Existing engines that employ Sobek-inspired approaches are tailored to specific MLIP architectures rather than providing broadly reusable implementations.
We thank the reviewer for the positive feedback and suggestion. We will ensure that LunaTP provides flexible, general-purpose interfaces that are easy for the community to use.
We thank the reviewer for highlighting LunaTP’s broad applicability and extensive evaluation. Below, we discuss further memory optimization and respond to the questions about Sobek.
We thank you for the insightful comment. We agree that extending the fusion scope to radial-weight generation can further reduce memory usage.
Sobek uses streaming execution to integrate radial projection, tensor products, and graph aggregation, avoiding the materialization of full per-edge weights and their gradients. This approach can complement LunaTP's computational reuse. In addition, Sobek's strategy of saving or recomputing node aggregates under a workspace memory budget could help control temporary memory usage as the fusion scope expands.
We have implemented a preliminary version that fuses the final Linear projection of the radial network with TP convolution. Specifically, the kernel computes radial projections in edge blocks and temporarily stores the resulting weight blocks in shared memory. The same kernel then performs CG contractions and local neighborhood aggregation, avoiding the global storage of full per-edge weights.
We evaluated the fused version on an H100 GPU (FP32, Linear input dimension 64). LunaTP + Linear and LunaTP (fused Linear) denote the separate and fused implementations, respectively. Entries report runtime (ms) / peak memory (MiB).
Across these configurations, LunaTP (fused Linear) reduces peak memory usage by 82.6%–97.1% and achieves 1.21–1.35× forward speedups over the separate implementation. Compared with Sobek, it achieves 2.39–29.01× speedups from forward through double backward with comparable or lower peak memory usage. In double backward, peak memory usage is reduced by 41.9%–59.1%. The fused version also supports triple backward.
LunaTP (fused Linear) is still at an early stage of optimization, and its higher-order backward passes currently take longer than those of the unfused version. We found that fusion increases register pressure, while shared-memory accesses and synchronization between tasks also introduce additional overhead. Nevertheless, these results show that fusing radial projection with TP convolution can significantly reduce LunaTP's memory usage. We are continuing to optimize this kernel and commit to open-sourcing the implementation for the community.