acceptodds
Under review as a conference paper at ICLR 2027

On the Steeper Scaling Laws of Equivariant Neural Force Fields

Abstract

Equivariance to Euclidean symmetries is a critical inductive bias for neural networks that model 3D atomistic systems. Whether to build equivariance into a network or learn it through data augmentation remains an active debate. Prior work has empirically shown that equivariant networks achieve steeper scaling laws on certain tasks, raising the open question of what causes this gap. This paper takes a step toward an answer in the setting of machine learning interatomic potentials (MLIPs), where the targets are equivariant forces. In particular, we show that a theory of scaling laws based on source-capacity conditions explains this gap. The capacity and source exponents of each architecture that reflect the decay rate of the network's kernel spectrum and its alignment with the target forces, respectively, account for the observed exponents in the empirical scaling laws. Our results show that for equivariant targets, equivariant architectures scale better than their unconstrained counterparts, because their kernels align better with the targets. Since these exponents can be estimated from an early training checkpoint, they provide a practical way to forecast the scaling laws of MLIP architectures without full-scale training.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
neural scaling laws, equivariance, source-capacity conditions

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9Jq4OpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the different scaling exponents of equivariant neural force fields and unconstrained neural force fields with rotational data augmentation. The authors construct empirical NTK from Jacobians at early training checkpoints and estimate their eigenvalue-decay and target-alignment exponents. In experiement comparing equivariant eSEN with rotationally augmented Orb on the OMol25, early-checkpoint empirical NTK analysis reveals broadly similar spectral decay but stronger alignment for eSEN, consistent with its steeper model-size scaling. Additional experiments with other architectures further examine the connection between early kernel measurements and scaling exponents.

Critical strengths

The experiments provide useful evidence linking target alignment to scaling behavior. After the initial training phase, eSEN consistently exhibits stronger measured target alignment than Orb and a steeper fitted model-size scaling exponent. The comparison across four architectures broadens the evidence beyond a single model pair. These findings may help guide architectural choices and the allocation of training resources for neural force fields.

Critical weaknesses
  • The conceptual advance appears limited, and the empirical contribution needs stronger support. The main potentially novel contribution is the connection between early-checkpoint force-NTK measurements and subsequent scaling behavior. However, the novelty of the multi-epoch evaluation is overstated: lines 87–91 characterize [A] as considering only single-epoch training, although Section 4.5 and Figures 8–9 of [A] already compare augmented unconstrained models with eSEN over 100 epochs using 1% of the dataset. The present experiments extend this comparison to Orb and a larger repeated-data subset. The authors should acknowledge this overlap and demonstrate the additional explanatory or predictive value of the NTK analysis beyond the previously observed scaling differences.
  • The experimental setup differs from the assumptions of the theory. The cited theory assumes squared loss and gradient descent/SGD, whereas the experiments use absolute energy error, an unsquared force norm, and Schedule-Free AdamW. The analysis also considers only the force-output NTK, although energy supervision contributes to updates of the shared parameters. The authors should justify why the prediction aP=αβa_P=\alpha\beta remains applicable to this setup or provide control experiments using force-only squared loss and SGD.
  • The practical benefit of NTK-based forecasting is not demonstrated. The paper does not report the runtime or peak memory required for the NTK calculation and spectral analysis. Using an early checkpoint does not establish that the complete forecasting procedure is inexpensive, since these computations introduce additional costs. The authors should report the total computational cost and compare the approach with cheaper forecasting baselines to substantiate its practical benefit.

[A]: Ngo et al. "Scaling laws and symmetry, evidence from neural force fields." ICLR 2026.

Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for their valuable feedback. We have accommodated these in the updated revision and would like to address the questions and weaknesses here. We would be happy to address any remaining issues during the discussion period.

Weakness 1

... the novelty of the multi-epoch evaluation is overstated ...

We have updated the contribution section to better acknowledge the prior multi-epoch training results. The reviewer is correct that our main contribution is explaining the different scaling exponents for architectures reported in [1], with the addition of the widely used Orb architecture, for which our analysis also provides a successful explanation/prediction.

Weakness 2:

... cited theory assumes squared loss and gradient descent/SGD, whereas the experiments use absolute energy error, an unsquared force norm, and Schedule-Free AdamW ... The analysis also considers only the force-output NTK, although energy supervision contributes to updates ...

We have added additional results in Appendix E showing that using SGD or Schedule-Free AdamW with squared error loss on forces only. The trends are similar, and the predicted exponents do not change substantially. We discuss this in detail in Appendix E.

Weakness 3

runtime or peak memory required for the NTK calculation and spectral analysis. Using an early checkpoint does not establish that the complete forecasting procedure is inexpensive.

We now report these quantities in Appendix C, which shows a negligible cost compared to training a single model, let alone a full stack of models at every scale. The rough difference is one day (for multiple random seeds) using our predictions compared to one week of GPU-hours required to train a model such as eSEN at every size.

[1] Ngo et al. "Scaling laws and symmetry, evidence from neural force fields." ICLR 2026.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer,

Thank you again for your time and effort in reviewing our paper.

As the discussion period is coming to a close, we would like to kindly ask whether our responses and additional experiments have satisfactorily addressed your concerns. We would be happy to provide any further clarification if needed.

Best regards,

The Authors

Reviewer_L3ndOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a detailed study of the scaling of networks trained to model 3D atomistic systems, comparing equivariant versus data-augmented. It characterizes the scaling by characterizing the networks as kernel regressors and measuring the capacity and source exponents of each architecture. Their main finding is that the steeper scaling of equivariant networks with respect to the network size and the training time is due to a better alignment between the training targets and the most expressive network directions. Furthermore, it introduces a neural tangent kernel (NTK) that includes the contributions from the higher-order representations of the equivariant network. All of its predictions are studied on realistic settings, and multiple architectures are tested.

Critical strengths
  • This paper applies notions from scaling law theory to the setting of neural force fields and adapts it to equivariant networks with high-order tensor representations.
  • It proposes a concrete explanation for the different observed exponents on equivariant and unconstrained networks and tests it in a realistic training setting.
Critical weaknesses
  • I believe that some of the assertions that are made about the results presented in the paper are not properly justified. More specifically, there is no description about the fitting procedure that is done for any of the presented fits. This is relevant for the results in Figure 2, where the R(k) evolution does not seem to follow a power law at all, and segments of the range seem to be arbitrarily picked to extract the exponent. Furthermore, the fits in Figure 4 are made while assuming that the irreducible loss is zero, but that is clearly not the case for some of the curves. Finally, there is no explanation for why a reduced number of networks is used for the Orb baseline in Figure 5. All of these points are important because they influence the main conclusions of the paper.
  • A point is made about how the empirical NTK captures the higher-degree features while a kernel built from the invariant l=0 block alone cannot, but its justification is insufficient. This is important because one of the main novelties of the paper is the construction of a new NTK for force field networks.
Questions for authors
  • What is the method used to produce all the fits?
  • It is claimed that the gap between architectures and their orderings stay fixed across steps, but that is not quantified properly. What is the actual gap and how little does it evolve as the network is trained?
  • What are the jumps in the slope for the R(k) curves in Fig. 2?
  • How do the exponents change if you fit the irreducible loss along with the other parameters in Fig. 4?
  • Why do you only fit the model-size scaling for the extended architectures? Why do you use only 4 points for Orb?
  • Have you explored how the scaling of your studied networks changes with depth? You say in the appendix that you only increase the network size by expanding the width
  • As far as I understand, the theory behind the NTK only applies to networks with infinite width or random feature linear models. What are the limitations of such an approach when applied to finite-width networks that feature non-linearities?
  • I do not understand how the force outputs by virtue of being a gradient allow you to capture higher-order representations better than just looking at the scalar outputs, specially when you seem to build the outputs as vector readouts and not as energy gradients. How is that different from just using the l=1 block outputs? Could you explain this better?
5 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for their valuable feedback. Below, we would like to address the weaknesses and questions in their review. Questions and weaknesses that are about the same issues are grouped together. We would be happy to address any remaining issues during the discussion period.

Weakness 1, Questions 1,3, 4

...no description about the fitting procedure that is done for any of the presented fits...assuming that the irreducible loss is zero, but that is clearly not the case for some of the curves...

  • Scaling law fits:
  1. We would like to clarify that we did not assume irreducible errors to be 0 when fitting data points in figure 4 (figure 5 in ther revised version). We fit a joint power law of training time tt, model size NN, and irreducible loss L∞L_\infty. Our fits consistently show L∞≈0L_\infty \approx 0 for both architectures. We link these results to the fact that ab initio force labels are a deterministic function of atomic configuration within the Born-Oppenheimer approximation and are therefore practically noise-free [1].

  2. Please note that the individual curves in the figure are not necessarily straight lines in log-log space. A pure power law would appear linear, but each run uses a fixed model size, which bounds the loss it can achieve. As the run progresses, the model’s capacity rather than the training time becomes the bottleneck, and the curve flattens toward a size-dependent floor. Therefore, we fit a joint scaling law that accounts for both factors simultaneously.

  • Residual curve fits: We agree with the reviewer that the empirical measurements of R(k) do not fully follow a clean power law. We believe this is because measuring empirical NTK in practice is computationally expensive, and we built our kernels using at most 10K atoms. Given the finite kernel, our fit procedure is to scan over a window of [kmin,kmax][k_{min}, k_{max}], where kmink_{min} indicates the first eigenmode index at which the residual curve has the first cliff, and kmaxk_{max} is the first eigenmode index at which the trends deviate from the power law because of the finite-sample tail. We discuss this procedure in detail in Appendix E.

  • Confidence Intervals: We reported the confidence intervals for the presented fits to ensure they are statistically meaningful.

Weakness 2, and Question 8

... point is made about how the empirical NTK captures the higher-degree features ..., but its justification is insufficient.

This is a valid issue, and we have updated the paper to explain this choice further. In particular, we have added results in Appendix F showing that using vector blocks (ℓ=1\ell=1) does not produce meaningful results. Beyond invariant features, it is also unclear what the right procedure is for obtaining the kernels, as one may imagine different approaches, such as using inner products of vectors. Related to our current argument is also [2], which shows that representations learned by higher-order networks exhibit different levels of information across tensor orders. Even when trained on scalar targets such as energies, these networks can still learn meaningful structures at ℓ=4\ell = 4, while their representations at ℓ=1\ell = 1 and ℓ=2\ell = 2 collapse.

Question 2

It is claimed that the gap between architectures and their orderings stay fixed across steps, but that is not quantified properly. What is the actual gap and how little does it evolve as the network is trained?

This claim is supported by our empirical results in figure 2 of the revised version, which show that the gap in αβ\alpha\beta between eSEN and Orb remains consistent from step 0 to step 50K. This gap is reflected in the time exponents of the resulting scaling laws, with eSEN achieving an exponent of 0.75 compared to 0.61 for Orb. This argument was originally discussed in [3], which suggests that the relative ordering of architectures during training should be preserved from initialization.

To provide a more precise comparison, we report below a table of the estimated αβ\alpha\beta values obtained from the residual curves at each training step shown in figure 2.

stepeSENOrbratio
10k0.62 [0.60, 0.65]0.21 [0.17, 0.26]2.9×
20k0.63 [0.62, 0.65]0.25 [0.23, 0.27]2.5×
30k0.68 [0.66, 0.70]0.27 [0.25, 0.29]2.5×
40k0.73 [0.71, 0.75]0.29 [0.27, 0.31]2.5×
50k0.78 [0.74, 0.81]0.30 [0.29, 0.32]2.6×
AuthorsOpenReview ↗

Official Comment by Authors (continued)

Question 5

Why do you only fit the model-size scaling for the extended architectures? Why do you use only 4 points for Orb?

This is an interesting point, and we have considered it. Since we estimate both αβ\alpha \beta and α\alpha, we can infer β\beta, which, for a single-epoch training, in theory corresponds to the data scaling exponent. However, the same theory tells us that the training-time exponent changes during training as a function of β\beta, and we should use its late values for this estimate (as opposed to early values for αβ\alpha \beta) [3]. As for the number of data-points for Orb, We have updated the figure by adding three more data points for Orb (seven in total).

Question 6

Have you explored how the scaling of your studied networks changes with depth? You say in the appendix that you only increase the network size by expanding the width

In scaling with width rather than depth, we are following the protocol that produced the clean scaling laws that we study [4]. In that work, the authors measure loss by scaling depth while keeping the total number of parameters fixed and report depth saturation for each architecture. Those optimal depth values are then fixed in the scaling studies while the width is varied.

Question 7

rephrased: What are the limitations of NTK applied to finite-width networks with non-linearity?

  1. A general limitation of the approach is that it cannot capture the training-time scaling exponents of strong feature-learning networks in practical settings, as discussed in several theoretical works [3]. Regarding our empirical fits, we observe that both eSEN and Orb exhibit strong feature-learning behaviours, with their β\beta exponents increasing over the course of training. Nevertheless, the gap between the two architectures remains relatively consistent throughout training.

  2. Model-size exponents are related to the fact that a model of size NN can, at best, learn the top NN eigenmodes as training time approaches infinity (t→∞t \rightarrow \infty), as well as how its leading eigenmodes align with the targets. [5] showed that LLMs exhibit a robust model-size scaling law with an exponent of 1/31/3, regardless of the data spectrum. This behaviour is attributed to superposition, which allows networks to represent substantially more features than their width would conventionally permit. However, to our knowledge, there is currently no concrete evidence that neural force fields operate in this regime. We also added a brief discussion about this in Section 6 (Discussions).

[1] Unke, O.T., Chmiela, S., Sauceda, H.E., Gastegger, M., Poltavsky, I., Schutt, K.T., Tkatchenko, A. and Muller, K.R., 2021. Machine learning force fields. Chemical reviews, 121(16), pp.10142-10186.

[2] Lee, K.L.K., Galkin, M. and Miret, S., 2024. Deconstructing equivariant representations in molecular systems. arXiv preprint arXiv:2410.08131.

[3] Bordelon, B., Atanasov, A. and Pehlevan, C., 2025, May. How feature learning can improve neural scaling laws. In International Conference on Learning Representations (Vol. 2025, pp. 51909-51939).

[4] Ngo et al. "Scaling laws and symmetry, evidence from neural force fields." ICLR 2026.

[5] Liu, Y., Liu, Z. and Gore, J., 2026. Superposition yields robust neural scaling. Advances in Neural Information Processing Systems, 38, pp.159269-159305.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer,

Thank you again for your time and effort in reviewing our paper.

As the discussion period is coming to a close, we would like to kindly ask whether our responses and additional experiments have satisfactorily addressed your concerns. We would be happy to provide any further clarification if needed.

Best regards,

The Authors

Reviewer_XGu4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work studies the difference in scaling laws of equivariant neural network compared to unconstrained architectures through the lens of Neural Tangent Kernels (NTK). Specifically, the authors, following previous work, first show empirically that NTK computed in early stages of training can provide accurate predictions of the scaling law in both the equivariant and non-equivariant networks they study that are trained on the task of molecular force predictions. Then they show how the steeper scaling laws observed for equivariant networks can be attributed to the better alignment of the target labels with the primary modes of the empirical NTK kernel rather than to the difference in decay of the eigenvalues of these kernels.

Critical strengths
  • The authors provide an interesting empirical study of the scaling laws for equivariant networks and empirically establish a connection between the NTK spectrum of networks during the early stages of training and their scaling laws.
  • The results provide an interesting interpretation of the advantage of equivariant neural networks over unconstrained alternatives, which can be interesting to the community that designs such networks.
Critical weaknesses
  • The network studies a very specific task and class of networks, making it difficult to judge how these conlcusions generalize in different types of tasks, even in the same field of molecular property prediction or generation
  • There is little discussion about works that showcase that relaxing the equivariant constraints can help with network learning, which suggests an opposite direction to the results showcased in this work. Some such works are:
    • A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models (YuQing Xie and Tess Smidt 2025)
    • Recurrent Equivariant Constraint Modulation: Learning Per-Layer Symmetry Relaxation from Data (Pertigkiozoglou et al. 2025)
    • Learning (Approximately) Equivariant Networks via Constrained Optimization (Manolache et al. 2025)
Questions for authors
  • How well we can access the generalization of these results to other networks and tasks even for similar molecular property prediction
  • How does this result go against or support previous works that observe benefits from relaxing equivariant constraints even in equivariant tasks?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for valuable feedback. Below, we would like to address all of their weaknesses and concerns. We would be happy to address any remaining issues during the discussion period.

Weakness 1 and Question 1

... difficult to judge how these conlcusions generalize in different types of tasks ...

  1. While we agree that a more general statement would be interesting, we are unaware of clean scaling law results with model and dataset size for other tasks within the same field. Current scaling-law studies require sufficiently large datasets with dense labels. To our knowledge, force-field modelling provides the cleanest setting for studying differences in the scaling behaviours of various architectures. This is because it offers dense, atom-level force labels, and the community is actively developing increasingly large datasets for this task. In contrast, molecular property prediction typically relies on small- to medium-scale datasets, which we believe belong to a different data regime.

  2. Our contribution targets equivariant tasks, such as force prediction. More precisely, the underlying mechanism—alignment between the leading modes of the kernel and an equivariant target—is specific to labels that transform as vectors or higher-order tensors. Beyond force-field modelling, we do not claim that our findings generalize to molecular property prediction tasks with scalar targets.

Weakness 2 and Question 2

... discussion about works that showcase that relaxing the equivariant constraints can help with network learning...

We have added a brief discussion, citing these papers. However, we are unaware of any prior work studying scaling laws associated with relaxed equivariance, beyond the equivariance penalty method, which [1] shows is ineffective in improving the scaling exponent. Since we are explaining the different scaling laws associated with different architectures, a prerequisite is for such laws to have been first identified in a class of networks or training methods. We would be happy to discuss this further if the reviewer has a specific point in mind.

[1] Ngo et al. "Scaling laws and symmetry, evidence from neural force fields." ICLR 2026.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer,

Thank you again for your time and effort in reviewing our paper.

As the discussion period is coming to a close, we would like to kindly ask whether our responses have satisfactorily addressed your concerns. We would be happy to provide any further clarification if needed.

Best regards,

The Authors

Reviewer_XGu4OpenReview ↗

Official Response by Reviewer XGu4

I thank the authors for answering my questions and concerns I raised during my review regarding possible limitations of this work. I agree that the task of providing scaling laws for a broader class of models and tasks can be challenging, and the current approach of focusing on the specific scaling laws for equivariant force field prediction allows the authors to provide novel results and insights regarding the task. Nevertheless, this more limited scope makes it harder to generalize these insights and place them within the broader literature of equivariant models, and due to that, I will keep my score of weak accept.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank all reviewers for their time and constructive feedback, which has helped us improve the paper. We acknowledge several valid concerns regarding the extent to which our results support the central claims. In response to the reviewers' suggestions, we have made the following key updates to the submission (highlighted in blue):

  1. Analysis of higher-order equivariant kernels. We have added Appendix F, which explains why using features from l=1l=1 blocks alone is insufficient to construct a meaningful kernel for higher-order equivariant networks. This section includes both theoretical derivations and empirical results to further support our argument.

  2. Additional data points for model-size scalings. We have updated the figure with three additional data points of Orb, bringing the total to seven. The main conclusion remains unchanged, with the power-law trend continuing to hold as additional data points are included. It is now Figure 1 in our revised submission.

  3. Computational cost and training objective ablations. We have added discussions to Appendices C and E addressing the computational cost of estimating the scaling coefficients. We have also included ablation studies examining how changing the training objective to MSE affects our central claims.

  4. Discussion of relaxed equivariance. We have expanded the Related Work and Limitations sections to discuss existing approaches to relaxing equivariance constraints.

  5. Additional details on empirical NTK estimation and curve fitting. We have expanded Appendix D to provide further details on our procedures for estimating the empirical NTK and fitting coefficients to the kernel spectra and residual curves.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.