- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper introduces a benchmark for fine-tuning universal machine learning interatomic potentials (uMLIPs) on experimental data. The benchmark consists of five tasks: titanium elastic constants, titanium hcp-bcc phase transition, titanium liquid structure factor, diamond elastic constants, and MgO elastic constants. The work finetunes six uMLIPs on the five tasks using differentiable trajectory reweighting (DiffTRe) then evaluates the models on the fitted observables and on other temperatures, polymorphs, properties, and compositions. The fine-tuned models consistently improve fitted observables while the transfer to other observables or compositions is inconsistent. The paper also finds that fine-tuning changes the ranking of pretrained models.
Critical strengths
Well Motivated: Most MLIP benchmarks evaluate pretrained models against DFT or experiment. Evaluating models after experimental fine-tuning is a meaningful extension.
Potential for standardization: Consolidating experiments from different works into a common framework provides reproducible, consistent, and fair comparison across potentials. The authors will provide code and benchmark data further supporting this.
Empirical evaluation across multiple potentials: Six models with different architectures and pretraining datasets are evaluated using a shared experimental fine-tuning procedure. The work examines model performance on the fine-tuning observable and other observables.
Critical weaknesses
Limited benchmark diversity. Although the paper presents five tasks, they involve only three compositions and three of the five tasks are elastic constants. A benchmark for uMLIPs, which are intended to represent a much broader range of elements and chemical environments, should include a more diverse set of compositions, material classes, and physical properties.
Limited novelty of the benchmark dataset. Three of the five tasks (diamond elastic constant, titanium elastic constants, and titanium hcp-bcc phase transition) reproduce previously published experimental MLIP fine-tuning applications.
Limited diversity of experimental fine-tuning methods. The benchmark uses one previously published experimental fine-tuning method. Including alternative fine-tuning strategies would strengthen the benchmark and determine whether the observed transferability, catastrophic forgetting, and changes in ranking depend on the fine-tuning method.
Questions for authors
Could the authors provide more materials, observables, and fine-tuning strategies to strengthen the benchmark for uMLIPs? In Section 3.2 when fine-tuning on diamond, the authors observe that corrections transfer to other cubic materials more reliably than the hexagonal one. Could the authors provide more examples to determine whether this observation depends on symmetry, chemistry or other factors? How much of the observed change in model ranking is due to the chosen strategy of fine-tuning instead of fine-tuning on experimental data in general? For all uMLIPs fine-tuned, can the authors report how the fine-tuning affects energy and force errors relative to the DFT datasets used to train the uMLIPs? Why are the regularization studies in Appendix A.5 only reported for the titanium elastic constant and for the MACE and SevenNet models?
4 comments
W3.2.
Including alternative fine-tuning strategies would strengthen the benchmark and determine whether the observed transferability, catastrophic forgetting, and changes in ranking depend on the fine-tuning method.
Transfer, forgetting and the change in ranking all depend on the strategy. The paper fine-tunes on the experimental loss alone, which we call EXP-ONLY, and we added three regularized strategies, DFT-REPLAY, PL-REPLAY and L2-SP, on all six models on the titanium elastic task. A strategy that fits the target less tends to keep the pretrained order more, as Table B shows. On average over the six models, the replay strategies raise the diamond and MgO errors, mostly through PET-MAD, while L2-SP leaves them almost unchanged. DFT-REPLAY also raises the force error on the pretraining data, as Q4 shows.
Table B. Pretrained against fine-tuned order under each strategy, titanium elastic task. Kendall between the pretrained and fine-tuned ranks of the six models, with pairs closer than the measurement floor counted as ties and 95% intervals over seeds and simulation blocks. A of +1 means an unchanged order. The titanium block leaves out ΔG, which only EXP-ONLY measured, so its EXP-ONLY value differs from the full titanium block with ΔG. The block of Table 4 of the revised paper adds diamond and MgO. The on-target SRME is the mean over the six models. DFT-REPLAY and PL-REPLAY replay titanium DFT structures under DFT labels or under labels from the pretrained model, and L2-SP penalizes distance from the pretrained weights. This is Table 20 of the revised paper.
| strategy | on-target SRME | , titanium block [95% CI] | , block of Table 4 [95% CI] |
|---|---|---|---|
| pretrained | 0.213 | – | – |
| EXP-ONLY | 0.022 | +0.45 [+0.28, +0.93] | +0.47 [+0.41, +0.60] |
| L2-SP | 0.055 | +0.41 [+0.33, +0.69] | +0.60 [+0.47, +0.73] |
| DFT-REPLAY | 0.088 | +0.60 [+0.60, +0.89] | +0.47 [+0.33, +0.60] |
| PL-REPLAY | 0.154 | +1.00 [+0.83, +1.00] | +0.47 [+0.47, +0.60] |
Q3.
How much of the observed change in model ranking is due to the chosen strategy of fine-tuning instead of fine-tuning on experimental data in general?
Changing the strategy reorders the models about as much as fine-tuning does. On average, the orders from two different strategies agree with a of +0.53 on the block of Table 4, against +0.50 between the pretrained order and each strategy's order. The strengths come from MACE and SevenNet in Appendix A.5, so these runs show trade-offs, and tuning each strategy for the leaderboard is future work.
Q4.
For all uMLIPs fine-tuned, can the authors report how the fine-tuning affects energy and force errors relative to the DFT datasets used to train the uMLIPs?
The revised paper now reports the energy and force errors on each model's own pretraining data, and under EXP-ONLY both barely change for a typical model and task, as Figure 18 shows. The force error rises by a median 3.5% and at most 1.69 times over the 25 pairs of five models and five tasks, and the energy error by a median 6% and at most 1.68 times. We compare energies after removing a constant shift per element, which the experimental loss leaves free and which changes no force, stress or reaction energy. We exclude MatterSim because its pretraining data are not public. DFT-REPLAY, one of the strategies in W3.2, replays only a titanium DFT set and more than doubles the force error in MACE, while L2-SP leaves it almost unchanged.
Q5.
Why are the regularization studies in Appendix A.5 only reported for the titanium elastic constant and for the MACE and SevenNet models?
The regularization study is an analysis of trade-offs and not part of the benchmark ranking, which uses EXP-ONLY on all six models. We swept its strength on two models, MACE and SevenNet, and on the titanium elastic task, whose source study released DFT data to replay, as far as our compute budget allowed. We have now run the three strategies at those strengths on all six models, as W3.2 shows.
Thank you to the authors for the detailed response and additional experiments.
W1, Q1.
The authors have added five new held-out compositions, but no new fine-tuning tasks and all are evaluated on elastic constants. This effort works to address limitations of the benchmark diversity but its training-task diversity is still limited.
Q2.
The additional transfer experiments work to understand how corrections transfer between materials sharing crystal structure and bonding type. However, these conclusions are based on only three source-target material pairs which have similar local atomic environments or coordination. The results support a limited empirical trend on the tested materials, not an established relationship.
W2.
I agree with the authors that the benchmark is still important to the community even with limited novelty.
W3, Q3.
The results confirm the concern that model rankings depend on the fine-tuning strategy. Consequently, the reported rankings characterize how uMLIPs perform under a particular experimental fine-tuning strategy rather than general ability to adapt to experimental data.
Q4.
Figure 18 shows that experimental fine-tuning can substantially increase energy and force errors on pretrained models. Could the authors discuss whether preserving the pretrained potentials accuracy should be incorporated into the benchmark? Would this be interesting for distinguishing models that achieve similar experimental accuracy but differ in how well they retain their original predictive capabilities?
Q5.
The authors now evaluate three regularization strategies on all six models for titanium elasticity partially addressing the question.
Overall:
The authors have included more experiments to work towards addressing weaknesses and questions. My remaining concerns are the diversity of the experimental fine-tuning tasks and that the benchmark characterizes uMLIP performance under one experimental fine-tuning strategy rather than general ability to adapt to experimental data. The benchmark would be interesting for the uMLIP community so I have increased my weak reject to a weak accept.
Thank you for the follow-up. We agree that both points are worth extending. The benchmark fine-tunes on five tasks and ranks the models under one shared protocol, so that every model is compared on equal terms. Models fine-tuned on more tasks or with other strategies can be scored on the leaderboard, which uses the same evaluation tasks.
Q4.
Could the authors discuss whether preserving the pretrained potentials accuracy should be incorporated into the benchmark? Would this be interesting for distinguishing models that achieve similar experimental accuracy but differ in how well they retain their original predictive capabilities?
Yes. We think the benchmark should report how much pretrained accuracy each model keeps after fine-tuning. This would distinguish models that match experiment equally well but lose different amounts of pretrained accuracy.
Thank you for the concrete and constructive requests. We quote each point and answer it below. The revised paper shows every change in blue, and its new Appendix C holds the new figures and tables.
W1, Q1.
The benchmark's breadth lies in what each fine-tuned model is scored on, and we have now added five new compositions as held-out test targets. Each model fine-tuned on an elastic task is also scored on the phonons and the other compositions, and the titanium models on a held-out polymorph, the transition line and the liquid structure factor. We now also score the paper's models fine-tuned on the diamond, MgO and titanium elastic constants, with no retraining, on the metals Cu and Mg, the covalent Si and the ionic NaCl and ZnO, shown in Table A. We did not add a new fitted observable, because each one needs its own measurements and an MD estimator checked against them, and the five fitted tasks already span elastic, thermodynamic and structural observables. We also scored three more fine-tuning strategies on all six models on the titanium elastic task, as W3.2 below shows.
Table A. Transfer to five new held-out compositions. Rows group the 15 pairs of a training and a new composition by what they share, counted once per model. SRME is the symmetric relative mean error of the elastic constants against experiment. A change counts only beyond the measurement floor, the disagreement between independent experiments, plus two MD standard errors, with all three seeds agreeing in sign. A negative mean ΔSRME means the error on the new composition fell.
Q2.
We added five compositions that separate these factors, and they show that neither symmetry nor chemistry alone explains the transfer, as corrections transfer most reliably when source and target share both crystal system and bonding. Sharing only the crystal system leaves the elastic-constant error unchanged on average, helping and hurting about equally often, while sharing both lowers the error in half of the cases and rarely raises it, as Table A shows. The three pairs that share both, diamond to Si, MgO to NaCl and titanium to Mg, share no element, so the transfer does not rest on chemistry. Section 3.2 now states this close-analogue rule in place of the symmetry explanation.
W2.
We agree that three tasks reuse published applications, but this is deliberate, and the novelty lies in benchmarking uMLIPs on them, which the source studies did not do. Reuse brings checked measurements and MD estimators of each observable, so differences between uMLIPs reflect the models rather than the task, and it gives the system-specific potential of each source study as a baseline. Each source study trained one specialised model. The benchmark instead compares six uMLIPs under one protocol, scores them away from the fitted target, and adds MgO elastic constants and the liquid Ti structure factor as new tasks. We changed "we reproduce" to "we adopt the targets and protocols of".
W3.1.
The benchmark fixes the experimental targets and how each fine-tuned model is scored, not the fine-tuning method, so any method can enter the leaderboard by declaring its training data, loss and schedule. The reference results use differentiable trajectory reweighting, the method of the source studies, as one fixed protocol, and we tested three more strategies on top of it, as W3.2 below reports.