- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper applies conformal risk control (CRC) to MLIPs and proposes using it to inform routing to the ground-truth (typically KS-DFT) backend when the MLIP output is classified as unreliable. Starting from a fully trained MLIP, it is proposed to label a small subset of the targeted evaluation / distribution (by "test" I mean w.r.t the MLIP, not their classifiers / the data split seems intact) They use handcrafted /predefined molecular point cloud features as input to an MLP to predict the energy MAE, Euclidean mean force error, and maximum force error, from which an uncertainty-type score is assigned. They calibrate the decision boundary using CRC on a small hold-out set.
The experiments contain three organic chemistry datasets at three levels of theory...
- COMP6: ωB97X/6-31G(d) (no dispersion correction / smaller basis than MLIPs)
- QCML: PBE0 (non-range-separated, no dispersion /some non-gaussian basis set)
- MACE-OFF23: ωB97M-D3(BJ)/def2-TZVPPD combined with three MLIPs trained on three different levels of theory (all range-separated hybrids with dispersion corrected / large gaussian basis sets):
- MACE-OFF23(M): ωB97M-D3(BJ)/ def2-TZVPPD
- AIMNet2: ωB97M-D3 / def2-TZVPP
- AceFF-2: ωB97M-V / def2-TZVPP
Critical strengths
S1: The method is a lot cheaper than the compared baselines, albeit both are negligible compared to DFT.
- The committees need 3 MLIP evaluations per configuration, the proposed method only 1 (Section 3.1).
Critical weaknesses
W1: In the most important scenario the proposed method underperforms compared to the baselines.
- The "error" being routed is mostly a level-of-theory offset.
- It seems wierd to ask a higher level of theory trained MLIP to predict lower quality energies.
- All MLIPs considered are trained on large basis sets at a range seperated hybrid level of theory that includes dispersion correction.
- The experiments on COMP6 essentially ask to match numerical basis set errors since it uses a much smaller basis set.
- The experiments on QCML remove dispersion corrections and the range seperation evaluating on lower quality hybrid DFT data
- The only remaining evaluations on MACE-OFF at matched theory show that the baseline is better
- Figure 2 excludes exactly this scenario.
- Figure 3a) baselines are better to equal at matched levels of theory
- Figure 3b) baselines save more DFT calls.
- Figure 22: the baselines achieve better relative energies with similar selection success.
- Figure 23: at all values alpha, the baselines achieve better relative energy MAE at similar success rates.
- Given the data, the numbers quoted in the abstract seem to be missleading, and should be clearly state that these improvements are limited to settings where the MLIPs are trained on a higher level of theory and asked to match lower fidelity data.
W2: The proposed routing metrics are miss-aligned with core observables of interest.
- Appendix J Figure 23
- Appendix K shows that the method fails to adapt to physical observables tolerances of interest.
W3: Baselines
- Missing: -learning or fine-tuning on the 40% fitting set, then routing what remains
- The supervised committee baseline seems handicapped
- Single seed-0 network with L1 loss, whereas the proposed score is a three seed ensemble with an additional ranking loss.
- On QCML × MACE-OFF23 it gets G = -0.28 (G = share of the random-to-oracle DFT savings, 0 = random, 1 = oracle) and the native variant (committee of MACE-OFF23 S/M/L models) G = -1.07, identical to sending everything to DFT.
- A log-transformed refit was excluded (see Q3).
Questions for authors
Q1: The prior work section is very thin.
- How does your pipeline compare to
- "calibrated gradient-based uncertainties" Zaverkin et al. 2024
- PROBE.
- uncertainty quantification methods used in active learning approaches, e.g.Rensmeyer et al. 2026, .
Q2: Regarding the reference mismatch
- Your gains come from reference mismatch, and you already label 40% at the target reference. Why not correct the MLIP first with -learning or finetuning on the 40% fitting set?
- Could you please repeat the experiments with exactly equal levels of theory of MLIPs and deployment, in for example a size extrapolation or chemical space extrapolation setting on a standard high fidelity dataset like OpenMolecules25?
Q3: I would be interested in the rationality behind the exclusion of some results mentioned in the supplementary material.
The "VERIFIED_RESULTS.md" file suggests that negative results might have been excluded from the final manuscript. "Transition1x campaigns [...] were excluded." How does the proposed model perform on transition state geometries? Also please provide the log1p results variants for the supervised committee baselines. Good scientific practice demands a complete reporting of all tested variant at least in the appendix.
7 comments
[W1/Q2]-continued
Changes to the claims.
- The old abstract's claim of 7% and 14% fewer DFT calls than committees is removed. The abstract now says that, on reference-mismatched pairs, our predictor matches a supervised three-MLIP committee (which was also improved and is to be mentioned in W3/Q3).
- We no longer claim that our predictor outperforms committees or model-internal predictors in general.
- The CRC guarantee is separated from score quality (Remark 1). When references match, a user can plug the native committee into the same rule and keep the same guarantee.
- The relative-energy comparisons the reviewer cited (old Figures 22–23) are addressed in our response to W2. Error-based routing is no longer presented as a method for relative energies.
[W2]
We thank the reviewer for this criticism. It led to the main new contribution of the revision: a rule that controls the chemical observable directly.
1. Physical tolerances.
We now certify physical tolerances directly across all 16 pairs: total energy error above 1 kcal/mol (V1), per-atom energy error above 0.05 kcal/mol/atom (V2), and maximum force error above 0.10 eV/Å (V3). At 1% and 5% violation budgets our score needs 98.75–98.82% and 94.41–94.52% DFT, and even the oracle needs 96.02–96.44% and 91.18–91.56% (Table 17, Figure 11a). We show that strict per-geometry accuracy is expensive for every score, even for the oracle, and this is stated in Section 4.1.
2. Why error-based routing fails for relative energies.
Replacing only some energies with DFT breaks error cancellation and creates new violations in 0.8–5.9% of previously accurate conformer pairs (Figure 16, Appendix F.2.3). Absolute-error scores also predict selection failure poorly, because they mostly track the error shared by all conformers of a molecule, and that error cancels.
| MLIP | MLIP energy margin | Our absolute-error score |
|---|---|---|
| UMA-S | 0.868 | 0.564 |
| AIMNet2 | 0.854 | 0.540 |
| ANI-2x | 0.799 | 0.552 |
AUROC for predicting conformer-selection failure (0.5 is random); 8,000 held-out DFT molecules (from Table 26).
3. A new rule that controls the decision (Section 2.4).
We use the same fundamental CRC principle, but calibrate an energy window around the MLIP minimum, evaluate the candidates inside it with DFT, and return their DFT minimum. Widening the window only adds candidates, so the failure loss always decreases (Lemma 1). The guarantee applies directly, without the conservative monotone envelope used in the old Appendix K.
At a 1% failure budget, the window needs 4.6%, 12.6% and 27.2% DFT for UMA-S, AIMNet2 and ANI-2x, respectively. Meanwhile, calibrated top- needs 30.4–42.9%, and error-based routing needs 63.1–86.7% (Table 2, Figure 2). The observed failure rates are 0.96%, 1.03% and 0.59%. All policies, tolerances and budgets were fixed in a dated protocol before any held-out energy was read (Appendix G.1).
4. Reported limitations.
- Cost grows with the error spread within a molecule (Figure 18c), and calibrated top- is cheaper on dense SPICE snapshots (Table 32).
- The guarantee is an average over molecules, so molecules with more than 45 atoms tend to fail more often (2.15% for UMA-S, Table 34).
- The rule controls the chosen conformer, not every pairwise energy. Controlling all pairs at a 5% budget still needs 73–94% DFT (Table 27).
In summary, we identified why error-based routing misses the observables, built a rule that targets the observable directly, and validated it on a pre-registered held-out set.
[W3/Q3]
We thank the reviewer for pointing this out. We also realized that the supervised committee was handicapped and that all excluded variants should be reported. We fixed the committee, restored all excluded results, and selected the strongest committee as the comparison for the entire paper.
1. Strengthening the supervised committee.
We retrained the supervised committee with the same fitting data, three seeds, architecture, -plus-ranking objective and training budget as our predictor. We also added logarithmic input preprocessing. On QCML-6950-9, single MLIP predictions reach energy errors of about eV, which collapsed the original network to a nearly constant score.
| Supervised committee variant | DFT at | |
|---|---|---|
| Raw inputs, 1 seed (original) | 86.90% | |
| inputs | 0.936 | 43.26% |
| Median-scaled inputs | 0.937 | 43.26% |
| Fair: log inputs + our training setup | 0.978 | 42.31% |
| Ours | 0.820 | 46.87% |
QCML-6950-9 MACE-OFF23(M) (from Table 14).
On this pair every log-input variant beats our predictor, and we report that.
2. All variants over 16 pairs (including the results).
| Supervised committee variant | Mean (16 pairs) |
|---|---|
| Raw inputs, 1 seed (original) | 0.520 |
| Log inputs, 1 seed | 0.633 |
| Fair: log inputs + our training setup | 0.785 |
| Ours | 0.822 |
From Figure 9 and Table 12. Per-pair values of the log-input variant are in Table 8.
The fair committee is now used as the main comparison. Against it, our predictor needs nearly the same DFT: 57.54% vs. 57.44% (Table 1). We therefore no longer claim an advantage over the committee, as the two are statistically tied. However, ours uses one MLIP instead of three (about 136 fewer CPU core-hours per pair, Appendix E.7). The native MACE-OFF23 S/M/L committee is also reported with strengthened training (Table 15).
3. -learning baseline.
Correcting the MLIP on the fitting set and then routing lowers DFT by 5.00–6.68 points in mismatched pairs, and does not help on the matched control (Table 18). Details are in our response to W1/Q2.
4. Transition1x was restored.
We added the Transition1x experiments: 19,930 configurations from 1,993 reactions (Table 16, Appendix E.4).
| MLIP | Ours | Fair sup. committee | Native ensemble |
|---|---|---|---|
| ANI-2x | 0.867 / 42.13 | 0.669 / 51.87 | 0.911 / 40.42 |
| AIMNet2 | 0.864 / 63.52 | 0.624 / 66.93 | – |
| MACE-OFF23(M) | 0.917 / 51.33 | 0.894 / 51.61 | – |
| AceFF-2 | 0.821 / 59.59 | 0.636 / 63.47 | – |
/ DFT (%) at (from Table 16).
Our predictor beats the supervised committee for all four MLIPs. The native ANI-2x ensemble is better than ours, and we state this in Section 5. These configurations come from reaction paths but are not Hessian-confirmed transition states, and some large force errors remain among the kept configurations. We therefore make no claim about transition-state or barrier accuracy.
5. Additional scores.
Beyond the requested baselines, we added nine error scores, including gradient, rigidity, latent and PROBE-style scores (Table 12; please see our response to Q1). Two scores that use MLIP internals rank slightly better than ours, and we report them as better choices when internals are available.
6. Complete reporting.
The appendix now contains the result tables behind every reported number, including all baseline variants and the restored experiments.
[W1/Q2] We appreciate the reviewer's concern, as it helped us rethink the entire scope and core principles of the paper. The reviewer is right on the main point, and we therefore changed the experiments and the claims accordingly.
1. What the error contains.
In all eight original pairs, the MLIP and the dataset used different DFT references, so the measured error mixed MLIP approximation error with differences between electronic-structure methods. Our composition-linear alignment removes per-element offsets but not geometry-dependent differences between methods. Table 6 now lists the training and evaluation reference of every pair.
2. Why is the mismatched setting still useful, and what is its new scope?
Practitioners often have a fixed reference (for example, to stay consistent with a specific dataset like ∇2DFT) while the best available pretrained MLIP was trained elsewhere. The question is when that MLIP can stand in for their reference. We agree this is not a test of uncertainty relative to the MLIP's own training reference.
The reviewer also noted that every MLIP was trained at a higher level than the data it had to match. The 16 pairs now include the reverse case: on OMol25-core (ωB97M-V/def2-TZVPD), AIMNet2 and ANI-2x were trained at lower or different levels (Table 6).
3. Matched references.
Following Q2, we evaluated OMol25 with UMA-S at the matched reference. We added strengthened native committees wherever a native ensemble exists, and we added Transition1x.
DFT calls (% of pool) at α=0.30 (from Tables 15 and 16).
When the references match and a native ensemble exists, the native committee is better. This is clearly stated in the main text (Table 1, Section 4.1, Section 5). On OMol25 a latent-mean predictor is better still (37.91%, Table 15). Our score still closes most of the random-to-oracle gap in these settings (G=0.66–0.83), so it remains useful when no native ensemble exists. On MACE-OFF23 it also beats the plain native disagreement score (G 0.763 vs. 0.730). We state clearly throughout the manuscript when our structural predictor outperforms others, and the specific cases where committees or latent predictors outperform ours.
4. Size extrapolation (further discussed in Q2).
On OMol25 with UMA-S, fitting on molecules with ≤150 atoms and testing on molecules with >200 atoms, our predictor reaches G=0.802, close to the native committee (G=0.819) (Appendix E.4). A matched nominal reference does not imply unseen data, since UMA-S was trained on OMol25 (this is stated in the manuscript).
5. Correcting the MLIP first (Δ-learning, further discussed in Q2).
We trained kernel and neural energy corrections on the fitting set, then routed the corrected model with separate CRC calibration (Table 18, Appendix E.6).
DFT calls (% of pool) at α=0.30 (from Table 18).
Correction lowers DFT by 5.00 (kernel) and 6.68 (neural) points on average in the mismatched pairs, and gives no benefit on the matched control. This supports the reviewer's view that much of the correctable error comes from the reference mismatch. Correction and routing can work together: a corrected MLIP is another frozen model, and CRC routes its remaining errors with the same guarantee.
6. Mismatched references.
On the 16 mismatched pairs, our predictor and the fairly trained supervised committee need nearly the same DFT: 57.54% and 57.44% (Table 1). The advantage of our method is that it needs only one MLIP instead of three, and no model internals.
7. Chemical-space shift.
We also moved fitted predictors between datasets with target recalibration. Transfer is poor (G=−0.26 to 0.50 over 18 transfers, against 0.73–0.95 in domain; Appendix E.5). The predictor must be refit per pair, while the guarantee holds whenever calibration uses the target population.