acceptodds
Under review as a conference paper at ICLR 2027

From Independent to Context-Aware Inference via Tabular Foundation Models for Energy Prediction on 3D Atomic Structures

Abstract

Machine Learning Interatomic Potentials (MLIPs) provide an efficient approach to approximating Density Functional Theory (DFT) energy calculations on 3D atomic structures. However, existing MLIPs largely follow an independent inference paradigm, leaving previously computed DFT observations unused and limiting potential performance gains from such observations. We introduce In-Context MLIPs (IC-MLIPs), a new inference paradigm that augments MLIPs with In-Context Learning (ICL) for context-aware energy prediction. Specifically, we attach Tabular Foundation Models (TFMs) as contextual predictors on top of pretrained MLIPs, conditioning predictions on atomic structures with previously computed energy targets without additional training or fine-tuning of the base MLIPs. Experiments across six large-scale datasets spanning diverse domains demonstrate substantial performance gains from ICL-based inference. To address the unique challenges arising when applying TFMs to IC-MLIPs, we further introduce Structure-Global Information Enrichment (SGIE) to enrich atom-wise MLIP representations with structure-global information, and Dynamic Context Construction to adaptively construct effective contexts during TFMs training. Together, these components yield further performance improvements.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Machine Learning Interatomic Potentials, In-Context Learning, Tabular Foundation Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sYEROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a new paradigm in MLIP inference by combining a standard MLIP model with a TFM. I think this work presents a really cool idea, but I just have questions before I can really see its impact towards practical applications.

Critical strengths

I think the main idea (TFMs for MLIP inference) is an interesting direction, and contains a lot of novelty.

Critical weaknesses

The inference benchmarks, which I consider a really critical point towards determining whether this paper presents something 'neat but impractical' or 'genuinely novel and useful', are not fleshed out. I didn't even find any more data about this in the appendix (which I did not read in detail). There is only Fig. 3a, which measures structures/second. It's not mentioned what structures are being used, how large the context is, why finetuning seems to make the direct model slower, etc. (see questions section).

Questions for authors
  1. I have a lot of questions specifically about Fig. 3a. Right now, MLIP accuracy (on benchmarks, although maybe not in applications utility) is mostly saturated, and the community is moving towards pushing established models (including UMA) towards rapid inference and larger scales. In that context, anything which adds computational bulk to existing models should be critically examined. (1) Can you provide more information on the UMA-FT baselines? What data are they finetuned on (only the context used by the TFM?), and why does finetuning (UMA-FT) have a much slower inference speed than UMA? The text mentions "In contrast, MLIP-FT methods are substantially slower because they fine-tuning for new structures " but I don't see how this could affect direct inference. (2) What datasets are these benchmarked on, and what was the context size? Can the authors provide a more extensive evaluation that would help us understand concretely, what is the additional cost of using ICL during inference, including runtime and memory consumption, at different context sizes?

  2. Another big point is scalability. I'm curious about scalability along two directions: (1) structure size, and (2) dataset size.

(1) Structure size: An interesting ablation to test, would be accuracy with molecule size. The simple sum over atomwise features to build the structure-level embedding obscures the number of features, while the goal of SGIE is to enhance these input representations. So the effectiveness might be seen in across different structure sizes. Can the authors decompose their accuracy comparisons into different # atoms size bins?

(2) Dataset size: TFM-ICL (based on UMA-S) underperforms UMA-S on the OMoL25 dataset, which I believe is the largest dataset considered. So the addition of ICL degrades the baseline performance of UMA-S in this case. Is there a correlation between training dataset size and ICL improvements? E.g., can I make the statement that ICL is best in low-data regimes?

  1. Moving beyond benchmark performance over fixed datasets, I'm wondering how this can be used for inference in real applications. What would define the context? Real data, typically produced continuously during MD trajectories, doesn't come with concretely-defined context. Would you have to execute ClusterMatch over massive existing training datasets to construct a context for every specific use-case? How much overhead does that create? And for increasing dataset sizes, the TFM needs to attend to massive number of rows - how does the inference overhead increase (eg, similar to my question 1 above)?
Reviewer_712BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes In-Context MLIPs (IC-MLIPs), a framework that integrates Tabular Foundation Models (TFMs) with pretrained MLIPs to leverage previously computed DFT observations for context-aware energy prediction without fine-tuning the base MLIPs. It further introduces Structure-Global Information Enrichment (SGIE) to enhance atomic representations and Dynamic Context Construction to improve robustness to mismatched contexts. Experiments on six large-scale datasets demonstrate substantial improvements in energy prediction accuracy.

Critical strengths
  • Introduces in-context learning to MLIPs, effectively leveraging existing DFT observations without modifying pretrained MLIPs.
  • Proposes SGIE and Dynamic Context Construction to address representation limitations and context distribution mismatches.
  • Demonstrates substantial accuracy improvements across six diverse datasets, supported by ablation studies and transfer evaluations.
Critical weaknesses
  • MLIP is used to predict both energy and force, but experiments focus exclusively on energy prediction, without evaluating force prediction or downstream applications such as geometry optimization and molecular dynamics.
  • The method requires previously computed DFT energies at inference time, potentially limiting applicability when labeled reference structures are unavailable or expensive to obtain.
  • Performance gains vary across datasets, for example, in table 3 UMA performs a lot better than the proposed method on OMol25
Questions for authors

please refer to the weakness part

Reviewer_9jFHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes In-Context MLIPs (IC-MLIPs), which predict the DFT energy of a query structure from context structures whose DFT energies are already known. A frozen MLIP encoder (UMA-s-1.1) produce embeddings for each structure, and a tabular foundation model (TFM; TabPFN-v2 or LimiX) predicts the query energies from the embeddings and energies of the context structures. TFM-ICL uses the pretrained TFM without further training, optionally with global composition, distance, and angle descriptors appended to the embedding (SGIE). TFM-CPT further trains the TFM on about 7.4M structures, with a self-supervised module that predicts these descriptors and a context sampling schedule (ClusterDrift) that gradually shifts the cluster composition of the context away from that of the queries. The authors report lower energy errors than direct UMA inference on ODAC25 and OC25, and lower errors than fine-tuning the UMA energy head on the same context structures on most datasets.

Critical strengths

Reusing DFT labels that are already available at inference time is a practical setting, and TFM-ICL requires no parameter updates of either the MLIP or the TFM. Also the evaluation covers six recent large datasets.

Critical weaknesses

Novelty and benefit of in-context prediction. Predicting DFT energies from frozen MLIP features with a model conditioned on labeled structures has been done with kernel ridge regression [1] and Gaussian process corrections [2], and Li et al. [3] apply TabPFN in context to learned structure embeddings of materials. The TFMs are also existing models, so the contribution lies in combining these components at a larger scale. However, the method is not compared with simple alternatives that use the same context labels, such as a per-element linear correction of the UMA energy or kernel ridge regression on the same embeddings, so it is unclear what in-context learning adds. In addition, only energies are predicted. Forces and stresses are neither derived nor evaluated, so the model is not shown to be usable for relaxation or MD, which limits its practical use.

Comparison with fine-tuned MLIPs. The authors present TFM-CPT as an effective alternative to conventional MLIP fine-tuning (Section 5.2), but the only fine-tuning baseline updates the energy head of a frozen MLIP on the 1,000–5,000 context structures of each window (Appendix C.2). Fine-tuning in practice usually updates the backbone as well, and this head-only baseline is insufficient, since UMA-s-1.1-FT is worse than direct UMA inference on OMol25 and OMC25(from Tables 2 and 3). The training data are also not matched on ODAC25, where TFM-CPT is trained on subsets that are not in the training data of UMA-s-1.1 (e.g., the GCMC configurations) and shows its largest gains. To support the claim, TFM-CPT should be compared with an MLIP whose backbone is fine-tuned on the same training data.

Errors relative to existing models. On OMol25 and OC25, models trained at the same level of theory are more accurate than every TFM variant. On OMol25, direct UMA-s-1.1 and UMA-s-1.2 have lower errors than all TFM results. On OC25, models trained on OC25 reach total-energy MAEs of about 0.1 eV [4], whereas the lowest TFM error reported in the paper is 1.68 eV (LimiX-2, Table 7), and UMA-s-1.2, which includes OC25 in its training data [5], is listed in Table 2 without an OC25 result. OC25 is also computed at a level of theory (RPBE-D3) that is not in the training data of UMA-s-1.1, and the OC25 paper does not compare zero-shot UMA energies for this reason [4]. Part of the 34.64 eV error of UMA-s-1.1 is therefore likely a systematic offset that the context labels can absorb, so the improvement in this transfer setting may come largely from correcting this offset rather than from generalization to a new domain. The TFMs report lower errors than existing models only on OMC25 and ODAC25, where many structures share the same system, such as frames of one relaxation trajectory (see the first question). On ODAC25, eSEN-sm-ODAC25 has an error of 1.105 eV on MOF+Ads in Table 2, whereas the ODAC25 paper reports 0.195–0.268 eV for eSEN models on its filtered MOF+Ads test set [6]. The two sets differ, but this gap suggests a possible inconsistency in the energy references used for the baselines and should be explained.

Novelty and evidence for SGIE and ClusterDrift. Both components are presented as contributions, but they build on existing methods. The SGIE descriptors are global one-, two-, and three-body distributions of the MBTR type [7], and the claim that sum-pooled MLIP features lack this information (Section 4.1) is not well supported. ClusterDrift combines cluster-based context sampling, as used for TFMs in [8], with a loss-driven schedule that gradually mismatches the context and the queries. Also the experimental support for both components is also weak. In Table 4, each component alone increases the LimiX-CPT error on most datasets, so the improvement depends on combining them, and no variance is reported. Finally, LimiX-2 used in context without continued pretraining (Table 7) outperforms every TFM-CPT model in Table 3 on five of the six datasets, which suggests that the choice of TFM matters more than these components.

Questions for authors
  1. The context and query structures are random subsets of the same 10,000-structure window (Appendix C.2), and these datasets contain many frames of the same system, such as relaxation trajectories with over 200 frames on average in ODAC25 [6]. What fraction of the queries have another structure from the same system or trajectory in their context, and how do the errors change when such structures are excluded? And, if predicting energies along a trajectory from a few labeled frames is an intended application [9], could you evaluate that setting directly?

  2. How does the method compare with a per-element linear correction of the UMA energy and with kernel ridge regression on the same embeddings, both fitted on the same context?

  3. Appendix C.2 describes MLIP-FT as fine-tuning the energy head with a frozen backbone. What learning rate, number of updates were used, and how was UMA+MLP (Figure 1b) trained? Section 5.2 attributes the degradation of UMA-s-1.1-FT on OMol25 to the limited context subset. Does this also explain the degradation on OMC25? How does TFM-CPT compare with UMA-s-1.1 fine-tuned, including the backbone, on the same training data, in particular on ODAC25?

  4. Figure 3a reports throughput comparable to direct UMA inference, but the measurement setup is not described. Which hardware, batch size, and context size were used, and which steps are included in the timing? How do the time, memory, and FLOPs of the TFM forward grow with the context size? For MLIP-FT, please report the fine-tuning time separately from the prediction time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.