- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a new paradigm in MLIP inference by combining a standard MLIP model with a TFM. I think this work presents a really cool idea, but I just have questions before I can really see its impact towards practical applications.
Critical strengths
I think the main idea (TFMs for MLIP inference) is an interesting direction, and contains a lot of novelty.
Critical weaknesses
The inference benchmarks, which I consider a really critical point towards determining whether this paper presents something 'neat but impractical' or 'genuinely novel and useful', are not fleshed out. I didn't even find any more data about this in the appendix (which I did not read in detail). There is only Fig. 3a, which measures structures/second. It's not mentioned what structures are being used, how large the context is, why finetuning seems to make the direct model slower, etc. (see questions section).
Questions for authors
-
I have a lot of questions specifically about Fig. 3a. Right now, MLIP accuracy (on benchmarks, although maybe not in applications utility) is mostly saturated, and the community is moving towards pushing established models (including UMA) towards rapid inference and larger scales. In that context, anything which adds computational bulk to existing models should be critically examined. (1) Can you provide more information on the UMA-FT baselines? What data are they finetuned on (only the context used by the TFM?), and why does finetuning (UMA-FT) have a much slower inference speed than UMA? The text mentions "In contrast, MLIP-FT methods are substantially slower because they fine-tuning for new structures " but I don't see how this could affect direct inference. (2) What datasets are these benchmarked on, and what was the context size? Can the authors provide a more extensive evaluation that would help us understand concretely, what is the additional cost of using ICL during inference, including runtime and memory consumption, at different context sizes?
-
Another big point is scalability. I'm curious about scalability along two directions: (1) structure size, and (2) dataset size.
(1) Structure size: An interesting ablation to test, would be accuracy with molecule size. The simple sum over atomwise features to build the structure-level embedding obscures the number of features, while the goal of SGIE is to enhance these input representations. So the effectiveness might be seen in across different structure sizes. Can the authors decompose their accuracy comparisons into different # atoms size bins?
(2) Dataset size: TFM-ICL (based on UMA-S) underperforms UMA-S on the OMoL25 dataset, which I believe is the largest dataset considered. So the addition of ICL degrades the baseline performance of UMA-S in this case. Is there a correlation between training dataset size and ICL improvements? E.g., can I make the statement that ICL is best in low-data regimes?
- Moving beyond benchmark performance over fixed datasets, I'm wondering how this can be used for inference in real applications. What would define the context? Real data, typically produced continuously during MD trajectories, doesn't come with concretely-defined context. Would you have to execute ClusterMatch over massive existing training datasets to construct a context for every specific use-case? How much overhead does that create? And for increasing dataset sizes, the TFM needs to attend to massive number of rows - how does the inference overhead increase (eg, similar to my question 1 above)?