- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies molecular and crystal property predictions using a self-supervised pretraining technique. In this work the authors proposed a JEPA-like architecture along with a JEPA-style loss function that operates on both the atom and substrate levels of 3D molecules and crystal systems. The author used the JEPA-style pretraining framework to pretrain equivariant GNNs using large untagged crystal and molecular datasets. After pretraining, they fine-tune the encoder for small property-tagged downstream tasks datasets for molecules and crystals.
Critical strengths
- This work proposed JEPA-style architecture to pretrain 3D equivarient GNN using large untagged molecules and crystal datasets and then fine-tune the encoder for downstream property prediction tasks using small tagged datasets.
- Experimental results show that the JEPA-style pretraining method outperforms both the supervised model and other self-supervised pretraining methods in 11 out of 12 properties on QM9, as well as surpassing the best supervised model in 6 out of 12 properties.
Critical weaknesses
- The main weakness of this work is that this work is not motivated correctly. Why would we use a new pretraining method when, as in the literature, there are many existing pretraining methods? Also, this work uses JEPA-style architecture, which is already well established; they used this algorithm and just ran it on molecules and crystals. So their method is novel with respect to application, not with respect to new pretraining technique.
- For crystal property prediction, their method does not outperform the SOTA pretraining property prediction method, which is present in Table 3. Also for formation energy and bandgap, their method cannot outperform CoGN. This is the main concern of the generalizability of their method.
Questions for authors
- Do the authors merge both the material and crystal datasets to pretrain 3D equivarient GNN, or do they pretrain individually? If merged pretraining, what will the performance?
- In Figure 3 for crystals, I observe that UMAP does not form meaningful clusters. Why is this kind of things is observed?
- The main motivation of pretraining and finetuning is used for enhance the performance of encoder for fewshot experiments. why this kind of experiment not is performed in this paper?
7 comments
- For crystal property prediction, their method does not outperform the SOTA pretraining property prediction method, which is present in Table 3. Also for formation energy and bandgap, their method cannot outperform CoGN. This is the main concern of the generalizability of their method.
It is correct that Atom-JEPA does not achieve the best overall performance on Matbench. We deliberately evaluate the method across three diverse domains—ADMET prediction, quantum-chemical property prediction, and crystalline materials—to establish both its strengths and its limitations. Atom-JEPA achieves state-of-the-art performance on several ADMET endpoints and QM9 properties, while its performance on crystals is competitive with the strongest existing methods but not the best overall. We view this breadth of evaluation as a strength: it provides a more complete assessment of the method’s transferability, including settings where it does not outperform state-of-the-art results.
For crystalline systems, the experiments nevertheless demonstrate that Atom-JEPA learns useful, transferable representations. The comparison with the same encoder trained from scratch directly measures the benefit of pretraining while holding the architecture fixed. As shown in Table 5 "Effect of Atom-JEPA pretraining" in the submitted paper, Atom-JEPA improves all eight tasks and all 40 task–fold combinations, reducing regression errors by 10.1–14.5%. These results establish a consistent benefit from pretraining in this domain.
The comparisons with other methods on Matbench also warrant several qualifications:
- Atom-JEPA achieves the lowest error on 6/8 tasks among compared pretrained models that use at most 4M structures, despite using only 1.7M unlabeled structures.
- Backbone choice materially affects performance: Atom-JEPA outperforms TriForces with MACE on 7/8 tasks, whereas TriForces with eSEN performs better on most tasks.
- coGN achieves lower errors on MP Gap and MP E Form, but these tasks provide approximately 106K and 133K labeled examples for learning from scratch. Atom-JEPA outperforms all compared scratch-trained models on the four smallest tasks, while still benefiting substantially from pretraining on both large tasks.
Taken together, the experiments support the transferability of Atom-JEPA across diverse molecular and crystalline properties, while showing that the magnitude of its advantage varies by domain. We hypothesize that scaling pretraining beyond the current 1.7M crystal structures could yield further gains, but leave this for future investigation.
Questions for Authors
- Do the authors merge both the material and crystal datasets to pretrain 3D equivarient GNN, or do they pretrain individually? If merged pretraining, what will the performance?
We thank the reviewer for raising this interesting question and consider joint pretraining on molecules and crystals an exciting direction for future work. We designed Atom-JEPA to support both domains using the same input types (atom types and 3D structure), equivariant encoder–predictor formulation, and prediction objectives. In particular, Atom-JEPA does not require explicit 2D bond information, allowing the same formulation to apply to both molecules and crystals, in contrast to previous approaches tailored specifically to either domain.
In the current experiments, we pretrained separate models for molecules and crystals; we did not merge the two datasets. As stated in Section 4 and detailed in Appendix G, the molecular models are pretrained on approximately 19M molecules from Uni-Mol, while the crystal model is pretrained on approximately 1.7M structures from Alexandria. The corresponding encoders are then fine-tuned on downstream tasks within their respective domains.
The strong molecular results and consistent improvements from pretraining across all evaluated crystal tasks demonstrate that the method learns useful representations in both domains. This provides motivation for investigating whether a shared encoder trained jointly on molecular and crystalline structures could benefit from information across domains. Our formulation provides a basis for this extension, but we have not evaluated joint pretraining and therefore cannot yet report whether it would improve performance over the separately pretrained models.
- In Figure 3 for crystals, I observe that UMAP does not form meaningful clusters. Why is this kind of things is observed?
We respectfully disagree that the crystal representations show no meaningful clustering. In our interpretation, Figure 3 shows the following:
- The individual clusters in the figure mostly contain the same crystal type
- Each crystal type do have several separated clusters, but we do not see this as a negative result; rather, we hypothesize that even within crystal systems, there is additional subclustering occuring according to other defining crystal properties
- The seperation is not perfect, but it is clear that the representation separates different crystal systems.
As noted in the paper, we treat embedding visualizations as qualitative indications rather than conclusive evidence, and acknowledge that their visual interpretation can differ. An empirical result is the quantitative neighborhood analysis in Table 9, which shows that embedding neighbors share space groups substantially more often than the comparison baselines. This further supports our interpretation that the models learn higher order features of the molecular systems.
That said, Atom-JEPA is trained to improve high-level downstream tasks, not classify crystal systems or space groups, so complete separation by these labels is not a requirement nor necessarily the goal for useful representations.
Critical Weaknesses:
Respectfully, we strongly disagree with both the suggestion that the existence of other pretraining methods removes the motivation for developing a new one, and the characterization of Atom-JEPA as an established algorithm that we simply apply to molecules and crystals.
Motivation for developing a new pretraining method
As stated in our abstract and introduction, our motivation is that, although large-scale self-supervised pretraining has substantially advanced generalization in language and vision, it has not yet achieved comparable downstream generalization for atomistic systems. A central open question is how to effectively exploit large unlabeled datasets to improve prediction on downstream tasks with limited labeled data, where even relatively simple fingerprint-based approaches often remain competitive. The existence of several pretraining methods reflects ongoing efforts to address this problem; it does not mean that the problem has been resolved.
To address the reviewer’s concern, we'd like to explain this motivation further. Supervised pretraining on quantum-chemical properties provides useful learning signals, but generating these labels is computationally expensive, and the information they emphasize may not be equally relevant to all downstream tasks. Self-supervised pretraining offers a route to exploiting structural data without this labeling cost. Existing approaches investigate different learning signals, including denoising, masking, and contrastive learning. Our work investigates whether predicting representations directly in latent space from partial 3D structural views can provide an effective alternative for learning transferable features.
This motivates our specific formulation: using atom types and 3D geometry alone, we develop atom-level and substructure-level prediction tasks within an equivariant architecture that supports both molecules and periodic crystals. The atom-level task requires spatially specific predictions of individual target representations, while the substructure task predicts an aggregate representation. Our hypothesis is that these learning signals can encourage representations that capture information useful across downstream properties, without requiring property labels, molecular descriptors, or explicit bond annotations.
The experimental results support pursuing this approach. Atom-JEPA achieves the strongest aggregate ranking on the evaluated ADMET benchmarks and state-of-the-art performance on 6/12 QM9 targets, while being best or within 5% of the best on 10/12 targets overall. On Matbench, pretraining significantly improves the same encoder on all eight tasks and all 40 task–fold combinations. Developing a new pretraining method is therefore motivated both by an unresolved learning problem and by the demonstrated benefits of the proposed approach.
Methodological novelty
Regarding methodological novelty, we'd first like to clarify that JEPA is a general framework for learning by predicting target representations from related context rather than reconstructing raw inputs (LeCun, 2022). It is not a fully specified algorithm that can simply be applied unchanged across domains, as the reviewer suggests. Developing a concrete method within this framework requires designing both the learning task (including the inputs, sampling and partitioning, and prediction objectives) and the system that learns from it (including the encoders and predictor). The image-based I-JEPA (Assran et al., 2023) and video-based V-JEPA (Bardes et al., 2024) are examples of such methodological development within the JEPA framework. Their performance has recently motivated significant research efforts into whether and how this architecture can support learning in other domains. We view Atom-JEPA as contributing to this family of research through our specific design choices for atomistic systems, validated through large-scale pretraining and downstream evaluation.
Atom-JEPA develops a distinct formulation using only atom types and 3D structure as inputs, with 3D equivariant GNNs as both encoder and predictor backbones. Within this formulation, we design atom-level and substructure-level prediction objectives, evaluate their effectiveness independently and jointly, and conduct large-scale pretraining. None of the prior JEPA methods (discussed in Section 2) follows this approach. For specific methodological contributions we refer to our response to review 1 (W1), and we would be happy to elaborate further.