acceptodds
Under review as a conference paper at ICLR 2027

Atom-JEPA: Joint-Embedding Predictive Architecture for 3D Atomistic Systems

Abstract

Large-scale self-supervised pretraining has reshaped modern machine learning, substantially advancing the ability of language and vision models to generalize across downstream tasks. While deep learning has driven considerable progress in modeling atomistic systems in recent years, self-supervised pretraining in this domain has not yet achieved comparable downstream generalization. To address this, we introduce Atom-JEPA, a self-supervised pretraining framework that learns latent representations from unlabeled 3D structures through complementary atom-level and substructure-level objectives inspired by joint-embedding predictive architectures. We pretrain Atom-JEPA on large-scale molecular and crystalline datasets and evaluate its transfer performance by fine-tuning on a diverse set of downstream property prediction tasks. Atom-JEPA achieves state-of-the-art performance on molecular ADMET and quantum-chemical property prediction tasks, and is highly competitive in predicting the physical properties of crystalline materials. These results demonstrate the potential of latent-space predictive pretraining to support broad downstream generalization from structural data alone.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Foundation Models, Self-supervised learning, Atomic Systems, Molecular Modelling, AI4Science, ADMET, ADME

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3H6uOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies molecular and crystal property predictions using a self-supervised pretraining technique. In this work the authors proposed a JEPA-like architecture along with a JEPA-style loss function that operates on both the atom and substrate levels of 3D molecules and crystal systems. The author used the JEPA-style pretraining framework to pretrain equivariant GNNs using large untagged crystal and molecular datasets. After pretraining, they fine-tune the encoder for small property-tagged downstream tasks datasets for molecules and crystals.

Critical strengths
  • This work proposed JEPA-style architecture to pretrain 3D equivarient GNN using large untagged molecules and crystal datasets and then fine-tune the encoder for downstream property prediction tasks using small tagged datasets.
  • Experimental results show that the JEPA-style pretraining method outperforms both the supervised model and other self-supervised pretraining methods in 11 out of 12 properties on QM9, as well as surpassing the best supervised model in 6 out of 12 properties.
Critical weaknesses
  • The main weakness of this work is that this work is not motivated correctly. Why would we use a new pretraining method when, as in the literature, there are many existing pretraining methods? Also, this work uses JEPA-style architecture, which is already well established; they used this algorithm and just ran it on molecules and crystals. So their method is novel with respect to application, not with respect to new pretraining technique.
  • For crystal property prediction, their method does not outperform the SOTA pretraining property prediction method, which is present in Table 3. Also for formation energy and bandgap, their method cannot outperform CoGN. This is the main concern of the generalizability of their method.
Questions for authors
  • Do the authors merge both the material and crystal datasets to pretrain 3D equivarient GNN, or do they pretrain individually? If merged pretraining, what will the performance?
  • In Figure 3 for crystals, I observe that UMAP does not form meaningful clusters. Why is this kind of things is observed?
  • The main motivation of pretraining and finetuning is used for enhance the performance of encoder for fewshot experiments. why this kind of experiment not is performed in this paper?
7 comments

Critical Weaknesses:

  1. The main weakness of this work is that this work is not motivated correctly. Why would we use a new pretraining method when, as in the literature, there are many existing pretraining methods? Also, this work uses JEPA-style architecture, which is already well established; they used this algorithm and just ran it on molecules and crystals. So their method is novel with respect to application, not with respect to new pretraining technique.

Respectfully, we strongly disagree with both the suggestion that the existence of other pretraining methods removes the motivation for developing a new one, and the characterization of Atom-JEPA as an established algorithm that we simply apply to molecules and crystals.

Motivation for developing a new pretraining method

As stated in our abstract and introduction, our motivation is that, although large-scale self-supervised pretraining has substantially advanced generalization in language and vision, it has not yet achieved comparable downstream generalization for atomistic systems. A central open question is how to effectively exploit large unlabeled datasets to improve prediction on downstream tasks with limited labeled data, where even relatively simple fingerprint-based approaches often remain competitive. The existence of several pretraining methods reflects ongoing efforts to address this problem; it does not mean that the problem has been resolved.

To address the reviewer’s concern, we'd like to explain this motivation further. Supervised pretraining on quantum-chemical properties provides useful learning signals, but generating these labels is computationally expensive, and the information they emphasize may not be equally relevant to all downstream tasks. Self-supervised pretraining offers a route to exploiting structural data without this labeling cost. Existing approaches investigate different learning signals, including denoising, masking, and contrastive learning. Our work investigates whether predicting representations directly in latent space from partial 3D structural views can provide an effective alternative for learning transferable features.

This motivates our specific formulation: using atom types and 3D geometry alone, we develop atom-level and substructure-level prediction tasks within an equivariant architecture that supports both molecules and periodic crystals. The atom-level task requires spatially specific predictions of individual target representations, while the substructure task predicts an aggregate representation. Our hypothesis is that these learning signals can encourage representations that capture information useful across downstream properties, without requiring property labels, molecular descriptors, or explicit bond annotations.

The experimental results support pursuing this approach. Atom-JEPA achieves the strongest aggregate ranking on the evaluated ADMET benchmarks and state-of-the-art performance on 6/12 QM9 targets, while being best or within 5% of the best on 10/12 targets overall. On Matbench, pretraining significantly improves the same encoder on all eight tasks and all 40 task–fold combinations. Developing a new pretraining method is therefore motivated both by an unresolved learning problem and by the demonstrated benefits of the proposed approach.

Methodological novelty

Regarding methodological novelty, we'd first like to clarify that JEPA is a general framework for learning by predicting target representations from related context rather than reconstructing raw inputs (LeCun, 2022). It is not a fully specified algorithm that can simply be applied unchanged across domains, as the reviewer suggests. Developing a concrete method within this framework requires designing both the learning task (including the inputs, sampling and partitioning, and prediction objectives) and the system that learns from it (including the encoders and predictor). The image-based I-JEPA (Assran et al., 2023) and video-based V-JEPA (Bardes et al., 2024) are examples of such methodological development within the JEPA framework. Their performance has recently motivated significant research efforts into whether and how this architecture can support learning in other domains. We view Atom-JEPA as contributing to this family of research through our specific design choices for atomistic systems, validated through large-scale pretraining and downstream evaluation.

Atom-JEPA develops a distinct formulation using only atom types and 3D structure as inputs, with 3D equivariant GNNs as both encoder and predictor backbones. Within this formulation, we design atom-level and substructure-level prediction objectives, evaluate their effectiveness independently and jointly, and conduct large-scale pretraining. None of the prior JEPA methods (discussed in Section 2) follows this approach. For specific methodological contributions we refer to our response to review 1 (W1), and we would be happy to elaborate further.

  1. For crystal property prediction, their method does not outperform the SOTA pretraining property prediction method, which is present in Table 3. Also for formation energy and bandgap, their method cannot outperform CoGN. This is the main concern of the generalizability of their method.

It is correct that Atom-JEPA does not achieve the best overall performance on Matbench. We deliberately evaluate the method across three diverse domains—ADMET prediction, quantum-chemical property prediction, and crystalline materials—to establish both its strengths and its limitations. Atom-JEPA achieves state-of-the-art performance on several ADMET endpoints and QM9 properties, while its performance on crystals is competitive with the strongest existing methods but not the best overall. We view this breadth of evaluation as a strength: it provides a more complete assessment of the method’s transferability, including settings where it does not outperform state-of-the-art results.

For crystalline systems, the experiments nevertheless demonstrate that Atom-JEPA learns useful, transferable representations. The comparison with the same encoder trained from scratch directly measures the benefit of pretraining while holding the architecture fixed. As shown in Table 5 "Effect of Atom-JEPA pretraining" in the submitted paper, Atom-JEPA improves all eight tasks and all 40 task–fold combinations, reducing regression errors by 10.1–14.5%. These results establish a consistent benefit from pretraining in this domain.

The comparisons with other methods on Matbench also warrant several qualifications:

  • Atom-JEPA achieves the lowest error on 6/8 tasks among compared pretrained models that use at most 4M structures, despite using only 1.7M unlabeled structures.
  • Backbone choice materially affects performance: Atom-JEPA outperforms TriForces with MACE on 7/8 tasks, whereas TriForces with eSEN performs better on most tasks.
  • coGN achieves lower errors on MP Gap and MP E Form, but these tasks provide approximately 106K and 133K labeled examples for learning from scratch. Atom-JEPA outperforms all compared scratch-trained models on the four smallest tasks, while still benefiting substantially from pretraining on both large tasks.

Taken together, the experiments support the transferability of Atom-JEPA across diverse molecular and crystalline properties, while showing that the magnitude of its advantage varies by domain. We hypothesize that scaling pretraining beyond the current 1.7M crystal structures could yield further gains, but leave this for future investigation.

Questions for Authors

  1. Do the authors merge both the material and crystal datasets to pretrain 3D equivarient GNN, or do they pretrain individually? If merged pretraining, what will the performance?

We thank the reviewer for raising this interesting question and consider joint pretraining on molecules and crystals an exciting direction for future work. We designed Atom-JEPA to support both domains using the same input types (atom types and 3D structure), equivariant encoder–predictor formulation, and prediction objectives. In particular, Atom-JEPA does not require explicit 2D bond information, allowing the same formulation to apply to both molecules and crystals, in contrast to previous approaches tailored specifically to either domain.

In the current experiments, we pretrained separate models for molecules and crystals; we did not merge the two datasets. As stated in Section 4 and detailed in Appendix G, the molecular models are pretrained on approximately 19M molecules from Uni-Mol, while the crystal model is pretrained on approximately 1.7M structures from Alexandria. The corresponding encoders are then fine-tuned on downstream tasks within their respective domains.

The strong molecular results and consistent improvements from pretraining across all evaluated crystal tasks demonstrate that the method learns useful representations in both domains. This provides motivation for investigating whether a shared encoder trained jointly on molecular and crystalline structures could benefit from information across domains. Our formulation provides a basis for this extension, but we have not evaluated joint pretraining and therefore cannot yet report whether it would improve performance over the separately pretrained models.

  1. In Figure 3 for crystals, I observe that UMAP does not form meaningful clusters. Why is this kind of things is observed?

We respectfully disagree that the crystal representations show no meaningful clustering. In our interpretation, Figure 3 shows the following:

  • The individual clusters in the figure mostly contain the same crystal type
  • Each crystal type do have several separated clusters, but we do not see this as a negative result; rather, we hypothesize that even within crystal systems, there is additional subclustering occuring according to other defining crystal properties
  • The seperation is not perfect, but it is clear that the representation separates different crystal systems.

As noted in the paper, we treat embedding visualizations as qualitative indications rather than conclusive evidence, and acknowledge that their visual interpretation can differ. An empirical result is the quantitative neighborhood analysis in Table 9, which shows that embedding neighbors share space groups substantially more often than the comparison baselines. This further supports our interpretation that the models learn higher order features of the molecular systems.

That said, Atom-JEPA is trained to improve high-level downstream tasks, not classify crystal systems or space groups, so complete separation by these labels is not a requirement nor necessarily the goal for useful representations.

Reviewer_Kc3cOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Atom-JEPA is a self-supervised pretraining framework for molecular and crystal representations using atomic identities and 3D geometry. It learns to predict representations of masked atoms and substructures from partial structures, with targets provided by an EMA target encoder. The pretrained encoder is then fine-tuned for property prediction. Experiments show improvements over training from scratch and transfer to downstream datasets, although gains over strong existing methods are mixed.

Critical strengths

Critical Strengths

S1. The experiments demonstrate a tangible contribution from pretraining. Comparisons against randomly initialized versions of the same encoder show improvements across molecular ADMET and crystal property prediction tasks. These comparisons support the practical value of Atom-JEPA pretraining beyond the representational capacity of its backbone.

S2. Transfer is demonstrated in a setting with minimal pretraining overlap. Strong performance on the temporally split ExpansionRx benchmark, together with improvements over training from scratch, provides evidence that the learned representations transfer beyond molecules encountered during pretraining. This is an important complement to results on benchmarks with substantial structural overlap.

Critical weaknesses

Critical Weaknesses

W1. The central joint-objective design is not sufficiently validated. Combining atom-level and substructure-level prediction is a central methodological distinction, yet the ablations do not clearly establish its contribution to downstream fine-tuning performance. The relative benefit varies across pretraining datasets, the larger-scale comparison changes optimization settings, and the evaluation primarily relies on frozen-feature probes. Consequently, the evidence does not isolate the added value of the joint objective over atom-level prediction alone.

W2. The results do not sufficiently establish a practical advantage over existing methods. The improvements over the same backbone trained from scratch demonstrate the value of pretraining, but do not establish a consistent advantage over strong alternatives. Scratch-trained models remain competitive on QM9, while existing methods generally achieve better performance on Matbench. Although the ADMET results are encouraging, the paper does not evaluate whether the accuracy gains justify the computational requirements of large-scale pretraining and conformer-based inference. Consequently, the conditions under which Atom-JEPA offers a preferable accuracy–cost trade-off remain unclear.

Questions for authors

Q1(clarification). What evidence in the current experiments isolates the downstream contribution of each pretraining objective?

Q2(clarification). Given the mixed results against strong baselines, what practical advantage do the current experiments establish for Atom-JEPA?

11 comments

We thank the reviewer for their assessment and feedback. Below, we respond to the concerns raised and outline the additional experiments and clarifications included in the revised paper. We hope these additions help resolve the reviewer’s concerns and inform their overall assessment, and we welcome further discussion of any remaining questions.

Critical Weaknesses:

  1. The central joint-objective design is not sufficiently validated. Combining atom-level and substructure-level prediction is a central methodological distinction, yet the ablations do not clearly establish its contribution to downstream fine-tuning performance. The relative benefit varies across pretraining datasets, the larger-scale comparison changes optimization settings, and the evaluation primarily relies on frozen-feature probes. Consequently, the evidence does not isolate the added value of the joint objective over atom-level prediction alone.

We would like to emphasize, that our central metodological contribution is the specific design of the substructure and atom level objectives. In particular, the atom-level predictor, with its equivariant relative positional encoding, constitutes the most novel and technically involved part of our method. We recognize that our presentation may have given the impression that the idea of combining an atom-level and a substructure-level prediction into a joint objective is our central methodological distinction. We view the two objecives as contributions in their own right, and the purpose of the ablation study is to investigate their effectiveness independently and jointly. Our results show that both objectives are effective: the atom-level objective performs particularly well on its own, while substructure prediction provides complementary benefits in the full 19M Uni-Mol scale. A substantial benefit from joint training is therefore not necessary to establish Atom-JEPA’s contribution, and strong performance of the atom-level objective is a positive finding rather than a shortcoming in our view.

That said, at the full 19M Uni-Mol scale, the joint configuration does improve over atom-only on 10/12 QM9 property probes and performs better on average on Biogen ADME. The latter uses full fine-tuning, so the evidence extends beyond frozen-feature probes. However, the reviewer correctly notes that these pretraining runs differ in batch size and maximum learning rate. To accelerate atom-only pretraining using multiple GPU nodes, we increased the global batch size from 128 to 384 and the maximum learning rate from 1.4×10−41.4\times10^{-4} to 2.42×10−42.42\times10^{-4}, approximately following square-root scaling; all remaining hyperparameters are identical. We will make this qualification explicit when discussing the full-scale comparison.

In order to directly compare under identical optimization, we have now also expanded the ablation study by pretraining each objective configuration on a random 50K sample from Uni-Mol, using the exact same setup and hyperparameters, and fine-tuning on Biogen ADME and ExpansionRx using the same procedure as in the paper:

Table 1: Finetuning results on Biogen ADME and ExpansionRx for each objective independetly and joined, pretrained on 50K random sample from Uni-Mol.

ModelLowest mean, all targetsBest or indistinguishable: BiogenBest or indistinguishable: ExpansionRx
Joint7/156/68/9
Atom-only5/156/66/9
Substructure-only3/156/63/9

Here too, joint training performs best overall, followed by atom-only pretraining. These results use full fine-tuning rather than frozen-feature probes and provide additional evidence supporting its use in our main downstream evaluations. However, as clarified above, a substantial advantage over atom-level prediction alone is not necessary to establish Atom-JEPA’s contribution, since both objectives are contributions in their own right. We have revised the paper to include this new ablation study and clarify the purpose of the ablations and their conclusions.

  1. The results do not sufficiently establish a practical advantage over existing methods. The improvements over the same backbone trained from scratch demonstrate the value of pretraining, but do not establish a consistent advantage over strong alternatives. Scratch-trained models remain competitive on QM9, while existing methods generally achieve better performance on Matbench. Although the ADMET results are encouraging, the paper does not evaluate whether the accuracy gains justify the computational requirements of large-scale pretraining and conformer-based inference. Consequently, the conditions under which Atom-JEPA offers a preferable accuracy–cost trade-off remain unclear.

We respectfully disagree that a practical advantage of Atom-JEPA is not established by the results. The evaluation covers 5 benchmarks across three domains, with the following outcomes:

  • ADMET prediction: Atom-JEPA is best or statistically indistinguishable from the best on 31/40 endpoints, the highest total among the evaluated methods, including the unsupervised /semi-supervised pre-trained methods CKERMT, KERMT, and Mol-JEPA.
  • Quantum-chemical property prediction (QM9): Atom-JEPA is best or within 5% of the best result on 10/12 targets, and outperforms alternative self-supervised methods on 11/12 targets while tying on the remaining target
  • Materials-property prediction (Matbench): Atom-JEPA is competitive but does not achieve state-of-the-art performance. Nevertheless, pretraining improves the same encoder across all eight evaluated tasks and all 40 task–fold combinations.

Additionally, we now provide additional experimental evidence supporting Atom-JEPA’s practical advantages, included in the revised paper with additions highlighted in blue.

  • Using the same Equiformer V3 backbone, the same 50K-molecule subset of Uni-Mol, and 25 pretraining epochs, Atom-JEPA achieves lower mean downstream errors than FRAD (RN) (Feng et al., 2023) and Tri-Forces (Ramlaoui, 2026) on both Biogen ADME and ExpansionRx (Table 2 below), while requiring approximately one-third of their pretraining time per epoch.
  • On the QM9 scaffold split used by Uni-Mol2, Atom-JEPA also achieves lower errors than Uni-Mol and all reported Uni-Mol2 model sizes across all evaluated properties shown in Table 3.

Together with the improvements over training from scratch on Matbench, these results demonstrate that Atom-JEPA can outperform alternative SSL methods with matched backbones, pretraining data, and epoch counts, and benefit crystalline-property prediction even where it does not achieve state-of-the-art performance. We hypothesize that further scaling of crystal pretraining could yield additional gains.

Table 2: Controlled comparison with FRAD and Tri-Forces using full fine-tuning. Lower errors are better.

MethodBiogen ADMEExpansionRx
No pretraining.3761 ± .0313.3626 ± .0044
FRAD.3632 ± .0202.3563 ± .0051
Tri-Forces.3693 ± .0255.3541 ± .0021
Atom-JEPA (joint).3566 ± .0261.3403 ± .0033

Table 3: QM9 results on the Uni-Mol2 scaffold split. The evaluation metric is MAE (↓), reported as mean ± standard deviation over three random seeds. Results for all methods other than Atom-JEPA are taken from the Uni-Mol2 paper (Ji et al., 2024). Model sizes are given in parentheses beside model names. Bold denotes the best result in each column.

TaskHOMO/LUMO/GAPα\alphaCvC_vμ\muR2R^2
UnitsmeVa03a_0^3calmol K\frac{\mathrm{cal}}{\mathrm{mol\,K}}mDa02a_0^2
Grover_base215.0 ± 82.365 ± .3021.103 ± .339618 ± 2113.01 ± 4.206
Grover_large225.9 ± 162.240 ± .385.853 ± .186623 ± 685.85 ± 6.816
GEM182.3 ± 1.589 ± .0042.237 ± .0137444 ± 1.525.67 ± .743
Uni-Mol (48M)117.0 ± .5.363 ± .009.183 ± .002155 ± 1.54.805 ± .055
Uni-Mol2 (84M)103.4 ± 1.376 ± .027.178 ± .012105 ± .94.968 ± .235
Uni-Mol2 (164M)98.0 ± .3.325 ± .004.157 ± .01793 ± .64.935 ± .189
Uni-Mol2 (310M)98.0 ± .3.315 ± .003.143 ± .00292 ± 1.34.672 ± .245
Uni-Mol2 (570M)98.0 ± .5.315 ± .004.147 ± .000789 ± 1.54.523 ± .080
Uni-Mol2 (1.1B)95.2 ± .3.305 ± .003.144 ± .00289 ± .44.265 ± .067
Atom-JEPA (39M)68.0 ± .5.147 ± .002.063 ± .00128 ± .5.105 ± .003

Continuation of response to W2:

Cost. A potential use case for Atom-JEPA is prioritizing drug candidates for experimental ADMET testing given a small labelled dataset. Screening 100K molecules with ten conformers takes approximately 36 minutes, or 20 minutes when CPU conformer generation and GPU inference overlap. With a single conformer and overlapping computation, this falls to approximately two minutes, while retaining performance statistically indistinguishable from the best on 30/40 ADMET endpoints, compared with 31/40 using ten conformers. Pretraining is a one-time investment, and the checkpoints that we plan to release publicly allow users to obtain these benefits without repeating it. QM9 and Matbench already provide coordinates and therefore incur no conformer-generation cost. We thank the reviewer for suggesting that we make these costs explicit; the measurements are now included in the revised paper (Section E.1), with additions highlighted in blue.

Table 4: Compute requirements measured over 1,000 Biogen molecules using an AMD EPYC 9354 CPU and an L40S GPU, with screening times extrapolated to 100K molecules.

Conformer generation (CPU)Atom-JEPA prediction (GPU)
Throughput1,030 conformers/s837 conformers/s
100K molecules, one conformer each1.6 min2.0 min
100K molecules, ten conformers each16 min20 min
Reviewer_EgaFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Atom-JEPA, a self-supervised framework for learning transferable representations of 3D molecular and crystalline structures. Its main contribution is a JEPA-based pretraining strategy that combines atom-level and substructure-level latent prediction, using SE(3)-equivariant GNNs without requiring property labels. The method is pretrained on approximately 19M molecules and 1.7M crystal structures. Experiments on ADMET, QM9, and Matbench demonstrate competitive or state-of-the-art performance across various property prediction tasks. Ablation studies further validate the benefits of pretraining, investigate the contributions of different prediction objectives, and examine the effects of data scaling. 45563_Atom_JEPA_Joint_Embeddin

Critical strengths
  • Strong and comprehensive empirical validation. Atom-JEPA demonstrates competitive or state-of-the-art performance across diverse benchmarks, including ADMET, QM9, and Matbench. The extensive comparisons with established baselines provide convincing evidence of its effectiveness and transferability.
  • A promising self-supervised learning framework for 3D atomistic systems. The combination of atom-level and substructure-level latent prediction offers a meaningful extension of JEPA to 3D molecular and crystalline structures. Its ability to learn transferable representations without property labels is particularly valuable for applications where labeled data are expensive or scarce.
Critical weaknesses
  • Limited methodological novelty. Atom-JEPA primarily adapts the established JEPA framework to 3D atomistic systems using equivariant GNNs, with the main modification being the combination of atom-level and substructure-level prediction objectives. The paper offers limited new insights into why this design should produce better representations.

  • Insufficient evidence for the proposed joint objective. The ablation results do not consistently demonstrate the benefit of combining atom-level and substructure-level objectives. For example, atom-only pretraining outperforms the joint objective on 10/12 QM9 properties when pretrained on QM9. This raises questions about whether the proposed combination is essential to the reported improvements.

Questions for authors
  • To what extent do the performance gains arise from the JEPA objective rather than the EquiformerV3 backbone or pretraining scale? Are there controlled comparisons with alternative self-supervised objectives using the same backbone, data, and computational budget?
  • The ablation results show that combining atom-level and substructure-level objectives does not consistently outperform atom-only pretraining. What explains this inconsistency, and under what conditions does the joint objective provide a substantial advantage?
5 comments

We thank the reviewer for their careful review and constructive feedback. In the comments below, we address each point in turn and describe the corresponding revisions to the paper. We hope these responses and revisions address the reviewer’s concerns and will be considered in their overall assessment. We are happy to engage in further discussion if any questions or lack of clarity remains.

Critical Weaknesses:

  1. Limited methodological novelty. Atom-JEPA primarily adapts the established JEPA framework to 3D atomistic systems using equivariant GNNs, with the main modification being the combination of atom-level and substructure-level prediction objectives. The paper offers limited new insights into why this design should produce better representations.

We recognize that our presentation may have underemphasized the methodological novelty of Atom-JEPA. Our instantiation of the JEPA framework for 3D atomistic systems differs from prior methods in its prediction objectives, input representation, backbone architecture, and applicability to both molecules and crystals:

  • We formulate novel atom-level and substructure-level prediction objectives (Appendix A.4):
    • Atom-JEPA is the first JEPA method to predict individual atom representations from partial 3D structural views, enabled by our novel relative geometric positional encoding (Appendix A.4.3).
    • We agree that simply combining substructure- and atom-level objectives is not itself groundbreaking. Our contribution is their specific formulation and experiments demonstrating their effectiveness, both individually and jointly.
  • We use atom types and 3D geometry alone
    • Atom-JEPA is the first method to define its learning task solely from partial 3D structures. Importantly, it learns without labels, fingerprints, or any other external prediction target.
  • We use equivariant GNNs as both encoder and predictor backbones
    • Prior JEPA methods primarily rely on either 2D GNNs or Transformer architectures that combine embeddings derived from 2D graphs, 3D graphs, or other modalities such as molecular descriptors, or that process sequences of atom tokens. It is the equivariant GNN that allows us to include geometric positional encodings.
  • Provide a common formulation for both molecules and periodic crystals
    • Prior JEPA methods are formulated for either molecules or crystals, not both.

We discuss the differences from prior work in Section 2 and would be happy to elaborate further.

Our design enables self-supervised learning without expert-based features and yields the most novel learned features among the models tested (Appendix E.2), helping explain its improved performance over prior JEPA methods such as mol-JEPA. Beyond these design differences, we validate the approach through pretraining on 19 million molecular and 1.7 million crystal structures—a larger scale than any previous atomistic JEPA method, to our knowledge.

We’d also like to make the point that JEPA is a general framework for learning by predicting target representations from context (LeCun, 2022). Turning this framework into a concrete method requires designing both the learning task and the system that learns from it. The image-based I-JEPA (Assran et al., 2023) and video-based V-JEPA (Bardes et al., 2024) are examples of such methodological development. As mentioned, the limited number of existing JEPA methods for atomistic systems differ substantially in both what they predict and how. These differences illustrate that instantiating JEPA requires resolving methodological questions that the framework itself does not answer. We view Atom-JEPA as contributing to this line of research through a substantially new approach.

Regarding why this design produces better representations, we view empirical validation as the primary test and assess our design through downstream evaluation across a wide range of tasks. Nevertheless, we agree that the paper could benefit from clearer intuition behind the design choices, and thank the reviewer for highlighting this. We have elaborated on this intuition in the new subsection A.1, “Design Rationale and Relation to Prior Work,” highlighted in blue in the revised paper. For the prediction objectives, our hypothesis is that predicting an individual atom’s representation at a particular position relative to the context requires learning information about atomic environments that predicting a pooled substructure representation may not require.

  1. Insufficient evidence for the proposed joint objective. The ablation results do not consistently demonstrate the benefit of combining atom-level and substructure-level objectives. For example, atom-only pretraining outperforms the joint objective on 10/12 QM9 properties when pretrained on QM9. This raises questions about whether the proposed combination is essential to the reported improvements.

We would like to clarify that the purpose of the ablations is to investigate the contributions of objectives independently, rather than to establish that joint training is consistently superior. We recognize that our presentation may have given the impression of the latter. As clarified above, the atom-level objective is itself a central contribution, and we view its strong standalone performance as a positive result.

That said, our ablations identify the joint objective as the best-performing variant overall at the full 19M Uni-Mol pretraining scale. Additionally, we have now expanded the ablation study by pretraining each objective configuration on a random 50K sample from the Uni-Mol dataset and fine-tuning on Biogen ADME and ExpansionRx:

Table 1: Finetuning results on Biogen ADME and ExpansionRx for each objective independetly and joined, pretrained on 50K random sample from Uni-Mol.

ModelLowest mean, all targetsBest or indistinguishable: BiogenBest or indistinguishable: ExpansionRx
Joint7/156/68/9
Atom-only5/156/66/9
Substructure-only3/156/63/9

Here too, joint training performs best overall, followed by atom-only pretraining. This supports its use in our main downstream evaluations, but does not mean that we view joint training as the most essential component or as the paper’s main contribution. We thank the reviewer for prompting us to expand this analysis and clarify its interpretation. We have added this additional ablation study to Section 4.2, “Ablation Studies,” and clarified the purpose of the ablations and their conclusions, with the additions highlighted in blue.

Questions for Authors

  1. To what extent do the performance gains arise from the JEPA objective rather than the EquiformerV3 backbone or pretraining scale? Are there controlled comparisons with alternative self-supervised objectives using the same backbone, data, and computational budget?

We agree that distinguishing the contributions of the objective, backbone, and pretraining scale is important. That said, small-scale objective comparisons may not reflect large-scale performance, while fully controlled comparisons at scale are computationally prohibitive—a constraint commonly faced in large-scale pretraining research, where such comparisons are not standard practice. That said, we provide comparisons with training from scratch using the same backbone, with Uni-Mol2 at the same pretraining data scale, and with alternative objectives under controlled small-scale conditions below.

Controlled comparison with the same backbone trained from scratch

For this we’d like to refer to Table 5 “Effect of Atom-JEPA Pretraining” in the submitted paper, which compares Atom-JEPA pretraining with training the same EquiformerV3 backbone from scratch on identical downstream splits, showing substantial gains across multiple benchmarks.

Controlled comparison with the same pretraining data and scale

To assess the role of pretraining scale, we have now additionally evaluated Atom-JEPA on the QM9 scaffold split used by Uni-Mol2 (Ji et al., 2024). Although this comparison does not hold the encoder backbone fixed, Atom-JEPA and Uni-Mol use the exact same pretraining data and scale. Atom-JEPA achieves significantly lower mean errors than Uni-Mol and all reported Uni-Mol2 model sizes, despite Uni-Mol2 using a larger pretraining dataset and substantially larger models. We have added these results to the QM9 results section of the revised paper, highlighted in blue. We thank the reviewer for encouraging this additional comparison.

Table 2: QM9 results on the Uni-Mol2 scaffold split. The evaluation metric is MAE (↓), reported as mean ± standard deviation over three random seeds. Results for all methods other than Atom-JEPA are taken from the Uni-Mol2 paper (Ji et al., 2024). Model sizes are given in parentheses beside model names. Bold denotes the best result in each column.

TaskHOMO/LUMO/GAPα\alphaCvC_vμ\muR2R^2
UnitsmeVa03a_0^3calmol K\frac{\mathrm{cal}}{\mathrm{mol\,K}}mDa02a_0^2
Grover_base215.0 ± 82.365 ± .3021.103 ± .339618 ± 2113.01 ± 4.206
Grover_large225.9 ± 162.240 ± .385.853 ± .186623 ± 685.85 ± 6.816
GEM182.3 ± 1.589 ± .0042.237 ± .0137444 ± 1.525.67 ± .743
Uni-Mol (48M)117.0 ± .5.363 ± .009.183 ± .002155 ± 1.54.805 ± .055
Uni-Mol2 (84M)103.4 ± 1.376 ± .027.178 ± .012105 ± .94.968 ± .235
Uni-Mol2 (164M)98.0 ± .3.325 ± .004.157 ± .01793 ± .64.935 ± .189
Uni-Mol2 (310M)98.0 ± .3.315 ± .003.143 ± .00292 ± 1.34.672 ± .245
Uni-Mol2 (570M)98.0 ± .5.315 ± .004.147 ± .000789 ± 1.54.523 ± .080
Uni-Mol2 (1.1B)95.2 ± .3.305 ± .003.144 ± .00289 ± .44.265 ± .067
Atom-JEPA (39M)68.0 ± .5.147 ± .002.063 ± .00128 ± .5.105 ± .003

Controlled comparison to alternative self-supervised methods with the same backbone:

Finally, we have added comparisons with FRAD (RN) (Feng et al., 2023) and Tri-Forces (Ramlaoui, 2026) using the same EquiformerV3 backbone, the same random sample of 50K Uni-Mol pretraining examples, and the same number of training steps (25 epochs). We match training steps rather than wall-clock time to assess each method’s effectiveness per step without confounding the comparison with implementation efficiency. Atom-JEPA achieves the lowest mean error on both Biogen ADME and ExpansionRx, supporting the effectiveness of its objective with the backbone, pretraining data, and number of training steps held fixed. Atom-JEPA also offers a practical speed advantage: both FRAD and Tri-Forces are approximately 3× slower in our experiments

We have added this comparison to Section 4.2, “Ablation Studies,” of the revised paper, highlighted in blue. We thank the reviewer for encouraging this controlled comparison.

Table 3: Controlled comparison to FRAD and Tri-Forces.

MethodBiogen ADMEExpansionRx
No pretraining.3761 ± .0313.3626 ± .0044
FRAD.3632 ± .0202.3563 ± .0051
Tri-Forces.3693 ± .0255.3541 ± .0021
Atom-JEPA (joint).3566 ± .0261.3403 ± .0033

Comments on the paper

1 comment

Summary of Revisions and Discussion

We would like to thank the reviewers for their reviews and the subsequent constructive discussion. Their suggestions for additional experiments strengthened the empirical foundation of this work, and their questions helped us clarify the presentation and scope of our contributions. Below, we summarize the main points raised, how we addressed them, and where the corresponding changes can be found in the uploaded revised paper. Additions to the original submission are highlighted in blue.

1. Clarifying the methodological contribution

Reviewer EgaF asked for additional clarity on what distinguishes Atom-JEPA from prior JEPA methods and what motivated its design choices. In our responses, we clarified four distinguishing aspects:

  1. Learning from atom types and 3D geometry alone, without property labels, fingerprints, or 2D bond graphs.
  2. The specific formulation of atom-level and substructure-level latent prediction objectives, including prediction of individual atom representations from partial 3D structural views through relative geometric positional conditioning.
  3. An equivariant encoder–predictor architecture that couples geometric queries to equivariant context features.
  4. A common formulation applicable to both molecules and periodic crystals.

We emphasized that the contribution lies in the formulation and demonstrated effectiveness of the objectives, individually and jointly, rather than solely in combining two prediction objectives. We expanded the discussion of the design rationale in Appendix A.1. Reviewer Kc3c also requested clarification on the distinction between Atom-JEPA and Point-JEPA, along with an ablation study to assess the contribution of the relative equivariant positional conditioning in the atom-level objective. We added this clarification and the ablation study in Appendix H.

2. Controlled comparisons to alternative SSL objectives

Reviewer EgaF requested controlled comparisons with alternative SSL objectives.

  • We added experiments comparing Atom-JEPA with Frad (RN) [1] and TriForces [2] using the same backbone, pretraining data, and number of pretraining steps, followed by the same downstream fine-tuning protocol. Atom-JEPA achieves the lowest mean error on both Biogen and ExpansionRx (Section 4.2, Table 7).
  • We additionally evaluated Atom-JEPA on the Uni-Mol2 QM9 scaffold split, obtaining lower MAE across all six reported metrics than the compared methods (Section 4.1, Table 3). This provides an additional benchmark comparison, including against Uni-Mol pretrained on the same dataset.

3. Establishing the contributions of atom-level, substructure-level, and joint training

Reviewers EgaF and Kc3c pointed out that the original QM9-pretraining ablations did not establish a consistent advantage of joint training over atom-only pretraining.

  • We clarified that atom-level prediction is itself a central contribution, and that its strong standalone results demonstrate its independent effectiveness.
  • We additionally compared atom-only, substructure-only, and joint pretraining on the same random sample of 50K structures from Uni-Mol, with identical pretraining settings and downstream fine-tuning on Biogen and ExpansionRx (Section 4.2, Table 6). Joint training performs best overall in this setting, further supporting its use in our main downstream evaluations. However, joint training is neither the main contribution nor essential to the method’s effectiveness. The revised discussion clarifies that its relative benefits depend on the pretraining data and downstream task. We offer complementary supervision across Uni-Mol’s broader structural diversity as a plausible interpretation of the gains in this setting.

4. Evaluating performance with limited downstream training data

Reviewer 3H6u requested an explicit data-efficiency experiment. We added evaluations using subsamples of the downstream training data, comparing pretrained Atom-JEPA with the same encoder trained without pretraining and evaluating both on the same held-out test sets. Pretraining yields lower mean error at every evaluated training fraction on both Biogen and ExpansionRx, and errors decrease monotonically as more training data becomes available. The experimental setup and complete results are included in Appendix G, Table 29.

5. Clarifying practical costs

Reviewer Kc3c asked whether the accuracy gains justify conformer-based inference. We added measurements of conformer generation and inference throughput in Appendix E.1, including the trade-off between using one and ten conformers.

We thank the reviewers again for the constructive suggestions that motivated these additional experiments and clarifications.

References

[1] Feng et al. Fractional denoising for 3D molecular pre-training.

[2] Ramlaoui et al. Triforces: Augmenting atomistic GNNs for transferable representations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.