acceptodds
Under review as a conference paper at ICLR 2027

Mol-JEPA: A multimodal Joint Embedding Predictive Architecture for Molecules

21 views

Abstract

Despite recent advances in molecular foundation models, several limitations remain, such as chemically invalid augmentations, modality collapse, and incomplete representation of biochemical environments. To address these challenges, we present **Mol-JEPA**, a scalable framework for learning molecular world models. Rather than relying on suboptimal molecular perturbations, our model uses modality masking to exploit information from molecular structures, cellular phenotypes, binding affinities, ADMET profiles, quantum chemistry simulations and other drug discovery data. Across various benchmarks, we show that the representations learned by Mol-JEPA deliver strong performance, demonstrating the value of incorporating biochemical context through latent space prediction.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Molecular Machine Learning, Foundation Model, World Model, Chemistry

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ZZAiOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Mol-JEPA, a multimodal framework that learns molecular representations by masking a subset of available modalities and predicting their latent representations from the remaining modalities. Rather than reconstructing raw inputs or perturbing molecular structures, the model uses cross-modal prediction to learn relationships among structural, physicochemical, and biological information. An isotropic regularization objective is used to prevent representation collapse. The framework is evaluated on downstream molecular property prediction benchmarks, with ablation studies examining the contributions of different modalities and training choices.

Critical strengths
  1. A well-motivated cross-modal prediction task for molecular representation learning. The framework randomly masks a subset of available modalities while retaining at least one as context, and predicts the latent representations of the masked modalities from the remaining information. This provides a meaningful learning signal without modifying the molecular structure and encourages the model to capture relationships among different molecular views.
  2. Integration of complementary molecular information with empirical support. The framework combines structural features, pretrained representations, computed descriptors, and experimental measurements, while accommodating missing modalities. The full-data comparison shows that using all modalities yields better downstream performance than using only two key modalities. Together with the downstream evaluations, this provides evidence for the practical value of the multimodal approach.
Critical weaknesses
  1. The provenance and meaning of the modalities need clearer presentation. A representation produced by a model trained on biological measurements is not equivalent to a direct biological observation of the current molecule. For each modality, the paper should explicitly identify the raw input used during extraction, the pretrained checkpoint and encoder branch, and the transformation applied before joint training. This distinction matters for assessing what information is actually available and what knowledge is transferred.

  2. The interpretation of performance gains would benefit from sharper attribution. I appreciate the modality and experimental-label ablations, but would like a clearer separation between benefits from the predictive training objective, the additional information sources, and downstream model selection. In particular, the “Best” configuration should be accompanied by an explicit account of how the configuration was selected without using test outcomes. These are addressable concerns that would strengthen the evidence rather than change the core contribution.

Questions for authors

Questions for Authors

  1. Could you clarify the raw input and preprocessing pipeline for each modality, before pretrained feature extraction or descriptor calculation? In particular, which encoder branches are used for CLOOME and BioXMol, and do their inputs consist of molecular structures or experimental biological measurements?
  2. Why does the modality-scaling experiment in Figure 4(a) stop at eight modalities rather than include all fourteen? Clarifying this would help distinguish the effect of modality count from the effect of the particular modalities included.
  3. The paper notes that sparse modalities are harder to optimize. How are the prediction and SIGReg losses normalized when modality availability varies substantially across samples? Is any sampling or weighting used to prevent frequently available modalities from dominating training, and how is SIGReg computed when a modality has very few valid samples in a batch?
  4. The discussion describes the training patterns in Figure 5, but could the authors explain why UMA’s SIGReg loss fluctuates without a clear downward trend, whereas ChEMBL’s prediction loss initially increases and then decreases? Do these patterns have implications for convergence or downstream representation quality?

Minor Comments

  1. In Appendix Table 4, should the number of remaining assays be non-increasing as the minimum measurement threshold increases? The reported count rises from 17 at a threshold of 500 to 24 at 1,000. Please check these entries or clarify whether the filtering procedure differs between thresholds.
  2. Please check whether Figure 11 around line 1151 in the submitted version has a complete descriptive caption and all necessary labels.
2 comments
AuthorsOpenReview ↗

Response to Reviewer ZZAi

Dear Reviewer ZZAi,

Thank you for the positive and constructive feedback, which helps us to further improve the quality of our work. Below we have addressed each of your open points to clarify the aspects you have raised.

[Modality encoding details]

Reviewer: What transformations are applied to each modality? What is used as an input?

Response: Thank you for this feedback. Generally, we have obtained all modalities with SMILES/SDF files as a starting point. For descriptors and pretrained models we have simply computed the representation based on Python libraries / Software or by querying the pretrained model. For CLOOME and BioXMol specifically, we have only used the molecule branch, i.e. the image encoder was not used. For Boltz, we have additionally included proteins from the safety screen panel, i.e. the encoder received both the protein sequence and ligand SMILES. Lastly, the multi-dimensional target vectors, which we refer to as experimental modalities, are simply all available measurements about a molecule (e.g. all in vitro assays), which can be considered as an in vitro / in vivo profile. In Appendix A we have included all preprocessing and dataset details.

[Modality scaling experiment]

Reviewer: Why does the modality-scaling experiment stop at eight modalities?

Response: Thank you for raising this point. To obtain a simple scaling law, we doubled the number of modalities to see if it also doubles the training time. As the next step would’ve been 16 modalities, which exceeds the number of available modalities, we decided to only show the variants 2, 4 and 8. We didn’t use any specific logic for the selection of modalities, despite that the 8 modalities are among the most important ones from our leave-one-modality-out experiment, to show the largest effects. Also, we tried to balance the embedding sizes, such that each experiment contains diverse enough modalities. Finally, we found an performance increase for the majority of modalities (see Figure 4 a), therefore the experiment would look similarly if the order of modalities was changed.

[Modality sparsity strategies]

Reviewer: Was any normalization / sampling used to deal with sparsity?

Response: Thank you for this suggestion. We have experimented with a weighted loss (available in the provided code), by weighing each sample in a batch by the relative frequency computed from the dataset (i.e. 1/N_modality). We didn’t observe any performance gains over regular training, why we didn’t further investigate weighting.

[SIGReg computation]

Reviewer: How is SIGReg computed when a modality has few samples in a batch? Why does UMA fluctuate and why has ChEMBl such a strange pattern?

Response: Thank you for this important feedback. SIGReg is applied separately to each modality using only the samples for which that modality is available. When fewer than two samples are available, the modality’s SIGReg contribution is set to zero. Otherwise, embeddings are projected onto 1024 unit directions. We do not apply an additional small-sample correction, but this might be something to try for future work. Regarding the modality-specific SIGReg behavior:

  • UMA: Here we hypothesize that the embeddings play a minor role in the optimization (see Figure 4 b) and therefore are only rarely considered in the loss. This might stem from the fact that 3D information is challenging to reconstruct for most of the modalities.
  • ChEMBl: The increasing and then decreasing trend can be explained by the model first learning general molecular patterns and later optimizing for the experimental properties

[Sharper attribution of performance gains]

Reviewer: A clearer separation between benefits from training objective, modalities and downstream model would be beneficial. Also, how was the “Best” configuration selected?

Response: Thank you for your suggestion. The selection of our training objective is based on previous literature and theoretical considerations that rule out other techniques. Contrastive methods rely on paired examples, which are not available for our multimodal dataset and reconstruction-based methods (such as autoencoders) are less favorable because of noise in reconstructed data (see response for reviewer 8Ufy). Regarding performance gains from modalities, we have studied the attribution of each modality in detail, as well as provided results for the performance without modalities (Figure 14 b). Similarly, we have evaluated the performance of each downstream model in Figure 3 b), which shows how much of the performance gains can be attributed to model choice. The best configuration is selected as the best performing combination of downstream model and Mol-JEPA features for each dataset.

[Minor comments]

Thank you for the comments. We added a description to Figure 11 and double-checked the number of assays in Table 4. The numbers were accurate but the table header was misleading.

Reviewer_ZZAiOpenReview ↗

Follow-up after author response

Thank you for the detailed response and clarifications. The additional explanations on modality encoding, the scope of the modality-scaling experiment, and the handling of sparse modalities are helpful. I also appreciate the clarification of the table header and the added figure description.

My overall assessment remains unchanged. I view the cross-modal prediction framework as a useful approach to molecular representation learning, and the empirical results support the value of combining complementary modalities. At the same time, clearer distinctions between direct experimental measurements and representations derived from pretrained models remain important for interpreting the contribution of each modality. The explanations of the modality-specific training behavior are plausible, but remain hypotheses rather than experimentally established conclusions. The attribution of performance gains and the selection of the “Best” configuration would also benefit from clearer presentation in the paper.

These remaining points do not change my positive evaluation. I maintain my original rating of 4 (Clear acceptance).

Reviewer_8UfyOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Mol-JEPA, a scalable framework for learning molecular world models. The authors curated a dataset with ~4.69M compounds, covering 14 different modalities such as molecular graphs, embeddings from pretrained models, molecular fingerprints, physicochemical descriptors, quantum chemical properties, binding affinity predictions, etc. The proposed model encodes these different modalities into a shared latent space, and predicts randomly masked modality embeddings with a Transformer model and a JEPA-style objective.

The learned representations are evaluated on 23 prediction tasks across 4 drug discovery benchmarks. Mol-JEPA achieves competitive performance compared to both traditional and novel baselines. Additional experiments have also been conducted on pertaining scale, contribution of different modalities, training, and different downstream prediction strategies.

Critical strengths

This framework integrates different types of biochemical data/modality and information at a larger scale. The authors curated a dataset with ~4.69M compounds covering 14 different modalities (as shown in Table 1). I think the dataset itself is a valuable contribution to the open source community, as it can be a useful resource for researchers working on various drug discovery tasks.

One concern we often have with multimodal representation learning is whether adding more modalities actually improves the performance, since the additional modalities can be a bonus or/and they can introduce noises. Of course it depends on tasks and context, but I appreciate how the authors examine this in the paper (as shown in Figure 4(a) and 4(b)). Figure 4(a) shows that including more modalities leads to better downstream performance, and Figure 4(b) demonstrates the contribution of each modality. The results suggest that in this context, incorporating various modalities does genuinely help the model.

I think this is an interesting extension of JEPA to multimodal molecular representation learning. While JEPA has already been applied for molecular data, this paper further incorporates different types of biochemical information, which has a lot of potential considering the abundance of data we have nowadays and the wide range of drug discovery tasks that could potentially benefit from general molecular representations.

The evaluation is also very thorough. It covers different aspects of the model, and the presentation is really clear.

Critical weaknesses
  1. This paper is built largely on established methods, including JEPA-style latent prediction, SIGReg regularization, and Transformer-based fusion. While I do agree that the results are strong, I think it is unclear how much of the improvement comes from the JEPA objective itself rather than the rich multimodal data. Although the authors compare JEPA against a semi-supervised alternative, I do not think it fully establish its advantage over other multimodal pretraining approaches.

  2. The pretraining data includes experimental measurements from ChEMBL, TDC, and PCBA, and some of these cover the same types of properties evaluated in the downstream tasks. The paper also uses several established pretrained models to generate the input modalities, which may have been trained on overlapping compounds or experimental measurements. While the authors use cluster-disjoint splits and similarity filtering, it might still be unclear whether the downstream test compounds and their corresponding measurements were included in either (1) Mol-JEPA's own pretraining data or (2) the training data of these pretrained models. If the model has already been exposed to the same measurements, the results might be inflated and look better than it actually is. This could also make it difficult to determine how well the model generalizes.

Questions for authors
  1. Did the authors double check whether any downstream test compounds and their corresponding experimental measurements were included in (1) either Mol-JEPA's pretraining data or the (2) training data of the pretrained models used to generate the modalities? How was the overlap handled? (See more details in point 2 in weakness section)

  2. Which modalities are actually used in different downstream inference? As the data collection of some modalities might be more complicated (extensive experiments or computations), how can the model handle new compounds where these measurements are not available?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 8Ufy

Dear Reviewer 8Ufy,

Thank you for your constructive feedback and valuable suggestions. In the following, we address each of your points to clarify the aspects you raised and provide further context for our approach.

[Contribution of JEPA objective]

Reviewer: It is unclear how much the objective contributes to the performance gains as no other multimodal pretraining objectives are evaluated.

Response: Thank you for this important feedback. Our objective was not to perform an exhaustive comparison of self-supervised learning objectives, but rather to evaluate whether Mol-JEPA can learn good representations from different information sources and deliver strong downstream performance. The reasons for selecting this architecture type are based on previous literature and theoretical considerations that rule out other training objectives:

  • Contrastive methods fundamentally rely on paired examples (i.e. all modalities are available). This is typically not the case for multimodal molecular datasets. Furthermore, contrastive methods require negative samples, which are difficult to compute because similarity is a vague term for molecule data (a molecule can behave similarly despite having a dissimilar structure).
  • Reconstruction-based methods such as autoencoders are theoretically and practically less favorable over JEPAs, because the reconstruction of data introduces noise (see [1])
  • Multimodal multi-task approaches are generally problematic, because it is difficult to balance the tasks and gradient conflicts occur that require additional treatment (e.g. see [2]).

Based on these considerations, we prioritized comparison against widely used baselines rather than alternative pretraining objectives. We agree that a systematic comparison between JEPA, contrastive, and reconstruction-based multimodal objectives would provide valuable additional insight. However, such a study would require extensive implementation and tuning of multiple alternative frameworks and is beyond the scope of this work. We view this as an important task for future research.

[Pretraining leakage check]

Reviewer: There might be leakage of molecules or target information in the Mol-JEPA training data or the training data of the integrated pretrained models.

Response: Thank you for this important feedback. We agree that the dataset requires careful curation to avoid leakage and overoptimistic evaluation. First, we want to highlight that our evaluation is based on very recent OpenADMET blind challenge datasets, which contain novel compounds that are not part of our pretraining data or the pretraining data of other models. Furthermore, as mentioned in the paper, we computed InChi codes for all molecules and made sure that none of the benchmarking compounds are present in the pretraining data. We acknowledge the reviewer's point regarding target-specific bias. A model pretrained on specific assay types, such as Caco-2 permeability, may show better downstream performance, even without molecule overlap. Isolating these target-assay attribution effects is challenging and we plan to incorporate better dataset attribution methods in future work.

[Modalities used during inference]

Reviewer: How are new compounds handled where measurements are not available?

Response: Thank you for highlighting this important point. At inference, we have always only used the SMILES string as input for Mol-JEPA and never computed any other modality. The SMILES is then converted to a graph with node and edge features, which is passed through the transformer module to predict all modalities / the CLS token. While it might further improve the performance, we have considered a multi-modal inference as slow and impractical, especially for computationally expensive modalities such as DFT features. Notably, we consider this as one of the main strengths of Mol-JEPA, as it allows to approximate all modalities based on a simple SMILES input. In theory, the model however accepts multiple inputs at inference and handles the missing ones with special masking tokens.

[1] Joint Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self-Supervised Learning, Van Assel et al. 2025

[2] Gradient Surgery for Multi-Task Learning, Yu et al. 2020

Reviewer_nsd2OpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

Mol-JEPA learns molecular representations in a JEPA-style framework, by predicting embeddings that are masked. It combines descriptors, graphs, pretrained-model embeddings, computational properties, etc. up to 14 modalities. Its largest contribution and its largest weakness, is that it expanded existing LeJEPA pipeline to a multi-modality setting.

Critical strengths
  • As stated before, the critical strength is regarding the import of LeJEPA on to multimodal training in molecular domain. The curation and processing of the multimodal data, as noted by authors, is the most critical strength of the paper.
Critical weaknesses
  • Since the inputs already include strong pretrained representations, including Boltz-2 embeddings, the comparison does not distinguish the gains from learning objective / multimodal training over simply using more powerful representation inputs. https://arxiv.org/abs/2602.13249 -> this paper has already shown potential in using boltz embeddings for downstream tasks, adding questions to the authors' claims in the necessity of such a 'multi-modal' learning.
  • The formatting of the paper is significantly wrong. (text width is much larger)
Questions for authors
  • [Experimental Questions] The leave-one-modality-out experiments provide useful evidence, but is not sufficient to isolate the contribution of the proposed pretraining framework. It might be argued, that JEPA is not suitable for extracting some of the information of certain modalities. it would be best if authors can experimentally distinguish such gaps.

Comments on the paper

1 comment

Dear reviewers,

As the authors have posted their responses, I would encourage you to actively parcitipate in the dicussion.

Thank you.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.