- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Mol-JEPA, a multimodal framework that learns molecular representations by masking a subset of available modalities and predicting their latent representations from the remaining modalities. Rather than reconstructing raw inputs or perturbing molecular structures, the model uses cross-modal prediction to learn relationships among structural, physicochemical, and biological information. An isotropic regularization objective is used to prevent representation collapse. The framework is evaluated on downstream molecular property prediction benchmarks, with ablation studies examining the contributions of different modalities and training choices.
Critical strengths
- A well-motivated cross-modal prediction task for molecular representation learning. The framework randomly masks a subset of available modalities while retaining at least one as context, and predicts the latent representations of the masked modalities from the remaining information. This provides a meaningful learning signal without modifying the molecular structure and encourages the model to capture relationships among different molecular views.
- Integration of complementary molecular information with empirical support. The framework combines structural features, pretrained representations, computed descriptors, and experimental measurements, while accommodating missing modalities. The full-data comparison shows that using all modalities yields better downstream performance than using only two key modalities. Together with the downstream evaluations, this provides evidence for the practical value of the multimodal approach.
Critical weaknesses
-
The provenance and meaning of the modalities need clearer presentation. A representation produced by a model trained on biological measurements is not equivalent to a direct biological observation of the current molecule. For each modality, the paper should explicitly identify the raw input used during extraction, the pretrained checkpoint and encoder branch, and the transformation applied before joint training. This distinction matters for assessing what information is actually available and what knowledge is transferred.
-
The interpretation of performance gains would benefit from sharper attribution. I appreciate the modality and experimental-label ablations, but would like a clearer separation between benefits from the predictive training objective, the additional information sources, and downstream model selection. In particular, the “Best” configuration should be accompanied by an explicit account of how the configuration was selected without using test outcomes. These are addressable concerns that would strengthen the evidence rather than change the core contribution.
Questions for authors
Questions for Authors
- Could you clarify the raw input and preprocessing pipeline for each modality, before pretrained feature extraction or descriptor calculation? In particular, which encoder branches are used for CLOOME and BioXMol, and do their inputs consist of molecular structures or experimental biological measurements?
- Why does the modality-scaling experiment in Figure 4(a) stop at eight modalities rather than include all fourteen? Clarifying this would help distinguish the effect of modality count from the effect of the particular modalities included.
- The paper notes that sparse modalities are harder to optimize. How are the prediction and SIGReg losses normalized when modality availability varies substantially across samples? Is any sampling or weighting used to prevent frequently available modalities from dominating training, and how is SIGReg computed when a modality has very few valid samples in a batch?
- The discussion describes the training patterns in Figure 5, but could the authors explain why UMA’s SIGReg loss fluctuates without a clear downward trend, whereas ChEMBL’s prediction loss initially increases and then decreases? Do these patterns have implications for convergence or downstream representation quality?
Minor Comments
- In Appendix Table 4, should the number of remaining assays be non-increasing as the minimum measurement threshold increases? The reported count rises from 17 at a threshold of 500 to 24 at 1,000. Please check these entries or clarify whether the filtering procedure differs between thresholds.
- Please check whether Figure 11 around line 1151 in the submitted version has a complete descriptive caption and all necessary labels.
2 comments
Follow-up after author response
Thank you for the detailed response and clarifications. The additional explanations on modality encoding, the scope of the modality-scaling experiment, and the handling of sparse modalities are helpful. I also appreciate the clarification of the table header and the added figure description.
My overall assessment remains unchanged. I view the cross-modal prediction framework as a useful approach to molecular representation learning, and the empirical results support the value of combining complementary modalities. At the same time, clearer distinctions between direct experimental measurements and representations derived from pretrained models remain important for interpreting the contribution of each modality. The explanations of the modality-specific training behavior are plausible, but remain hypotheses rather than experimentally established conclusions. The attribution of performance gains and the selection of the “Best” configuration would also benefit from clearer presentation in the paper.
These remaining points do not change my positive evaluation. I maintain my original rating of 4 (Clear acceptance).
Response to Reviewer ZZAi
Dear Reviewer ZZAi,
Thank you for the positive and constructive feedback, which helps us to further improve the quality of our work. Below we have addressed each of your open points to clarify the aspects you have raised.
[Modality encoding details]
Reviewer: What transformations are applied to each modality? What is used as an input?
Response: Thank you for this feedback. Generally, we have obtained all modalities with SMILES/SDF files as a starting point. For descriptors and pretrained models we have simply computed the representation based on Python libraries / Software or by querying the pretrained model. For CLOOME and BioXMol specifically, we have only used the molecule branch, i.e. the image encoder was not used. For Boltz, we have additionally included proteins from the safety screen panel, i.e. the encoder received both the protein sequence and ligand SMILES. Lastly, the multi-dimensional target vectors, which we refer to as experimental modalities, are simply all available measurements about a molecule (e.g. all in vitro assays), which can be considered as an in vitro / in vivo profile. In Appendix A we have included all preprocessing and dataset details.
[Modality scaling experiment]
Reviewer: Why does the modality-scaling experiment stop at eight modalities?
Response: Thank you for raising this point. To obtain a simple scaling law, we doubled the number of modalities to see if it also doubles the training time. As the next step would’ve been 16 modalities, which exceeds the number of available modalities, we decided to only show the variants 2, 4 and 8. We didn’t use any specific logic for the selection of modalities, despite that the 8 modalities are among the most important ones from our leave-one-modality-out experiment, to show the largest effects. Also, we tried to balance the embedding sizes, such that each experiment contains diverse enough modalities. Finally, we found an performance increase for the majority of modalities (see Figure 4 a), therefore the experiment would look similarly if the order of modalities was changed.
[Modality sparsity strategies]
Reviewer: Was any normalization / sampling used to deal with sparsity?
Response: Thank you for this suggestion. We have experimented with a weighted loss (available in the provided code), by weighing each sample in a batch by the relative frequency computed from the dataset (i.e. 1/N_modality). We didn’t observe any performance gains over regular training, why we didn’t further investigate weighting.
[SIGReg computation]
Reviewer: How is SIGReg computed when a modality has few samples in a batch? Why does UMA fluctuate and why has ChEMBl such a strange pattern?
Response: Thank you for this important feedback. SIGReg is applied separately to each modality using only the samples for which that modality is available. When fewer than two samples are available, the modality’s SIGReg contribution is set to zero. Otherwise, embeddings are projected onto 1024 unit directions. We do not apply an additional small-sample correction, but this might be something to try for future work. Regarding the modality-specific SIGReg behavior:
[Sharper attribution of performance gains]
Reviewer: A clearer separation between benefits from training objective, modalities and downstream model would be beneficial. Also, how was the “Best” configuration selected?
Response: Thank you for your suggestion. The selection of our training objective is based on previous literature and theoretical considerations that rule out other techniques. Contrastive methods rely on paired examples, which are not available for our multimodal dataset and reconstruction-based methods (such as autoencoders) are less favorable because of noise in reconstructed data (see response for reviewer 8Ufy). Regarding performance gains from modalities, we have studied the attribution of each modality in detail, as well as provided results for the performance without modalities (Figure 14 b). Similarly, we have evaluated the performance of each downstream model in Figure 3 b), which shows how much of the performance gains can be attributed to model choice. The best configuration is selected as the best performing combination of downstream model and Mol-JEPA features for each dataset.
[Minor comments]
Thank you for the comments. We added a description to Figure 11 and double-checked the number of assays in Table 4. The numbers were accurate but the table header was misleading.