- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents two complementary contributions for molecule–text retrieval. 1. MolSynText: a synthetic molecule–text dataset in which textual descriptions are generated by Qwen3.8-27B from RDKit-derived “molecular cards” and subsequently filtered using the same model as a judge. Each molecule in MolSynText is also paired with its corresponding PubChem-derived text from STM. 2. MolTokSyn: a retrieval model that augments a Chemprop D-MPNN with K one-way virtual readout nodes, following the design of Llip. Experimentally, the authors report substantial improvements on MolSynText. The proposed approach also yields improvements in linear probing and on the Semantic Benchmark.
Critical strengths
-
The experimental setup is generally well controlled, e.g., using the same molecules, splits, and architectures across the two text sources allows a relatively direct comparison of textual supervision. Retrieval is evaluated over 12K candidates and three scaffold splits.
-
The paper reports the data construction pipeline in useful detail, including prompts, judge criteria, acceptance thresholds, and rejected examples.
-
The architecture is simple and easy to understand, preserves the pretrained message-passing backbone, and allows molecular tokens to be cached.
Critical weaknesses
- The ablations do not show that text conditioning is responsible for the main gains. Several results make it difficult to determine whether the improvements come from text conditioning, the molecular backbone, pretraining, or training configuration.
- With K=1, the softmax in Eq. (4) is identically 1, so the model effectively reduces to a dual encoder. Yet K=1 is best on all eight STM metrics in Table 12, and K=1 and 8 are nearly identical on MolSynText in Table 13. The only evidence in favor of query conditioning is Table 11, but that comparison uses InfoNCE and a smaller batch, does not specify how the “Global” variant pools, and does not report K=1 under the same setting.
- The linear probing study compares against a randomly initialized model rather than frozen CheMeleon. Without a frozen CheMeleon control, it is difficult to separate the effect of molecule–text training from the pretrained backbone.
- MolSynText needs stronger validation
- The same model is used to generate and judge the captions, which may produce correlated errors. The paper does not report the rejection rate, retry count, or a human-audited error rate. A blind expert audit of a random sample would strengthen the dataset contribution.
- Some captions contain exact descriptor values or IUPAC names. With ~12K retrieval candidates, combinations of exact numerical values may strongly narrow down molecule identity.
- Higher absolute retrieval performance on MolSynText than on STM does not by itself show that MolSynText provides more informative supervision; it may simply indicate that MolSynText is an easier retrieval task.
-
The Semantic Benchmark contains only 150 items, including 20 Relational questions. The benchmark was written by the authors using phrasing close to the MolSynText templates, and some labels follow the same RDKit conventions. The improvement may therefore partly reflect alignment between the training and evaluation text rather than stronger chemical understanding.
-
The architectural novelty is limited, as the model largely adapts Llip to molecular graphs, while its multi-token readout is related to set-based graph pooling methods such as PMA in Set Transformer and Graph Multiset Transformer. These connections could be discussed more clearly.
Questions for authors
- What pooling is used in the “Global” variant in Table 11, and how does it differ from MolTokSyn with (K=1)?
- Do the baselines use their original molecular encoders (e.g., GIN or Uni-Mol) or the same CheMeleon-initialized D-MPNN? If the former, how large is MolTokSyn's advantage on MolSynText when a strong baseline uses the same CheMeleon encoder?
- What are the linear-probe results for frozen CheMeleon and for RDKit descriptors?
- What are the judge rejection rate and the human-audited error rate of the final captions?
- What fraction of MolSynText test captions contain exact descriptor values or IUPAC names, and how does retrieval change after removing them?
- For the Semantic Benchmark, the paper states that the test cases were manually defined. How were the correct answers verified, e.g., programmatically with RDKit for computable properties, or by independent annotators?
- is CheMeleon pretrained on classical molecular descriptors, many of which overlap with MolSynText content, including logP, TPSA, HBD/HBA, and ring counts? This may contribute to the gains on MolSynText.
8 comments
Response to Reviewer FYb8 (2)
W2: Caption validation and identity cues. Please see our general response B on caption validation, which reports the independent judges, the regeneration rates and a blinded expert review (Appendix B.4). In addition, we want to address hypothesis that the larger number of stored captions could translate into more training examples. For that we created two datasets based on STM and MolSynText with exactly one molecule-text pair per molecule (so each of them has 125k captions in total), which can be seen in Appendix D.1, Table 14. Even in this setting, MolSynText reaches 78.11% and 78.37% R@1, compared with 51.94% and 51.38% for STM. This shows that the improvement cannot be explained by the larger number of stored MolSynText captions. Moreover, to address higher descriptor density hypothesis we created new dataset (one caption per molecule), with removed explicit molecular names (IUPAC), SMILES, and numerical sequences. MolSynText still achieves 73.10% and 73.67% R@1. Summing it up, these controls show that the observed gains are not simply a consequence of greater text diversity or an easier retrieval problem driven by simple distinguishing cues. Moreover, MolSynText supervision improves downstream linear probing and the Controlled Semantic Benchmark across multiple architectures, showing that the richer information is useful beyond simply matching easier retrieval captions (what was also a goal of this paper). These results are discussed in Appendix D.1 and Sections 5.2-5.3.
W3. Train/evaluation text alignment and benchmark size. We partly agree, and the revised paper now states this explicitly (Section 5.3, Table 12, and Appendix E in the revised paper). MolSynText verbalizes an RDKit card (atom and ring counts, H-bond donors/acceptors, rotatable bonds, TPSA, logP, charge, functional groups). Categories that test these concepts are intentionally close to the training signal and share its RDKit conventions: they check whether a model acquired what the text teaches, and this is where MolSynText helps most (Global family: +21.4 points for our model, Table 6).
The other categories are out of distribution. Regiochemistry, ring topology, tautomer and protonation forms, analog edits and bioisosteric replacement are absent from the card and are phrased as a medicinal chemist would query the model. Several are nonetheless solved well: the Relational family reaches 61.7% for our model against 25% chance, bioisosteric replacement is among the highest-scoring categories for most models, and tautomer/protonation is well above chance (Figure 12). Ring topology and regiochemistry remain near chance, which we report. These five categories are not computed in the cards, so performance there is less tied to the cards’ RDKit conventions.
Two further points limit what alignment can explain. First, MolSynText has no fixed sentence templates: the templates are instructions telling the generator which aspect of a molecule to emphasize, and the descriptions themselves are free-form text of 2-4 sentences. Benchmark items are short single-claim statements, and several deliberately avoid descriptor vocabulary (e.g. "greasiest" or "most water-averse" instead of logP wording). Second, all six models are trained on the same text, so alignment is identical across them and does not by itself explain the between-model ranking.
We agree the benchmark is small. Per-category results (10 items) are diagnostic only; we draw conclusions from the overall score and the Global and Local families, and the revised paper states this limitation in Appendix E.
Response to Reviewer FYb8 (3)
W4: Relation to Llip and set-based pooling. We thank the Reviewer for pointing out these connections. We agree that these relations should be discussed more clearly.
Llip. Our model is inspired by the mixture-token idea of Llip (Lavoie et al., 2024), which we credit in Sections 1 and 4. Our adaptation is on the molecular side and is designed for message-passing networks: we add K learned one-way virtual readout nodes to the D-MPNN. These nodes collect information from the molecular graph but send no messages back, so message passing between the real atoms and bonds is unchanged (Section 4.1). Each virtual node has its own learned features and projection, producing several molecular representations without predefined chemical meaning. To our knowledge, this one-way virtual-node readout has not previously been used for molecule-text retrieval.
The second difference is how these representations are used. The molecular tokens are computed once per molecule and can be cached, and each text query combines them differently (Section 4.2). The query-conditioned representation is therefore the retrieval score itself, rather than a re-ranking step after first-stage retrieval as in MolCA and MolFM, and the text combines the tokens with learned attention weights rather than a hard maximum over them.
Set-based pooling. Producing several pooled vectors per graph from learned slot parameters is related to attention-based pooling with learned seed vectors, as in PMA of the Set Transformer (Lee et al., 2019) and the Graph Multiset Transformer (Baek et al., 2021), which computes its keys and values with a GNN. MolTokSyn differs in two ways. First, our slots do not attend over atoms: each virtual node aggregates atom messages through one-way edges inside the D-MPNN. Second, PMA and GMT produce a query-independent graph representation, whereas our slot outputs are mixed by the text query at scoring time, following Llip. The query-independent controls in Appendix D.2, a single virtual token () and uniform weights over the same tokens, measure what this query-dependent mixing adds on top of a multi-slot readout.
Ablations. Finally, we also provide ablations separating these design choices rather than treating the architecture as one block. The K sweep tests the value of multiple molecular tokens, the Query vs Uniform experiment isolates text-dependent aggregation, and the attention analysis verifies that routing actually changes with the query. Additionally, we study the effect of the contrastive objective and physical batch size separately.
Contribution. More broadly, the main contribution is the combination of a structure-grounded textual supervision pipeline with a molecular architecture designed to exploit that richer signal. The aim is not only better retrieval but molecular representations that capture chemical concepts from text. This is why we also evaluate downstream property prediction (Section 5.2) and the Controlled Semantic Benchmark (Section 5.3), which tests specific chemical concepts directly.
Q1: Global pooling versus K=1. We thank the Reviewer for this question and we clarified this distinction in the revised Appendix D.2. The Global variant uses the standard Chemprop graph readout: the final atom-level representations are aggregated using MeanAggregation into a single CheMeleon graph fingerprint, which is then projected to the shared 128-dimensional space, L2-normalized, and compared with the independently encoded SciBERT text representation. It therefore does not use virtual molecular nodes, molecular-token keys or values, or query-dependent attention.
MolTokSyn with K=1 is different. It replaces the standard global graph pooling with one learned one-way virtual readout node and uses the MolTokSyn token/value/output projection pathway. Because there is only one molecular token, the softmax weight in Eq. (4) is always 1, so the resulting molecular representation is query-independent. Therefore, K=1 is not equivalent to the Global baseline. Global uses conventional mean-pooled graph readout, while K=1 tests our learned virtual-token readout in its query-independent special case. Table 15 compares both variants under the same training setup. Global pooling obtains 58.78% and 59.40% R@1, whereas the learned K=1 virtual readout reaches 76.39% and 77.29%. The full query-conditioned K=8 model further improves these scores to 80.71% and 80.82%, respectively.
Response to Reviewer FYb8 (1)
We thank the Reviewer for the valuable feedback. All changes are marked in blue in the revised paper. We address each point below.
W1: Sources of the gains. We agree that part of the previous model ablations was run under varying setup. To address this, we reran the key experiments using the final training recipe with SigLIP, physical batch size 512, and matched optimization settings. Here is what we completed so far: the full K sweep on MolSynText (K=1, 4, 8, 16), the STM multi-token sweep (K=1, 4, 8, 16), the MolSynText-15x sweep (K=1, 8, 32), and the matched K=8 Query vs Uniform aggregation experiment.
We also added a matched comparison with conventional Global pooling (Table 15), using the final training recipe with SigLIP and physical batch size 512. Replacing Global pooling with a single learned virtual token improves R@1 by 17.61 and 17.89 points in the two retrieval directions. Increasing capacity from K=1 to query-conditioned K=8 provides further gains of 4.32 and 3.53 points, respectively.
In addition, we added a matched control that keeps the same K=8 molecular tokens, keys, values, and scoring architecture, but replaces text-dependent attention with uniform aggregation, which can be seen in section Appendix D.2. As shown in Table 16, query conditioning improves R@1 by 6.96 percentage points for molecule-to-text and 6.61 points for text-to-molecule retrieval, with corresponding MRR gains of 5.49 and 5.05 points. This shows that the gain is not explained simply by using multiple molecular tokens, but also conditioning it with text. To address it further we added a token analysis showing that the learned routing actually changes with the query, which can be seen in Appendix D.2. With the molecular graph and K=8 tokens fixed, 84.75% of molecule-attention-head combinations change their dominant slot across natural descriptions. Under eight controlled semantic-focus queries, the dominant slot differs in 48.1% of pairwise molecule-head comparisons on average (Figure 6). Summing it up, these results support query-dependent reweighting of the molecular representation.
Additionally, K=1 is indeed a special case of our model. With only one molecular token, the attention weight in Eq. (4) is always 1, so the molecular representation no longer depends on the text query. Therefore, at the readout level, it behaves as a standard molecular embedding, which makes the K=1 comparison very useful, because the gains for K>1 show the benefit of having multiple tokens that can be weighted differently for different queries.
Additionally, on MolSynText, increasing K from 1 to 8 improves R@1 from 74.41% to 78.76% and from 75.08% to 79.02% in the two retrieval directions, while K=16 gives the best central value on 7 of 8 metrics. On STM, increasing K from 1 to 4 improves R@1 by 3.61 and 3.57 percentage points in the two retrieval directions. Performance then remains largely stable across K=4, 8, and 16, with differences lying within the reported standard deviations. This supports our claim that extra token capacity is useful when the textual supervision is rich enough to exploit it in both MolSynText and STM setup (what was a main goal of this architecture).
CheMeleon pretraining. To assess the contribution of CheMeleon pretraining, we added an ablation in Appendix D.6, “Initialization of the Molecular Encoder”, comparing pretrained and random initialization of MolTokSyn’s molecular encoder. Across both pretraining datasets, the randomly initialized variant achieves higher mean scores on every retrieval metric and lower mean errors on all regression tasks, while the relative performance on classification tasks varies by task. We provide the full ablation results and further discussion in our general response A on molecular encoder initialization, downstream transfer and our response to Question 3 (What are the linear-probe results for frozen CheMeleon and for RDKit descriptors?).