- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies red-teaming of molecule language models for toxic molecule generation. Its main new contribution is a decoding-time attack that uses a learned surrogate to estimate whether a partial SMILES is likely to lead toward molecules structurally similar to known toxic compounds, and then uses that score to guide beam decoding. The paper also curates toxicity datasets and trains endpoint-specific toxicity evaluators, but these components are mostly supporting infrastructure rather than the main methodological novelty.
I will therefore mainly evaluate the paper on whether the structural-similarity prior is well justified across different toxicity endpoints, whether the surrogate-guided decoding method is sound and genuinely effective, and whether the toxicity evaluators are reliable enough to support the reported conclusions.
Critical strengths
-
The method is conceptually simple and well motivated: it converts an expensive rollout-based toxicity-related objective into a lightweight surrogate signal that can be used during decoding, making the attack practical across multiple autoregressive molecule models.
-
The structural-similarity prior shows strong discrimination for some toxicity endpoints, with ROC-AUC reaching 0.961 at the high end. This supports the idea that structural similarity can provide a useful signal for guiding red-teaming in at least some toxicity settings.
-
The proposed decoding strategy is empirically effective across eight molecule language models, consistently increasing overall toxicity rates and substantially outperforming greedy, top-k, top-p, and standard beam decoding. The guidance-interval analysis also shows that more frequent surrogate guidance generally leads to stronger attacks.
Critical weaknesses
-
The structural-similarity prior is not uniformly strong across toxicity endpoints. The paper reports ROC-AUCs as low as 0.653 for stress response and 0.700 for micronucleus induction, indicating only modest discrimination for some endpoints. This suggests that structural similarity provides useful signal in some cases, but is not a consistently strong toxicity prior across all endpoint types.
-
The ablation analysis is insufficient to explain why the method works. The paper mainly varies the guidance interval, but does not report the surrogate’s held-out prediction quality or isolate key components with controls such as random/unguided scoring, rollout-based oracle guidance, prompt-vs-prefix conditioning, or matched-compute beam search. As a result, the overall attack is clearly effective, but the specific contribution of the learned surrogate is less well established.
-
The benchmarking lacks competing red-teaming or guided-decoding baselines, making it difficult to assess the proposed attack against alternative methods.
Minors
-
Several important datasets/resources are mentioned without a proper citation. Appendix C relies on resources such as HCIS / Safe Work Australia, OECD QSAR Toolbox, REACH/ECHA, CPDB, ISS ISSTOX, and OECD eChemPortal, but several of these are only named in prose rather than given a bibliographic reference or URL. The same problem applies to some primary toxicity datasets.
-
In the introduction, the paper states that the toxicity evaluators achieve “over 90% average accuracy across all toxicity endpoints.” However, Table 4 reports MoLFormer accuracies of 95.05%, 83.53%, 95.42%, 65.05%, 82.50%, 85.45%, 75.02%, and 85.04% across the eight endpoints. The simple average is only about 83.4%, not over 90%. The reported 93.43% is explicitly a sample-size-weighted overall accuracy, rather than an average across endpoints. This makes the statement in the introduction misleading and overstates the evaluator performance.
-
There is a wording conflict in the conclusion. The authors state that their method succeeds while “requiring only one forward pass in the target model.” However, the proposed method performs repeated surrogate-guided beam decoding: at each guidance interval, it runs beam search over candidate continuations, selects one, updates the prefix, and repeats this procedure until decoding terminates. This description appears inconsistent with the claim of requiring only one forward pass through the target model.
Questions for authors
Could the authors clarify how the train/validation/test splits were constructed for each toxicity evaluator, especially when combining datasets from multiple sources? In particular, were scaffold- or similarity-based splits used, or were near-duplicate/closely related analogs controlled across splits beyond exact molecule deduplication?
4 comments
Response to Reviewer 7qG3 (Part 2)
In response to Weakness 2 (Continued)
Prompt-vs-prefix conditioning. We retrained a prefix-only surrogate model using the same molecular prefix encoder and training protocol, while removing the prompt-conditioning branch. We evaluated this variant on ChemAligner-T5 and ChemAligner-T5-Pro across all eight toxicity endpoints.
| Model | Greedy | Prefix-only | Prompt + Prefix (Ours) |
|---|---|---|---|
| ChemAligner-T5 | 16.79% | 56.64% | 58.15% |
| ChemAligner-T5-Pro | 36.24% | 55.18% | 60.86% |
As shown above, the prefix-only surrogate already achieves substantial improvements over greedy decoding, while joint prompt-prefix conditioning achieves better performance on average. However, when initially designing our surrogate model, we recognized that relying solely on prefixes would overlook the semantic information in the input captions. In particular, the same molecular prefix may require different continuations under different prompts, whereas a prefix-only surrogate cannot distinguish these contexts, potentially limiting the diversity and flexibility of prompt-conditioned generation. Joint conditioning addresses this limitation by considering both molecular structure and input semantics.
Furthermore, prompt-only conditioning cannot provide meaningful guidance during decoding, as all candidate continuations receive identical surrogate scores under the same input prompt, causing the decoding process to revert to standard beam search. Taken together, these considerations motivate our joint prompt-prefix conditioning design, which leverages both semantic and structural information for effective generation guidance.
Matched-compute beam search. As reported in Table 5 of the supplementary material, we have already compared our method against standard beam search across all eight molecule language models. Our surrogate-guided decoding consistently achieves substantially higher average toxicity rates, demonstrating that the improvements are not simply attributable to beam search itself, but rather to the effectiveness of our proposed framework.
We believe these results provide a clearer understanding of why our method works and will incorporate them into the revised manuscript or supplementary materials, as appropriate. We hope these additional analyses and controlled experiments adequately address the reviewer's concerns.
In response to Weakness 3
To the best of our knowledge, our work is the first to systematically investigate red-teaming of molecule language models for toxicity-oriented generation, which represents a key novelty of this study. Existing red-teaming methods are primarily designed for natural language models and cannot be applied to molecule language models due to the distinct characteristics of SMILES-based molecular generation. Therefore, there are no established baselines for this setting, as explicitly discussed in Section 5.1 (Experimental Settings). Instead, we compare our method against widely adopted decoding strategies, including greedy, top-k, top-p, and standard beam decoding in the original Table 5, across eight molecule language models, demonstrating the consistent effectiveness of our approach.
Response to Reviewer 7qG3 (Part 3)
In response to Minor Weakness 1
We thank the reviewer for pointing out the missing references. We will carefully review the resources highlighted by the reviewer, as well as the primary toxicity datasets used in our study, and provide the corresponding bibliographic references or official URLs in the revised version. The references we will add are listed below.
Resources to be cited. The Hazardous Chemical Information System of Safe Work Australia will be cited as [1]. The OECD QSAR Toolbox will be cited as [2]. The skin-irritation records will cite the REACH Regulation itself [3] together with the ECHA registered-substances database from which they were retrieved [4]. The Carcinogenic Potency Database will be cited as [5], and the ISS Carcinogens Data Bank / ISSTOX resource used for the strain-specific Ames annotations as [6]. The OECD eChemPortal, will be cited as [7]. Bibliographic citations for the primary toxicity datasets (hERGCentral, FDA DICTrank, UniTox, ICCVAM/LLNA, EPA ToxValDB, STopTox, and Tox21) were already provided in the original manuscript.
References to be added in the revised version.
[1] Safe Work Australia. Hazardous Chemical Information System (HCIS). https://hcis.safeworkaustralia.gov.au.
[2] OECD. The OECD QSAR Toolbox. https://qsartoolbox.org.
[3] European Parliament and Council of the European Union. Regulation (EC) No 1907/2006 of 18 December 2006 concerning the Registration, Evaluation, Authorisation and Restriction of Chemicals (REACH). Official Journal of the European Union, L 396, 30.12.2006, p. 1.
[4] European Chemicals Agency. Registered substances database. https://echa.europa.eu/information-on-chemicals/registered-substances.
[5] Gold, L. S., et al. Supplement to the Carcinogenic Potency Database (CPDB): results of animal bioassays published in the general literature through 1997 and by the National Toxicology Program in 1997–1998. Toxicological Sciences, 85(2):747–808, 2005. Database available at https://www.leadscope.com/CPDB.
[6] Benigni, R., et al. New perspectives in toxicological information management, and the role of ISSTOX databases in assessing chemical mutagenicity and carcinogenicity. Mutagenesis, 28(4):401–409, 2013. Resource available at https://www.iss.it/web/iss-en/databases.
[7] OECD. eChemPortal: The Global Portal to Information on Chemical Substances. https://www.echemportal.org.
In response to Minor Weakness 2
As explicitly stated in the caption of Table 4, the overall accuracy is weighted by the number of test samples for each toxicity endpoint. Nevertheless, we acknowledge that the phrase "over 90% average accuracy across all toxicity endpoints" in the Introduction may be ambiguous. To avoid any misunderstanding, we will revise it to “over 90% overall accuracy across all toxicity endpoints (weighted by the number of test samples per endpoint)” in the revised version.
In response to Minor Weakness 3
We acknowledge that the phrase "requiring only one forward pass in the target model" may be imprecise. Our intended meaning is that the proposed method performs a single autoregressive decoding procedure without requiring additional target-model rollouts for toxicity evaluation. We recognize that, as with standard beam search, multiple forward computations are involved in token generation. To avoid ambiguity, we will revise the statement in the Conclusion to "without requiring additional target-model rollouts" in the revised version.
In response to Question
As described in Section 3.2 (Toxicity Assessment), we preserve the original train/validation/test splits of existing toxicity datasets whenever available. For datasets without predefined splits, as well as newly collected samples, we incorporate them exclusively into the training set. For duplicate molecules with conflicting toxicity labels, we conservatively assign the toxic label and remove duplicate entries. Finally, we remove training molecules that overlap with the validation or test sets to prevent exact-molecule leakage.
We clarify that scaffold- or similarity-based splits were not employed, and closely related molecular analogs were not explicitly excluded across splits. Our primary objective is to systematically investigate the red-teaming vulnerabilities of molecule language models, rather than develop toxicity prediction models with optimized out-of-distribution generalization. Importantly, both our attack method and all baseline decoding strategies are evaluated using the same toxicity evaluators, ensuring a fair and consistent comparison of their relative effectiveness. Nevertheless, we recognize the value of evaluating toxicity predictors under more stringent structural splits and leave this direction for future work. Upon acceptance, we will publicly release our curated toxicity benchmark to facilitate further research and the development of more robust toxicity evaluators.
Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts to address my concerns.
The rebuttal has addressed several of my original comments:
- Surrogate evaluation: The held-out prediction results provide good evidence that the surrogate can approximate the rollout-based scores.
- Ablation studies: The random-scoring and prefix-only experiments help clarify the contribution of the learned surrogate and prompt conditioning.
However, I still have a few remaining concerns and questions:
-
Structural-similarity prior: I agree that a guidance signal does not need to be a strong classifier. However, statistical significance does not necessarily mean strong discrimination, and the method still shows inconsistent improvements for some endpoints. I think this limitation should be acknowledged more clearly.
-
Matched-compute comparison: The authors refer to Table 5, but my original concern was about matched-compute beam search, not standard beam search. Could the authors clarify whether the computational budgets are comparable?
-
Competing baselines: I understand that there are no established red-teaming baselines for this specific setting. However, comparisons with alternative guided-decoding approaches would still help establish the effectiveness of the proposed method.
-
Toxicity evaluator reliability: Thank you for clarifying the dataset splits. My remaining concern is that the attack favors molecules structurally similar to known toxic compounds, while the evaluators were not tested under scaffold- or similarity-based splits. Using the same evaluators across all methods ensures consistency, but does not rule out structural bias in the reported toxicity rates. Could the authors provide further evidence or discuss this limitation more explicitly?
-
Code availability: I could not find a code repository in the manuscript or rebuttal. Have the authors released their implementation or provided an anonymous repository for reviewers? I would like to examine the code before finalizing my assessment.
Overall, the rebuttal has strengthened the paper and resolved several of my concerns. I will also examine the released code before deciding whether to update my rating.
Response to Reviewer 7qG3 (Part 1)
We sincerely thank the reviewer for the constructive feedback, as well as for recognizing the clear motivation, computational efficiency, and empirical effectiveness of our proposed surrogate-guided decoding framework. Below, we provide point-by-point responses to the reviewer's comments and clarify the corresponding aspects of our work.
In response to Weakness 1
While ROC-AUC varies across toxicity endpoints, the similarity distributions of toxic and non-toxic molecules exhibit statistically significant differences across all eight endpoints (all p≤1.0×10−4). Notably, even for Stress Response (ROC-AUC = 0.653) and Micronucleus Induction (ROC-AUC = 0.700), the differences remain highly significant (p=1.7×10−50 and p=1.0×10−4, respectively). These findings demonstrate that structural similarity provides a statistically informative signal across all endpoints. Importantly, our method leverages structural similarity as a guidance signal rather than a highly accurate toxicity classifier, without requiring strong discrimination between toxic and non-toxic molecules. This is further supported by our experimental results, which show that surrogate-guided decoding can improve toxicity rates even for endpoints with relatively low ROC-AUCs, demonstrating the practical utility of structural similarity as a decoding prior.
In response to Weakness 2
We conducted additional experiments and provide detailed responses to each point below.
Surrogate’s held-out prediction quality. To directly assess the predictive reliability of our surrogate scorer, we conducted additional experiments on 1,000 held-out samples, comparing its predictions with rollout-derived reference scores. We report the mean absolute error (MAE) and Pearson correlation across all eight toxicity endpoints:
As shown above, our surrogate scorer consistently achieves low prediction errors (MAE: 0.03–0.05) and strong correlations (Pearson: 0.88–0.91) with the rollout-derived reference scores across all eight endpoints. These results demonstrate that the learned surrogate reliably approximates the completion-based scoring signal, validating its effectiveness in estimating the toxicity-related potential of partial SMILES without requiring additional target-model rollouts during decoding.
Random scoring. To further isolate the contribution of the learned surrogate, we conducted additional experiments by replacing surrogate-guided scoring with random scoring during decoding. We evaluated this control on ChemAligner-T5 and ChemAligner-T5-Pro, comparing it against greedy decoding and our proposed method across all eight toxicity endpoints. The results are summarized below.
ChemAligner-T5
ChemAligner-T5-Pro
As shown above, our surrogate-guided decoding achieves substantially higher average toxicity rates than random scoring. These results demonstrate that the effectiveness of our method stems from the learned surrogate guidance rather than random candidate selection during beam decoding, further validating the contribution of our surrogate scorer.
Rollout-based oracle guidance. The primary motivation for our learned surrogate is to avoid the substantial computational overhead of repeated target-model rollouts during decoding. While surrogate-guided generation typically completes within a few seconds per molecule, rollout-based oracle guidance can require several hours per molecule due to repeated candidate completions at each guidance step. This substantial computational cost makes evaluation at scale impractical, particularly given the limited time and computational resources available during the rebuttal period. Nevertheless, our surrogate prediction results demonstrate strong agreement with rollout-derived reference scores, supporting its effectiveness as a computationally efficient approximation.