- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MolDeTox, a benchmark for evaluating stepwise molecular detoxification using LLMs. The authors construct ToxicityCliff, containing 52,600 structurally similar toxic/non-toxic molecular pairs across 49 toxicity endpoints. Based on SAFE representations, they formulate three tasks: toxic fragment identification, non-toxic fragment generation, and final molecule generation. Experiments across multiple LLMs reveal difficulties in fragment editing and frequent input copying. Supervised fine-tuning improves reference-based performance and shows additional gains under an external toxicity-predictor-based evaluation. The main contribution is a benchmark providing intermediate supervision for molecular detoxification.
Critical strengths
- Fine-grained evaluation of molecular detoxification. The three-task formulation enables more detailed analysis than evaluating only the final molecule. It helps identify failures in fragment localization, replacement, and molecule reconstruction. The analysis of input copying is particularly useful, showing that high molecular validity or similarity does not necessarily indicate successful editing.
- Evidence of training utility. Fine-tuning substantially improves performance across the benchmark. The external ToxiEval evaluation further shows improvements in Safety (11.5% to 42.2%) and Success (4.8% to 16.5%) for the reported Qwen3-4B configurations. These results provide encouraging evidence that the training data can support molecular editing beyond exact reference recovery, although the external evaluation relies on predicted toxicity.
Critical weaknesses
- Validity and ambiguity of the benchmark targets. The toxicity-associated fragment labels are derived from structural differences between molecular pairs with opposite toxicity labels. However, these differences do not necessarily identify the fragments responsible for toxicity. This raises concerns about whether Task 1 evaluates toxicity-related structural understanding or simply recovery of reference-specific fragment differences. Furthermore, the same toxic molecule may have multiple non-toxic counterparts, whereas evaluation relies on matching a designated reference. Valid alternative edits may therefore be penalized, potentially affecting reported accuracy and the stepwise failure analysis. The paper would benefit from quantifying this ambiguity and evaluating against multiple valid references.
- Generalization and attribution of training improvements. The scaffold split is performed separately for each toxicity endpoint using the toxic molecule's scaffold. It remains unclear whether molecules or closely related structures overlap across training and test sets through different endpoints or paired counterparts. Moreover, the fine-tuning experiments do not fully isolate the contribution of intermediate supervision. All Tasks and Only Task 3 use the same number of epochs but may differ in training budget. In Table 2, All Tasks performs worse than Only Task 3 on multi-fragment generation without CoT (2.29% versus 3.39%). The external evaluation also omits Task 3-only variants, making it difficult to distinguish the benefit of stepwise supervision from additional training and CoT.
Questions for authors
- Benchmark validity: How frequently do toxic molecules have multiple non-toxic references for the same endpoint? Would multi-reference evaluation change the model rankings or failure analysis? What evidence supports the reliability of the automatically derived toxicity-associated fragment labels?
- Data independence: What molecular and scaffold overlap remains between training and test sets when considering all endpoints and both sides of each molecular pair?
- Training effectiveness: Does joint training on all three tasks still outperform Task 3-only training under matched training budgets? Could the authors report external ToxiEval results for the Task 3-only variants to better establish the contribution of intermediate supervision?
5 comments
Response to Reviewer Zduo (W2-1 & Q2)
W2-1 & Q2. Train–test overlap and generalization
Thank you for requesting this analysis. The original scaffold split was performed separately for each toxicity endpoint using the toxic input molecule. Consequently, the same molecule can appear in training for one endpoint and in testing for another. Prior studies distinguish evaluation on previously observed compounds from evaluation on new compounds [1] and discuss cross-task overlap under task-specific splits [2, 3]. We therefore audited all endpoints jointly, comparing both molecules in each test pair against both sides of every training pair.
We identified three mutually exclusive groups:
| Group | Definition | Test pairs | Ratio |
|---|---|---|---|
| G1: Seen Pair | The same toxic–non-toxic pair appears in training under another endpoint | 422 | 21.4% |
| G2: Unseen Pair & Seen Molecule | The pair is unseen, but at least one constituent molecule appears in training | 451 | 22.8% |
| G3: Both Molecules Unseen | Neither molecule appears in training | 1,103 | 55.8% |
To assess how training exposure relates to performance, we report Task 3 accuracy separately for each group.
Group-wise Acc.
| Model | G1 Single Acc. (%) | G1 Multi Acc. (%) | G2 Single Acc. (%) | G2 Multi Acc. (%) | G3 Single Acc. (%) | G3 Multi Acc. (%) | All Single Acc. (%) | All Multi Acc. (%) |
|---|---|---|---|---|---|---|---|---|
| GPT-5.2 SAFE Gen. | 5.32 | 0.00 | 1.90 | 0.13 | 3.85 | 0.25 | 3.78 | 0.16 |
| Qwen3-4B-Inst. | 0.00 | 0.00 | 0.00 | 0.00 | 0.06 | 0.00 | 0.04 | 0.00 |
| + Only Task 3 SFT | 36.74 | 9.80 | 11.11 | 3.05 | 5.40 | 0.57 | 12.65 | 3.39 |
| + Only Task 3 w/ CoT SFT | 34.77 | 3.13 | 10.36 | 1.65 | 5.75 | 0.25 | 12.34 | 1.29 |
| + All Tasks SFT | 52.69 | 6.94 | 19.77 | 2.16 | 8.10 | 0.19 | 19.09 | 2.29 |
| + All Tasks w/ CoT SFT | 62.72 | 31.84 | 14.12 | 5.60 | 7.41 | 1.27 | 19.58 | 9.62 |
All SFT configurations achieve their highest accuracy on G1, where scores may reflect reuse of previously observed edits. We therefore base the revised main evaluation on G3, excluding test pairs with either molecule present in fine-tuning. We will provide the original full-test results and the group-wise breakdown in the appendix.
We further report Acc. and MR-Acc on G3 to assess performance when neither test molecule appeared in fine-tuning.
G3 (Both Molecules Unseen) Acc. and MR-Acc.
| Model | Single Acc. (%) | Single MR-Acc (%) | Multi Acc. (%) | Multi MR-Acc (%) |
|---|---|---|---|---|
| GPT-5.2 SAFE Gen. | 3.85 | 5.86 | 0.25 | 1.14 |
| Qwen3-4B-Inst. | 0.06 | 0.06 | 0.00 | 0.00 |
| + Only Task 3 SFT | 5.40 | 6.90 | 0.57 | 2.98 |
| + Only Task 3 w/ CoT SFT | 5.75 | 7.01 | 0.25 | 0.82 |
| + All Tasks SFT | 8.10 | 9.31 | 0.19 | 2.03 |
| + All Tasks w/ CoT SFT | 7.41 | 9.25 | 1.27 | 2.85 |
The G3 results show that the training benefit persists after exact molecular overlap is removed. For single-fragment editing, All Tasks exceeds Only Task 3 on both Acc. and MR-Acc, and both outperform prompted Qwen3-4B-Instruct. For multi-fragment editing, All Tasks w/ CoT achieves the highest Acc. among the compared models, while Only Task 3 achieves the highest MR-Acc. These results support the utility of MolDeTox training data on unseen molecules, while showing that the preferred training configuration depends on the editing setting and metric. We address intermediate supervision under matched training conditions separately below.
We also assessed scaffold overlap using Bemis–Murcko scaffolds computed with RDKit MurckoScaffold, comparing both molecules in each test pair against both sides of all training pairs across endpoints. At least one constituent scaffold appeared in training for 997 of the 1,976 original test pairs (50.5%), compared with 139 of the 1,103 G3 pairs (12.6%). Under a generic-framework definition that abstracts atom types and bond orders, the corresponding overlap rates were 67.4% and 43.0%. Restricting evaluation to G3 therefore removes exact molecular overlap and substantially reduces scaffold overlap under both definitions. We will report these statistics and describe G3 as a molecule-disjoint evaluation subset, with its residual scaffold overlap explicitly documented.
[1] Walter et al. “Analysis of the benefits of imputation models over traditional QSAR models for toxicity prediction.” Journal of Cheminformatics, 2022.
[2] Simm et al. “Splitting chemical structure data sets for federated privacy-preserving machine learning.” Journal of Cheminformatics, 2021.
[3] Mun and Fazli. “CheMLT-F: multitask learning in biochemistry through transformer fusion.” Journal of Cheminformatics, 2026.
Response to Reviewer Zduo (W2-2 & Q3)
W2-2 & Q3. Contribution of intermediate supervision under matched training data
Following the reviewer’s suggestion, we trained an All Tasks variant using the same total number of examples as Only Task 3 and the same three epochs. On G3, the matched All Tasks model provides modest improvements in single-fragment Acc. and MR-Acc, but performs worse on both metrics for multi-fragment editing. Thus, allocating part of a fixed example budget to auxiliary tasks yields mixed benefits across editing settings.
| Model | Single Acc. (%) | Single MR-Acc (%) | Multi Acc. (%) | Multi MR-Acc (%) |
|---|---|---|---|---|
| Only Task 3 SFT | 5.40 | 6.90 | 0.57 | 2.98 |
| All Tasks SFT (matched examples, 3 epochs) | 6.21 | 7.53 | 0.25 | 1.33 |
A complementary comparison examines how intermediate targets are incorporated into Task 3 itself. All Tasks jointly trains the three tasks, while All Tasks w/ CoT additionally supervises fragment identification and replacement within the Task 3 output sequence before final molecule generation. On G3, this structured sequence improves both multi-fragment Acc. and MR-Acc, although single-fragment performance decreases slightly.
| Model | Single Acc. (%) | Single MR-Acc (%) | Multi Acc. (%) | Multi MR-Acc (%) |
|---|---|---|---|---|
| All Tasks SFT | 8.10 | 9.31 | 0.19 | 2.03 |
| All Tasks w/ CoT SFT | 7.41 | 9.25 | 1.27 | 2.85 |
These results distinguish auxiliary-task joint training from supervision on an explicit editing sequence. The matched-example comparison shows a limited, setting-dependent benefit from auxiliary tasks, while the sequence-based comparison supports the usefulness of generating intermediate editing decisions for multi-fragment reference recovery. We will qualify our claims accordingly and report training examples, optimizer steps, and processed tokens. The sequence-based comparison uses longer targets, so its training-token budget must also be considered when interpreting the gains.
Response to Reviewer Zduo(W1 & Q1)
W1 & Q1. Target validity and reference ambiguity
Thank you for raising these concerns. We agree that structural differences between a toxic/non-toxic pair alone cannot establish that a fragment causes toxicity. Our labels identify editing locations associated with an observed toxicity difference for the same endpoint, as acknowledged in Section 8 (L480–482).
Existing evidence supporting fragment reliability. Appendix B already compares the extracted fragments with endpoint-specific structural alerts from ToxAlerts, a curated resource based on expert knowledge and published literature. Figures 5 and 6 examine the 20 most frequent toxicity-associated fragments for Ames Mutagenicity and Skin Reaction. Of these, 11 and 12 fragments, respectively, are marked as matching documented alerts, with two additional fragments in each endpoint showing ECFP4 Tanimoto similarity of at least 0.8.
This agreement with independently curated toxicity knowledge supports the chemical relevance of the extracted targets. Task 1 therefore evaluates localization of observed editing differences with supporting evidence of toxicity relevance. The analysis covers frequent fragments in two aligned endpoints, and establishing the causal contribution of each fragment would require further evidence. We will bring this existing validation into the main discussion and clarify the intended interpretation of the labels.
Reference ambiguity in the final test set. For each of the 1,976 final test combinations, defined by dataset, endpoint, and canonical toxic-input structure, we counted distinct non-toxic counterparts in the original pair corpus before reference selection. None had multiple eligible counterparts in that corpus (0/1,976, 0.00%). During dataset construction, combinations with multiple counterparts were assigned entirely to training. Thus, the final test targets are unambiguous within the curated pair corpus.
Additional valid edits may exist beyond the curated pair corpus. We therefore added multi-reference exact-match accuracy (MR-Acc) for Tasks 2 and 3, retaining the designated reference and additionally crediting eligible structures with public non-toxic labels for the same endpoint. Experimental results for new metric can be found in our separate official comment, [To All Reviewers]
Effect on model rankings. Multi-reference evaluation changes some comparisons. In multi-fragment Task 2, All Tasks SFT and GPT-5.2 with four-shot prompting differed by only 0.02 percentage points under Acc., while four-shot prompting performs better under MR-Acc. In Task 3, All Tasks w/ CoT retains the highest Acc. and MR-Acc in both fragment settings. Reporting both metrics therefore exposes reference sensitivity while showing which performance conclusions persist after publicly supported alternatives are accepted.
Effect on the step-wise failure analysis. We re-scored the same three inference runs used in Figure 3, applying MR-Acc at Steps 2 and 3 while retaining the designated fragment targets for Step 1. At Step 2, we retained original fragment-level exact matches and additionally credited eligible reconstructed molecules under MR-Acc.
Accepting publicly supported alternatives increases complete-path success by 1.08 percentage points, recovering approximately 21 additional trajectories per run. Replacement remains a substantial source of failure after successful localization, and no final reconstruction failures are observed after successful Steps 1 and 2. The downstream failure pattern therefore persists under the expanded evaluation.
Step 1 remains evaluated against the designated editing location. An average of 17.7 predictions per run receive final-answer MR-Acc credit while failing this Step 1 criterion. This quantifies a distinction raised by the reviewer: a valid alternative output may follow a different editing pathway. We therefore distinguish final-answer MR-Acc from complete-path success, which additionally requires agreement with the designated Step 1 target.