- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces IF-DrugBench, a benchmark of 2,400 natural-language tasks for localized molecular editing. Tasks combine three axes: the requested goal, comprising property changes or chemical operations; the edit location, comprising substituents or linkers; and reference information, comprising no reference, explicit donor-motif transfer, or implicit reference guidance. Tasks are constructed from ChEMBL molecular pairs and provided in aligned SMILES and molecular-image representations. A hidden, experimentally reported endpoint must pass the assigned evaluators, providing a feasible solution for each retained task.
Evaluation checks molecular validity, nonidentity to the starting molecule, localization and anchor preservation, operation fidelity, explicit reference transfer when requested, and RDKit descriptor directions. The authors evaluate eleven text-input and seven image-conditioned model settings on a balanced 480-task subset. The best task pass rates are 48.1% and 51.5%, respectively. Additional analyses report criterion-family failures and paired representation outcomes. The validation concerns instruction compliance and computed molecular descriptors; it does not establish experimental activity or developability of generated molecules.
Critical strengths
- Exact preservation of protected context and receiver anchors, combined with operation and donor-transfer constraints, captures failures that molecular validity or overall similarity can overlook. Appendix D explicitly addresses stereochemistry, symmetric mappings, and donor boundary topology. Requiring the hidden endpoint to pass all assigned checks also reduces the risk of unsatisfiable tasks, while allowing other outputs that satisfy the instruction.
The contribution is most compelling as an extension of existing instruction-specific evaluation: S²-Bench already checks functional-group modifications and property objectives automatically, whereas IF-DrugBench combines these kinds of requirements with explicit localization and reference-transfer constraints. This more precise assessment of whether an output obeys the requested edit is a meaningful reason to consider acceptance.
- The same 480 task identities are used across representations for the shared models, and Tables 6–11 report criterion-family scores, instruction-axis marginals, paired outcomes, and counts of failed checks. These analyses substantiate an evaluation gap: for example, Gemini’s text-input validity is 98.8%, while its complete task success is only 48.1% (Table 6). The paired outcomes also reveal substantial model-dependent sensitivity to representation. This combination of a reusable task collection and diagnostic reporting could help researchers identify which constraints their molecular editing systems fail to satisfy.
Critical weaknesses
- Section 3.4 mentions LLM-assisted checking and manual inspection, and Appendix D.5 confirms that all 2,400 retained hidden endpoints pass. However, the manuscript does not report the manual-audit sample size, adjudication protocol, agreement, or evaluator false-acceptance and false-rejection rates on generated alternatives. Checking the construction endpoints establishes satisfiability under the implemented predicates; it does not establish that those predicates faithfully distinguish instruction-compliant alternatives from violations.
There is also a concrete semantic ambiguity. Appendix D.3 accepts “linker shortening” when either the protected-anchor path becomes shorter or the editable-region heavy-atom count decreases. These conditions are not equivalent when atoms outside the connecting path can be removed. The endpoint displayed for Case 2,193 in Appendix E.4 appears to reduce atom count while preserving the corresponding connecting-path lengths. This may be consistent with the implemented evaluator, but the complete instruction and annotations are needed to establish whether it matches what the model was asked to do. Since these evaluators define benchmark correctness, independent validation and clarification of this operation directly affect my assessment.
- The experimental protocol is underspecified, and some interpretations go beyond the comparisons reported. Section 4 identifies the models and the number of tasks per cell, but the manuscript does not specify the decoding and reasoning settings, output-token limits, retry/repeat policy, or selection procedure for the 480 evaluated instances. The exact modality comparison is also unclear: some passages describe replacing molecular strings with diagrams, while others describe adding diagrams. Canonicalization is emphasized specifically for image-conditioned outputs, whereas Table 17 describes normalization as a general validity rule. This does not demonstrate unequal treatment, but it leaves important uncertainty about what differed between conditions and how the results can be reproduced.
The stronger explanations of model behavior also need qualification. R2 deliberately has no reference-use criterion, so success cannot establish whether a model used the reference, and failure cannot establish that it failed to extract the reference’s design rationale. Likewise, the text/image comparison does not separately identify image-recognition and edit-execution errors. The small Gemini improvement is particularly uncertain: Table 10 reports 106 image-only versus 90 text-only successes. An exact paired test calculated from these counts gives approximately (p=0.28), even before accounting for dependence from reused transformations. The observed scores are informative, but they provide limited evidence for a systematic benefit from visual highlighting.
Questions for authors
-
What was the manual-inspection protocol in Section 3.4, including sample size, examiner expertise, and observed disagreements or corrections? Is there an independently adjudicated sample of model outputs measuring false accepts and false rejects, including valid alternatives to the hidden endpoint?
-
For Case 2,193, please provide the complete model-facing instruction and annotated receiver anchors. Does the instruction explicitly allow “shortening” through a reduction in editable atoms without a shorter anchor-to-anchor path? How many of the 379 linker-shortening tasks rely only on the atom-count condition?
-
Please specify the model identifiers, complete prompt wrappers, decoding/reasoning settings, output limits, and retry/repeat policy, together with how the 480 tasks were selected. Does the image condition replace or retain the SMILES, and are identical output extraction and canonicalization rules applied to both representations?
-
What evidence distinguishes using an R2 reference from independently satisfying the remaining objectives? Is there a matched comparison with the reference removed from otherwise identical tasks? Such evidence would materially strengthen the conclusions about implicit reference guidance; without it, those conclusions should be limited to performance when a reference is present.
8 comments
Q2 and W1: For Case 2,193, does the instruction allow atom-count reduction without a shorter anchor path, and how many linker-shortening tasks rely only on atom count?
We thank the reviewer for identifying this ambiguity. The complete model-facing instruction for Case 2,193 explicitly allowed the model to “shorten the path between the two anchor points by reducing its path length or the number of heavy atoms in the linker.” Therefore, the case was consistent with the literal instruction in the task and the implemented evaluator: the hidden target reduces the editable linker from two heavy atoms to one, although its anchor-to-anchor path length remains unchanged. Nevertheless, we agree that our terminology was imprecise. A reduction in linker-region heavy atoms is not necessarily equivalent to strict linker shortening. We have therefore revised the term usage.
We confirmed through a task-level audit that all 379 tasks originally categorized as linker shortening explicitly informed the model that the operation could be satisfied either by reducing the anchor-to-anchor path length or by reducing the number of heavy atoms in the editable linker region. The evaluator implements exactly the same disjunctive condition. The 379 tasks decompose as follows:
| Hidden-target change satisfying the original disjunction | Tasks | Percentage |
|---|---|---|
| Anchor-to-anchor path decreases; linker heavy-atom count does not | 12 | 3.2% |
| Linker heavy-atom count decreases; anchor-to-anchor path does not | 246 | 64.9% |
| Both quantities decrease | 121 | 31.9% |
| Total | 379 | 100.0% |
Case 2,193 belongs to the second group: its anchor-to-anchor path-length change is 0, while its editable-region heavy-atom change is -1. We therefore rename the broad operation linker shortening or simplification and reserve strict linker shortening for the 133 tasks in which the protected-anchor path actually becomes shorter. This change clarifies the semantics without retroactively changing the model-facing requirement or its evaluator.
Q3 and W2: Please specify the exact inference protocol and clarify whether 2D replaces or retains SMILES and whether normalization is identical.
The 1D and 2D conditions use the same underlying task, semantic instruction, output contract, and evaluator. In 1D, the user message contains canonical SMILES with a SMILES-aligned ASCII annotation of the receiver region and, when applicable, a reference SMILES. In 2D, these molecular strings and ASCII annotations are replaced by the corresponding annotated molecular images; the operation, preservation constraints, property objectives, and output instruction remain unchanged. The 2D condition therefore replaces rather than supplements model-visible SMILES.
| Input component | 1D | 2D |
|---|---|---|
| Starting structure | Canonical SMILES plus ASCII region annotation | Annotated x0 image |
| Reference structure | Optional reference SMILES; R1 donor annotation when applicable | Optional annotated reference image |
| Hidden feasible target | Not shown | Not shown |
| Operation and property text | Same semantic instruction | Same semantic instruction |
| Expected output | Exactly one canonical SMILES | Exactly one canonical SMILES |
| Output validation | Same deterministic extraction and RDKit canonicalization | Same deterministic extraction and RDKit canonicalization |
All tasks were frozen before evaluation. Each case was sent in an independent request without conversation history, tools, hidden targets, criteria, or evaluator information, and no task-specific prompt engineering was used. The revised experimental section will include the exact provider/checkpoint identifiers and the frozen decoding, reasoning, output-token, repeat, and retry settings listed in our response to First Report Q2; these fields must be copied verbatim from the released run configuration.
Both modalities use the same output-extraction and canonicalization pipeline. Evaluation considers only the final generated molecule. Then, the required answer is one short SMILES string. When a small chemistry-specific model wrapped an otherwise identifiable SMILES in presentation text, a deterministic parser removed only code fences, quotation marks, or an explicit answer prefix and extracted the unique RDKit-parseable single-component candidate. It never changed atoms or bonds, used evaluator feedback, or requested a corrected molecule. If no unique valid candidate was identifiable, the response was scored as invalid. The extracted candidate was sanitized and converted to RDKit canonical isomeric SMILES before evaluation.
The 480-task subset is a deterministic, stratified 20% sample of the full benchmark, with exactly 40 tasks from each of the 12 taxonomy combinations. It contains 291 unique starting molecules, 303 unique molecular transformations, and all 64 biological targets represented in the full benchmark. Distributional agreement is summarized below:
| Characteristic | Test-vs-full discrepancy |
|---|---|
| Taxonomy composition | 0.0% total-variation distance |
| Number of property objectives | 0.8% total-variation distance |
| Property directions | 0.2% total-variation distance |
| Editable-region size | 0.5% total-variation distance |
| Chemical-operation subtype | 1.2% total-variation distance |
| Property identity | 1.6% total-variation distance |
| Largest difference for any biological target | 0.79 percentage points |
Rare targets were deliberately retained, which explains the small residual differences in target frequencies.
Q1 and W1: What was the manual-inspection protocol, and is there an independently adjudicated evaluator-validation sample?
We initially constructed 21,504 candidate tasks. These candidates first underwent deterministic chemical validation using RDKit 2024.09.4; this stage did not use an LLM. In this way, we removed cases with disconnected or mapping-ambiguous regions, incorrect S1/S2 assignments, repeated molecular transformations, invalid reference transfers, and other violations of hard chemical or structural constraints detectable through deterministic RDKit-based checks. We also evaluated x* through our defined evaluation tools using every criterion assigned to the task and removed the task if x* failed any criterion. Candidates passing these checks were then audited by GPT-5.6 Sol, and only tasks assigned PASS were retained. The combined deterministic and GPT-assisted filtering yielded 2,883 candidates.
Specifically, GPT-5.6 Sol screened the post-RDKit candidates for potential issues involving prompt–criterion alignment, evaluator correctness, chemical coherence, hidden-target feasibility, or reference usage. We used the following checker requirement:
The 2,883 tasks that passed the deterministic and GPT-assisted filters were divided among the four experts, with each expert manually reviewing approximately 720 tasks using a rubric similar to the GPT-based audit. In addition, the experts inspected the corresponding evaluator implementations to determine whether the code faithfully enforced the stated criteria. Expert-flagged tasks were discussed collectively. Through this adjudication process, the experts confirmed that 21 cases contained substantive inconsistencies among the model-facing prompt, criteria, evaluator implementation, and hidden feasible target; these cases were excluded. We subsequently selected 2,400 tasks from the 2,862 eligible candidates through stratified sampling, with 200 tasks from each taxonomy combination.
Our evaluation tools operate on stereochemistry-aware molecular graphs and can distinguish constitutional isomers, enantiomers, and cis–trans isomers by checking atom connectivity, bond order, formal charge, aromaticity, and encoded stereochemistry. They also account for symmetry-related atom mappings and verify stereochemistry at protected contexts and donor boundaries. Nevertheless, a small number of edge cases may remain difficult to evaluate automatically, particularly tautomeric or protonation-equivalent representations and rare boundary edits that change whether a stereocenter or double bond remains stereogenic. Such cases occurred infrequently in our benchmark and are therefore unlikely to materially affect the aggregate results.
To further validate evaluator reliability, we recruited two independent experts with four and five years of molecular-design experience, respectively. We conducted a score-blinded audit of a stratified random sample of 120 model outputs with two experts. The sample contained ten outputs per taxonomy category—five each from 1D and 2D—with randomized, approximately balanced model assignments.
The single false rejection resulted from an output for which the extraction pipeline could not identify a unique valid molecule. We count this conservatively as a pipeline-level false rejection even though the downstream chemical predicates were not the source of the disagreement. Within this stratified sample, the automated pipeline exhibited a low task-decision error rate.