acceptodds
Under review as a conference paper at ICLR 2027

What, Where, and How to Edit: Benchmarking Instruction Following for Localized Molecular Editing.

Abstract

Large language models are increasingly used to generate and modify drug-like molecules from natural-language instructions, yet existing benchmarks rarely test whether a model follows compositional editing requests precisely. We introduce IF-DrugBench, a benchmark of 2,400 localized small-molecule editing tasks organized along three complementary axes: what to achieve, where to act, and how to use reference information. Each task is released in aligned one-dimensional SMILES and two-dimensional molecular-image forms, grounded in an experimentally reported molecular transformation, and paired one-to-one with machine-readable criteria and automated evaluation tools. The benchmark evaluates molecular validity, localization of substituent or linker edits, chemical-operation fidelity, explicit reference-motif transfer when requested, and property directions relative to the starting molecule. On a balanced 480-task evaluation subset, the evaluated models achieve at most 48.1% task pass rate (TPR) with text input and 51.5% TPR with image-conditioned input. These results show that current models still have substantial room for improvement in reliably following compositional molecular editing instructions and satisfying multiple constraints simultaneously. The benchmark and evaluation tools can be checked at https://anonymous.4open.science/r/ICLR-DrugBench-6D27.

Primary area:
datasets and benchmarks
Keywords:
drug design, large language model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6Mm7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces IF-DrugBench, a benchmark of 2,400 natural-language tasks for localized molecular editing. Tasks combine three axes: the requested goal, comprising property changes or chemical operations; the edit location, comprising substituents or linkers; and reference information, comprising no reference, explicit donor-motif transfer, or implicit reference guidance. Tasks are constructed from ChEMBL molecular pairs and provided in aligned SMILES and molecular-image representations. A hidden, experimentally reported endpoint must pass the assigned evaluators, providing a feasible solution for each retained task.

Evaluation checks molecular validity, nonidentity to the starting molecule, localization and anchor preservation, operation fidelity, explicit reference transfer when requested, and RDKit descriptor directions. The authors evaluate eleven text-input and seven image-conditioned model settings on a balanced 480-task subset. The best task pass rates are 48.1% and 51.5%, respectively. Additional analyses report criterion-family failures and paired representation outcomes. The validation concerns instruction compliance and computed molecular descriptors; it does not establish experimental activity or developability of generated molecules.

Critical strengths
  1. Exact preservation of protected context and receiver anchors, combined with operation and donor-transfer constraints, captures failures that molecular validity or overall similarity can overlook. Appendix D explicitly addresses stereochemistry, symmetric mappings, and donor boundary topology. Requiring the hidden endpoint to pass all assigned checks also reduces the risk of unsatisfiable tasks, while allowing other outputs that satisfy the instruction.

The contribution is most compelling as an extension of existing instruction-specific evaluation: S²-Bench already checks functional-group modifications and property objectives automatically, whereas IF-DrugBench combines these kinds of requirements with explicit localization and reference-transfer constraints. This more precise assessment of whether an output obeys the requested edit is a meaningful reason to consider acceptance.

  1. The same 480 task identities are used across representations for the shared models, and Tables 6–11 report criterion-family scores, instruction-axis marginals, paired outcomes, and counts of failed checks. These analyses substantiate an evaluation gap: for example, Gemini’s text-input validity is 98.8%, while its complete task success is only 48.1% (Table 6). The paired outcomes also reveal substantial model-dependent sensitivity to representation. This combination of a reusable task collection and diagnostic reporting could help researchers identify which constraints their molecular editing systems fail to satisfy.
Critical weaknesses
  1. Section 3.4 mentions LLM-assisted checking and manual inspection, and Appendix D.5 confirms that all 2,400 retained hidden endpoints pass. However, the manuscript does not report the manual-audit sample size, adjudication protocol, agreement, or evaluator false-acceptance and false-rejection rates on generated alternatives. Checking the construction endpoints establishes satisfiability under the implemented predicates; it does not establish that those predicates faithfully distinguish instruction-compliant alternatives from violations.

There is also a concrete semantic ambiguity. Appendix D.3 accepts “linker shortening” when either the protected-anchor path becomes shorter or the editable-region heavy-atom count decreases. These conditions are not equivalent when atoms outside the connecting path can be removed. The endpoint displayed for Case 2,193 in Appendix E.4 appears to reduce atom count while preserving the corresponding connecting-path lengths. This may be consistent with the implemented evaluator, but the complete instruction and annotations are needed to establish whether it matches what the model was asked to do. Since these evaluators define benchmark correctness, independent validation and clarification of this operation directly affect my assessment.

  1. The experimental protocol is underspecified, and some interpretations go beyond the comparisons reported. Section 4 identifies the models and the number of tasks per cell, but the manuscript does not specify the decoding and reasoning settings, output-token limits, retry/repeat policy, or selection procedure for the 480 evaluated instances. The exact modality comparison is also unclear: some passages describe replacing molecular strings with diagrams, while others describe adding diagrams. Canonicalization is emphasized specifically for image-conditioned outputs, whereas Table 17 describes normalization as a general validity rule. This does not demonstrate unequal treatment, but it leaves important uncertainty about what differed between conditions and how the results can be reproduced.

The stronger explanations of model behavior also need qualification. R2 deliberately has no reference-use criterion, so success cannot establish whether a model used the reference, and failure cannot establish that it failed to extract the reference’s design rationale. Likewise, the text/image comparison does not separately identify image-recognition and edit-execution errors. The small Gemini improvement is particularly uncertain: Table 10 reports 106 image-only versus 90 text-only successes. An exact paired test calculated from these counts gives approximately (p=0.28), even before accounting for dependence from reused transformations. The observed scores are informative, but they provide limited evidence for a systematic benefit from visual highlighting.

Questions for authors
  1. What was the manual-inspection protocol in Section 3.4, including sample size, examiner expertise, and observed disagreements or corrections? Is there an independently adjudicated sample of model outputs measuring false accepts and false rejects, including valid alternatives to the hidden endpoint?

  2. For Case 2,193, please provide the complete model-facing instruction and annotated receiver anchors. Does the instruction explicitly allow “shortening” through a reduction in editable atoms without a shorter anchor-to-anchor path? How many of the 379 linker-shortening tasks rely only on the atom-count condition?

  3. Please specify the model identifiers, complete prompt wrappers, decoding/reasoning settings, output limits, and retry/repeat policy, together with how the 480 tasks were selected. Does the image condition replace or retain the SMILES, and are identical output extraction and canonicalization rules applied to both representations?

  4. What evidence distinguishes using an R2 reference from independently satisfying the remaining objectives? Is there a matched comparison with the reference removed from otherwise identical tasks? Such evidence would materially strengthen the conclusions about implicit reference guidance; without it, those conclusions should be limited to performance when a reference is present.

8 comments
AuthorsOpenReview ↗

Q1 and W1: What was the manual-inspection protocol, and is there an independently adjudicated evaluator-validation sample?

We initially constructed 21,504 candidate tasks. These candidates first underwent deterministic chemical validation using RDKit 2024.09.4; this stage did not use an LLM. In this way, we removed cases with disconnected or mapping-ambiguous regions, incorrect S1/S2 assignments, repeated molecular transformations, invalid reference transfers, and other violations of hard chemical or structural constraints detectable through deterministic RDKit-based checks. We also evaluated x* through our defined evaluation tools using every criterion assigned to the task and removed the task if x* failed any criterion. Candidates passing these checks were then audited by GPT-5.6 Sol, and only tasks assigned PASS were retained. The combined deterministic and GPT-assisted filtering yielded 2,883 candidates.

Specifically, GPT-5.6 Sol screened the post-RDKit candidates for potential issues involving prompt–criterion alignment, evaluator correctness, chemical coherence, hidden-target feasibility, or reference usage. We used the following checker requirement:

You are auditing a molecular-editing benchmark task.

Requirements

  1. Extract every requirement stated in the model-facing instruction, including property objectives, editable-region constraints, chemical operations, preservation constraints, and reference usage.
  2. Check whether each requirement has a corresponding machine-readable criterion and whether any criterion introduces an unstated requirement.
  3. Check whether each assigned evaluator directly and correctly tests its criterion.
  4. Check whether the prompt, criteria, evaluators, and hidden target agree on the requested edit region, property directions, chemical operation, and use of reference information.

Output

Return structured JSON containing the extracted requirements, prompt-criterion-evaluator mappings, identified issues, explanations, and an overall decision of PASS, REVISE, or REMOVE.

The 2,883 tasks that passed the deterministic and GPT-assisted filters were divided among the four experts, with each expert manually reviewing approximately 720 tasks using a rubric similar to the GPT-based audit. In addition, the experts inspected the corresponding evaluator implementations to determine whether the code faithfully enforced the stated criteria. Expert-flagged tasks were discussed collectively. Through this adjudication process, the experts confirmed that 21 cases contained substantive inconsistencies among the model-facing prompt, criteria, evaluator implementation, and hidden feasible target; these cases were excluded. We subsequently selected 2,400 tasks from the 2,862 eligible candidates through stratified sampling, with 200 tasks from each taxonomy combination.

Our evaluation tools operate on stereochemistry-aware molecular graphs and can distinguish constitutional isomers, enantiomers, and cis–trans isomers by checking atom connectivity, bond order, formal charge, aromaticity, and encoded stereochemistry. They also account for symmetry-related atom mappings and verify stereochemistry at protected contexts and donor boundaries. Nevertheless, a small number of edge cases may remain difficult to evaluate automatically, particularly tautomeric or protonation-equivalent representations and rare boundary edits that change whether a stereocenter or double bond remains stereogenic. Such cases occurred infrequently in our benchmark and are therefore unlikely to materially affect the aggregate results.

To further validate evaluator reliability, we recruited two independent experts with four and five years of molecular-design experience, respectively. We conducted a score-blinded audit of a stratified random sample of 120 model outputs with two experts. The sample contained ten outputs per taxonomy category—five each from 1D and 2D—with randomized, approximately balanced model assignments.

Validation statisticResult
Outputs reviewed120
Individual criteria reviewed580
Criterion-level agreement with expert labels571/580 (98.45%)
Task-level agreement with expert labels119/120 (99.17%)
Expert-labeled task failures106
False acceptances0
False rejections1

The single false rejection resulted from an output for which the extraction pipeline could not identify a unique valid molecule. We count this conservatively as a pipeline-level false rejection even though the downstream chemical predicates were not the source of the disagreement. Within this stratified sample, the automated pipeline exhibited a low task-decision error rate.

AuthorsOpenReview ↗

Q2 and W1: For Case 2,193, does the instruction allow atom-count reduction without a shorter anchor path, and how many linker-shortening tasks rely only on atom count?

We thank the reviewer for identifying this ambiguity. The complete model-facing instruction for Case 2,193 explicitly allowed the model to “shorten the path between the two anchor points by reducing its path length or the number of heavy atoms in the linker.” Therefore, the case was consistent with the literal instruction in the task and the implemented evaluator: the hidden target reduces the editable linker from two heavy atoms to one, although its anchor-to-anchor path length remains unchanged. Nevertheless, we agree that our terminology was imprecise. A reduction in linker-region heavy atoms is not necessarily equivalent to strict linker shortening. We have therefore revised the term usage.

We confirmed through a task-level audit that all 379 tasks originally categorized as linker shortening explicitly informed the model that the operation could be satisfied either by reducing the anchor-to-anchor path length or by reducing the number of heavy atoms in the editable linker region. The evaluator implements exactly the same disjunctive condition. The 379 tasks decompose as follows:

Hidden-target change satisfying the original disjunctionTasksPercentage
Anchor-to-anchor path decreases; linker heavy-atom count does not123.2%
Linker heavy-atom count decreases; anchor-to-anchor path does not24664.9%
Both quantities decrease12131.9%
Total379100.0%

Case 2,193 belongs to the second group: its anchor-to-anchor path-length change is 0, while its editable-region heavy-atom change is -1. We therefore rename the broad operation linker shortening or simplification and reserve strict linker shortening for the 133 tasks in which the protected-anchor path actually becomes shorter. This change clarifies the semantics without retroactively changing the model-facing requirement or its evaluator.

AuthorsOpenReview ↗

Q3 and W2: Please specify the exact inference protocol and clarify whether 2D replaces or retains SMILES and whether normalization is identical.

The 1D and 2D conditions use the same underlying task, semantic instruction, output contract, and evaluator. In 1D, the user message contains canonical SMILES with a SMILES-aligned ASCII annotation of the receiver region and, when applicable, a reference SMILES. In 2D, these molecular strings and ASCII annotations are replaced by the corresponding annotated molecular images; the operation, preservation constraints, property objectives, and output instruction remain unchanged. The 2D condition therefore replaces rather than supplements model-visible SMILES.

Input component1D2D
Starting structureCanonical SMILES plus ASCII region annotationAnnotated x0 image
Reference structureOptional reference SMILES; R1 donor annotation when applicableOptional annotated reference image
Hidden feasible targetNot shownNot shown
Operation and property textSame semantic instructionSame semantic instruction
Expected outputExactly one canonical SMILESExactly one canonical SMILES
Output validationSame deterministic extraction and RDKit canonicalizationSame deterministic extraction and RDKit canonicalization

All tasks were frozen before evaluation. Each case was sent in an independent request without conversation history, tools, hidden targets, criteria, or evaluator information, and no task-specific prompt engineering was used. The revised experimental section will include the exact provider/checkpoint identifiers and the frozen decoding, reasoning, output-token, repeat, and retry settings listed in our response to First Report Q2; these fields must be copied verbatim from the released run configuration.

Both modalities use the same output-extraction and canonicalization pipeline. Evaluation considers only the final generated molecule. Then, the required answer is one short SMILES string. When a small chemistry-specific model wrapped an otherwise identifiable SMILES in presentation text, a deterministic parser removed only code fences, quotation marks, or an explicit answer prefix and extracted the unique RDKit-parseable single-component candidate. It never changed atoms or bonds, used evaluator feedback, or requested a corrected molecule. If no unique valid candidate was identifiable, the response was scored as invalid. The extracted candidate was sanitized and converted to RDKit canonical isomeric SMILES before evaluation.

The 480-task subset is a deterministic, stratified 20% sample of the full benchmark, with exactly 40 tasks from each of the 12 taxonomy combinations. It contains 291 unique starting molecules, 303 unique molecular transformations, and all 64 biological targets represented in the full benchmark. Distributional agreement is summarized below:

CharacteristicTest-vs-full discrepancy
Taxonomy composition0.0% total-variation distance
Number of property objectives0.8% total-variation distance
Property directions0.2% total-variation distance
Editable-region size0.5% total-variation distance
Chemical-operation subtype1.2% total-variation distance
Property identity1.6% total-variation distance
Largest difference for any biological target0.79 percentage points

Rare targets were deliberately retained, which explains the small residual differences in target frequencies.

Reviewer_j53JOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces IF-DrugBench, a benchmark for evaluating whether LLMs follow compositional instructions for localized small-molecule editing. It contains 2,400 tasks derived from ChEMBL molecular pairs and organized along three axes: editing objective, edit location, and reference usage. Maximum common substructures help identify local differences between a starting molecule and a hidden feasible target. The target supports task construction and validation but is not shown to the model or treated as the unique correct answer. Reference-based tasks provide a separate molecule, either for explicit fragment transfer or implicit design guidance. Each task has aligned SMILES-based and molecular-image versions. Prompts, machine-readable criteria, and evaluator configurations are derived from a shared structured specification. RDKit-based and graph-based tools check molecular validity, non-identity to the starting molecule, edit localization, anchor preservation, chemical operations, explicit reference-fragment transfer, and property directions. Multiple solutions are permitted, but every assigned criterion must pass. Experiments cover eleven text-input and seven image-conditioned model settings on a balanced 480-task subset. Gemini 3.7 Flash achieves the highest task pass rates in both settings: 48.1% and 51.5%, respectively.

Critical strengths
  1. The benchmark addresses a meaningful problem in precise molecular editing. A generated molecule can be valid or exhibit desirable properties while violating the requested edit location, protected context, or reference-transfer requirements. IF-DrugBench evaluates these requirements jointly through explicit chemical and graph-based checks. Its treatment of anchor connectivity, donor boundaries, stereochemistry, and ambiguous mappings in symmetric structures makes the evaluation design technically substantive.
  2. The task construction provides a structured resource with feasible, open-ended solutions and paired representations. Tasks are grounded in ChEMBL molecular pairs from the same study and biological-target context, and hidden feasible targets are checked against the assigned criteria without becoming unique required answers. The three-axis taxonomy and aligned SMILES/image versions support systematic comparisons across editing objectives, locations, reference settings, and input representations.
Critical weaknesses
  1. The accuracy of the automated evaluators lacks sufficient independent validation. Evaluator reliability is central to this benchmark, yet having all hidden targets pass primarily establishes internal feasibility: these targets are selected to satisfy the implemented rules. It does not establish whether the evaluators reliably reject incorrect edits or accept other valid solutions. The paper does not report quantitative agreement with independent expert judgments on model outputs, or systematic false-acceptance and false-rejection analyses covering alternative solutions, symmetry, stereochemistry, and donor boundaries. Consequently, low model scores cannot be fully interpreted without establishing the accuracy of the measuring instrument. Addressing this concern requires empirical validation of the evaluators.
  2. The experiments do not sufficiently isolate the capabilities underlying the reported performance differences. R1 introduces an additional donor-transfer requirement absent from R2, while S1 and S2 differ in operation distribution and editable-region size. Operation and reference checks also depend on successful grounding, allowing one upstream failure to produce several failed criteria. These factors weaken the attribution of score differences to reference usage, anchor complexity, or multiple independent reasoning failures. Moreover, the absence of simple programmatic or tool-assisted editing baselines leaves unclear how much difficulty arises from interpreting instructions versus executing graph edits. Matched comparisons, dependency-aware diagnostics, and targeted baselines are needed to substantiate the benchmark’s diagnostic claims.
Questions for authors
  1. Can you clarify and complete the reproduction path for the reported results? The inspected scoring entry point expects 1d-test/manifest.json and 2d-test/manifest.json, whereas the released data use benchmark/ and benchmark-2d/. Can you provide the required manifests or conversion instructions, the exact 480-task selection, and the raw responses and scoring version corresponding to the main tables?
  2. What exact inference protocol was used for each model? Please specify model IDs and versions, system/user prompts, reasoning and output budgets, sampling settings, and policies for API failures, truncation, and retries. Were format-correction reruns used in the reported main results, or only for separate diagnostics? How were these settings selected and frozen before evaluation?
  3. How representative and statistically stable are the results on the 480-task subset? Please explain its selection procedure and report its numbers of unique starting molecules and underlying transformations. Can uncertainty be estimated from the existing outputs while accounting for reused transformations? In particular, how robust are the model rankings and Gemini’s approximately 3.3-percentage-point increase with image input? Please also reconcile the inconsistent CPR values for Claude and Gemini.
  4. What expert involvement and quality assurance were already performed? Please describe the seven-chemist formative study, including how the taxonomy was derived, and distinguish it from subsequent manual inspection of tasks or model outputs. For the latter, what were the sample size, annotation protocol, reviewer expertise, and outcomes? Any existing agreement measurements or documented evaluator errors would help assess the validation concern above.
7 comments
AuthorsOpenReview ↗

Q1: Can you provide a complete reproduction path, including manifests, the exact 480-task selection, raw responses, and the scoring version?

We have completed the reproduction path in the anonymous repository linked in the paper (see the Abstraction). The benchmark/ and benchmark-2d/ directories contain the complete 2,400-task benchmark, whereas 1d-test/ and 2d-test/ contain the matched 480-task subsets used for the reported experiments. We now provide 1d-test/manifest.json and 2d-test/manifest.json. We have also released the model outputs and scoring implementation of GPT-5.6-Sol, which serves as an example that can run both 1D and 2D testing datasets, to facilitate inspection and provide a better understanding of the model's inputs and outputs.

AuthorsOpenReview ↗

Q2 What exact inference protocol was used for each model, including prompts, budgets, sampling, failures, retries, and format correction?

All benchmark tasks were prepared before model evaluation, and every model received the same frozen task instructions. The complete prompts are available in the 1d-test/ and 2d-test/ directories of the anonymous GitHub repository linked in the paper. For 1D evaluation, the stored instruction was passed directly to the model. For 2D evaluation, the corresponding molecular images were provided in the declared order, followed by the same task instruction. For example, a 1D task is presented as:

Starting molecule with the annotated substituent edit region:
CCCCC/C=C/Cc1cc(O)c2c(c1)OC(C)(C)C1CC=C(C)CC21
^^A
|__ MODIFY THIS REGION
Legend: each ^ marks an atom in the editable region, and A marks the protected
attachment anchor connecting that region to the remaining molecule.
Replace or otherwise redesign this local substructure. Preserve the attachment
anchor and keep structures outside the requested region unchanged. Increase the
number of heteroatoms relative to the starting molecule. Return exactly one
canonical SMILES in RDKit canonical atom order and no other text.

No additional task-specific prompt engineering was performed. Each case was evaluated in an independent request, without conversation history, tools, hidden targets, criteria, or evaluator information. We used the model versions listed in the paper and the providers' or released checkpoints' default inference configurations, including default decoding, reasoning, sampling, and output-budget settings. Each task was evaluated independently under the same prompt and output contract. Provider/API transport failures followed the client or provider default retry behavior, whereas parseable model responses were never regenerated or corrected using evaluator feedback. No format-correction reruns were substituted into the reported results.

Test-set sampling. The evaluated subset was selected as follows:

Sampling propertyValue
Full benchmark size2,400 tasks
Evaluated underlying tasks480 tasks (20%)
Taxonomy cells12
Tasks per taxonomy cell40
Unique starting molecules291
Unique molecular transformations303
Biological targets covered64/64
1D/2D relationshipExactly matched underlying tasks; input representation only differs

Within each taxonomy cell, deterministic stratified sampling preserved diversity in biological targets, property directions, numbers of objectives, edit-region sizes, and molecular transformations. Exact identifiers are recorded in 1d-test/manifest.json and 2d-test/manifest.json.

Output handling. Our evaluation considers only the final generated molecule. The same deterministic policy was applied across models and modalities:

Response conditionHandling
Exactly one valid single-component SMILESParse, sanitize, canonicalize, and evaluate
One identifiable SMILES wrapped in presentation textRemove only code fences, quotation marks, or an explicit answer prefix; then parse and evaluate
No unique valid molecular candidateScore as invalid
Provider/API transport failureUse the client/provider default retry behavior; do not use evaluator feedback
Semantically wrong but parseable moleculeDo not correct or regenerate; evaluate as returned

The parser never altered atoms or bonds, used evaluator feedback, or requested a corrected molecular structure. Because the required answer is a short SMILES string, no output-truncation case was observed in the reported runs.

AuthorsOpenReview ↗

Q3: How representative and statistically stable are results on 480 tasks, and how robust are model rankings and the Gemini 1D/2D difference?

The 480-task subset is a deterministic, stratified 20% sample of the full benchmark, with exactly 40 tasks from each of the 12 taxonomy combinations. It contains 291 unique starting molecules, 303 unique molecular transformations, and all 64 biological targets represented in the full benchmark. Distributional agreement is summarized below:

CharacteristicTest-vs-full discrepancy
Taxonomy composition0.0% total-variation distance
Number of property objectives0.8% total-variation distance
Property directions0.2% total-variation distance
Editable-region size0.5% total-variation distance
Chemical-operation subtype1.2% total-variation distance
Property identity1.6% total-variation distance

Rare targets were deliberately retained, which explains the small residual differences in target frequencies.

We conducted three matched evaluations on the same 480 tasks as a sensitivity analysis. The revised main tables report the frozen primary records; the repeated evaluations are used only to assess whether representation-level conclusions are stable.

Each cell shows the change in percentage points [95% paired confidence interval]. Each model is compared with its own matched baseline, with equal weight across the 12 task categories.

ModelΔTPRΔCPR
DeepSeek-V4-Pro-2.08 [-8.33, +4.17]+2.35 [-5.78, +10.52]
GPT-5.6 Sol+3.75 [-6.67, +14.17]+1.93 [-4.32, +8.53]
GLM-5.20.00 [-16.25, +16.25]-1.53 [-10.30, +7.36]
Claude Opus 5-3.33 [-10.83, +4.17]-2.22 [-5.25, +0.63]
Gemini 3.7 Flash-2.08 [-10.42, +4.17]+0.89 [-5.79, +8.01]
Kimi K30.00 [-10.00, +10.00]+1.01 [-4.48, +6.98]
Qwen3.7 Plus-2.08 [-10.42, +6.25]-0.19 [-5.53, +5.19]
ChemDFM-v1.5-8B-2.08 [-7.92, +3.33]+4.49 [-1.06, +9.97]
ChemLLM-20B-Chat-SFT+1.67 [0.00, +5.00]+4.72 [-2.40, +11.52]
ChemDFM-X0.00 [0.00, 0.00]-2.51 [-8.69, +3.81]

Score changes were generally modest. The median absolute change was 2.08 percentage points for both TPR and CPR, and all 20 paired confidence intervals included zero. These are confidence intervals for changes relative to each model's matched baseline, rather than for absolute performance; all baseline and revised point estimates remain within the valid 0--100% range. Ranking shifts were also generally small, with an average absolute movement of 0.50 places.

Our additional Gemini evaluations showed that the direction and magnitude of the 1D--2D difference are not stable across runs. In the final frozen evaluation, Gemini obtains the same TPR in 1D and 2D (48.75% in both), while CPR is slightly lower with 2D input (78.28% in 1D versus 76.20% in 2D). We therefore removed the earlier claim that Gemini benefits from image input. The revised paper treats representation differences as descriptive observations and does not claim a systematic 2D advantage.

We also corrected the apparent inconsistencies in the reported values, which arose solely from using different decimal-rounding conventions across the text and tables; the revision now uses one aggregation and rounding rule throughout.

Reviewer_UruYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of editing molecular structures based on language prompts and proposes a new benchmark, "IF-DrugBench," with 2,400 samples organized along three axes: what to edit, such as increasing or decreasing certain properties or changing specific structures; where to edit, which constrains the location to be modified; and how to learn to edit, such as providing a reference and asking the model to infer editing strategies from it and apply them to the molecule.

Each sample generally contains an original molecule, a language instruction describing the editing requirements, and a hidden target molecule satisfying those requirements. Because a language instruction may allow multiple resulting structures, the hidden target molecule represents one possible answer.

The authors first group ChEMBL molecules from the same study and biological target, then identify similar pairs that differ in a localized region that can be edited. Each pair provides a starting molecule, a hidden feasible target, and structural and property differences that serve as specifications generated using computational chemistry tools. These specifications are then used to generate natural-language instructions.

The dataset is available in two representations, SMILES and images, enabling the evaluation of both LLMs and VLMs. The paper evaluates several open, closed, and domain-specific LLMs and VLMs and reports their numerical performance.

Critical strengths
  1. I think the problem studied in this paper is well motivated and important, although several existing works have explored this direction.

  2. I appreciate the authors' effort to share the entire dataset in the anonymous repository, although I have not been able to examine every detail to verify that the artifacts and reported results are reproducible.

  3. The paper is quite easy to follow, although some details are vague.

Critical weaknesses
  1. The dataset quality is not sufficiently assessed or reported. Although properties and atom indices can be generated automatically using computational tools, and I believe these are quite trustworthy, the language instructions generated from this information may be error-prone. Instructions generated by an LLM might not faithfully reflect the information retrieved from the computational tools. The authors mention LLM-assisted checking and manual inspection, but the details of these procedures should be explicitly disclosed, especially for benchmark curation.

  2. The construction of the evaluation criteria is unclear to me. The paper describes the criteria as being derived from the prompt. I feel that extracting these criteria using a rule-based method would be impossible because language is highly flexible. However, if an additional LLM is used, the correctness of the extraction also needs to be ensured.

  3. The paper could be substantially improved through a deeper analysis of the reasons for failure, rather than simply reporting numerical results indicating whether a model passes or fails. For property-optimization tasks, it is important to distinguish failures in structure-property reasoning from failures in applying an otherwise correctly reasoned edit, since language-prompt-based molecular editing is not perfect. This distinction would indicate whether subsequent efforts should focus on strengthening structure-property reasoning or improving the execution of molecular structure edits. The same concern applies to reference-based tasks: failures may arise when inferring strategies from the reference or during subsequent steps.

  4. In my view, the paper's contribution is very limited. Language-based chemical structure editing has already been studied, and I believe existing papers also directly benchmark molecular editing for property optimization. Automatically curating a benchmark and reporting results without new insights does not, in my view, meet the bar for acceptance at ICLR.

Questions for authors

Almost similar to the weaknesses, but with more context details.

  1. The paper states, "we use LLM-assisted checking and manual inspection to identify inconsistent prompts, missing criteria, weak evaluation tools, or chemically ambiguous cases". How is this checking performed? For LLM-assisted checking, what prompt is used? I could not find it in the repository or the paper, unless I overlooked it. For human inspection, how many samples are inspected, and what review rubric is used?

  2. The paper states, "Formally, let P_i be the prompt for task i, Ci = {ci1, . . . , cik} the criteria derived from it, and Ei = {ei1, . . . , eik} the corresponding evaluation tools". How are the criteria extracted from the prompt? Is a rule-based method used, or is an additional LLM involved? If an LLM is used, how is the correctness of the extracted criteria ensured?

  3. For property-optimization tasks, do failures arise from incorrect chemical reasoning, or do models reason correctly but encounter problems when applying the edits? Similarly, for reference-based tasks, do failures arise when inferring editing strategies from the reference or during subsequent steps? Could the paper provide a deeper analysis of these different sources of failure?

  4. The following example is taken from the anonymous repository:

Starting molecule with the annotated substituent edit region:

NC(=O)c1ccc(-c2cnc3[nH]ccc3n2)cc1
^^  ^ A
|__ MODIFY THIS REGION

Replace or otherwise redesign this local substructure.
preserve the substituent attachment anchor to the parent
structure and keep structures outside the requested region
unchanged. The property goal is to decrease number of amide
bonds relative to the starting molecule. Return exactly one
canonical SMILES in RDKit canonical atom order and no other text.

I feel that the prompt here is not sufficiently explicit (also, after checking the repository, I think there is no system prompt explaining it). It should explicitly explain what these symbols mean, such as “^” and “A”. I can guess what they mean from the context, but I feel it is necessary to explain them in the prompt rather than letting the LLM guess or reason about their meaning.

7 comments
AuthorsOpenReview ↗

Q1 and W1: How was LLM-assisted checking performed, what prompt was used, and what was the scale and rubric of human inspection?

We initially constructed 21,504 candidate tasks. Quality assurance proceeded in four stages:

StageInputProcedureResult
Deterministic chemical screening21,504 candidatesRDKit 2024.09.4 checks for connected and unambiguous edit regions, valid S1/S2 assignments, non-duplicated transformations, valid reference transfers, hidden-target feasibility, and other hard graph constraintsCandidates passing all deterministic checks proceed to LLM-assisted screening
LLM-assisted screeningCandidates passing deterministic screeningGPT-5.6 Sol audits prompt–criterion–evaluator alignment, chemical coherence, feasibility, and reference use; only tasks assigned PASS are retained2,883 candidates retained after the combined deterministic and GPT-assisted filter
Expert audit2,883 retained candidatesFour domain experts each inspect approximately 720 tasks and relevant evaluator code; flagged cases are collectively adjudicated21 inconsistent cases removed; 2,862 eligible candidates remain
Balanced benchmark selection2,862 eligible candidatesStratified selection across the 12 taxonomy combinations2,400 tasks retained (200 per combination)

GPT-5.6 Sol proposed potential issues involving prompt-criterion alignment, evaluator correctness, chemical coherence, hidden-target feasibility, and reference usage. We used the following checker prompt:

You are auditing a molecular-editing benchmark task.
1. Extract every requirement stated in the model-facing instruction, including
   property objectives, editable-region constraints, chemical operations,
   preservation constraints, and reference usage.
2. Check whether each requirement has a corresponding machine-readable
   criterion and whether any criterion introduces an unstated requirement.
3. Check whether each assigned evaluator directly and correctly tests its
   criterion.
4. Check whether the prompt, criteria, evaluators, and hidden target agree on
   the requested edit region, property directions, chemical operation, and use
   of reference information.
Return structured JSON containing the extracted requirements,
prompt-criterion-evaluator mappings, identified issues, explanations, and an
overall decision of PASS, REVISE, or REMOVE.

The human review used the same core rubric, supplemented by source-code inspection of evaluators. Only tasks assigned PASS by GPT-5.6 Sol after deterministic screening entered the 2,883-candidate pool. The experts then reviewed this retained pool. Expert-flagged tasks were discussed collectively, and 21 cases were confirmed to contain substantive inconsistencies among the model-facing prompt, criteria, evaluator implementation, and hidden feasible target. These cases were excluded. The remaining 2,862 tasks formed the eligible pool from which the balanced 2,400-task benchmark was selected.

AuthorsOpenReview ↗

Q2 and W2: How were criteria extracted from flexible natural-language prompts, and how was extraction correctness ensured?

No criteria are extracted from free-form prompt text. Our original wording was misleading. For each task, we first derive a structured set of candidate requirements from the molecular data. These requirements may include property directions computed from x0 and x*, chemical operations identified from their graph difference, and so on. We then select a compatible combination of these requirements according to the instruction taxonomy and construct a structured task specification. The natural-language instruction is then generated from this structured criteria specification. Therefore, the criteria are defined before language generation rather than extracted retrospectively from the generated prompt.

AuthorsOpenReview ↗

Q3 and W3: Can the paper distinguish failures of chemical reasoning or reference-strategy inference from failures in executing an otherwise appropriate edit?

We agree that pass/fail scores alone cannot distinguish between failures in devising an appropriate editing strategy and failures in correctly executing an otherwise appropriate edit. To address this concern, we conducted a supplementary, trace-informed analysis of model failures by examining the reasoning traces alongside the corresponding molecular editing outputs. Specifically, during error analysis, where supported by the provider, we examined the models’ exposed reasoning summaries or thinking traces alongside their final outputs. These traces were used to categorize failure patterns and generate hypotheses about potential error sources. We randomly sampled 600 erroneous responses, comprising 300 1D and 300 2D responses, and asked two domain experts to annotate the underlying failure causes. We have summarized four types of errors.

Failure category1D Cases1D Percentage2D Cases2D Percentage
Task understanding and structural grounding errors24882.7%22274.0%
Objective and property reasoning errors3210.7%3812.7%
Plan-to-output implementation errors144.7%144.7%
Mixed reasoning and implementation errors62.0%268.6%
Total300100.0%300100.0%

First, task understanding and structural grounding dominate failures in both modalities. This category accounted for 82.7% of attributable failures in 1D and 74% in 2D. Common patterns included selecting the wrong editing site, expanding a local modification to an entire ring, and incorporating unmarked reference atoms into the transferred fragment. In 1D, models sometimes misaligned textual annotations with the corresponding molecular atoms. In 2D, models misinterpreted highlighted boundaries or reconstructed protected molecular regions incorrectly. In these cases, the intended plan already violated the task specification, and the final output often followed that mistaken plan.

Second, objective and property reasoning errors reflect incomplete constraint verification. These errors represented 10.7% of attributable failures in 1D and 12.7% in 2D. Models sometimes weakened a strict decrease requirement into a non-increase condition or inferred property improvements from an incomplete chemical heuristic. Such failures show that a plausible chemical modification can still fail the requested objective. They motivate explicit verification of every property direction and structural constraint before accepting a candidate.

Third, correct local intentions do not always translate faithfully into molecular outputs. Plan-to-output implementation errors accounted for 4.7% of attributable failures in 1D and 4.7% in 2D. These cases contained evidence of a specific correct local intention that was contradicted by the final structure. Examples included explicitly preserving a chain but omitting a carbon during SMILES generation, changing protected connectivity during an attempted canonical rewrite, and identifying the required absolute configuration correctly but encoding its opposite.

Fourth, mixed failures distinguish the observed 2D error profile. Reasoning and implementation errors co-occurred in 8.6% of attributable 2D failures, compared with 2.0% in 1D. For example, a model could expand the donor region incorrectly while also reversing a protected stereocenter that it explicitly intended to preserve.

These observations motivate evaluation and verification at three distinct stages: identifying the permitted edit, selecting a strategy that satisfies the objectives, and faithfully realizing that strategy in the output structure. Region and anchor checks could support grounding, descriptor calculations could support objective verification, and graph comparisons could detect unintended structural changes during output generation. The effectiveness of these interventions remains to be tested. We have added this analysis to the Appendix.

Comments on the paper

1 comment

Dear Reviewers,

We sincerely thank the reviewers for their thoughtful comments and constructive suggestions. We have carefully considered all the concerns and substantially revised the manuscript to improve the rigor, transparency, reproducibility, and diagnostic value of IF-DrugBench. In particular, we have conducted additional analyses and validation experiments, clarified the benchmark construction and evaluation protocols, and refined our interpretations to better align with the empirical evidence.

The major revisions are summarized as follows:

  1. Strengthened benchmark construction and quality assurance. We have substantially expanded the description of our benchmark curation pipeline, including deterministic chemical validation using RDKit, LLM-assisted auditing, and manual inspection by four domain experts. We now provide the specific requirements used for LLM-assisted checking and clarify how inconsistencies among model-facing instructions, evaluation criteria, evaluator implementations, and hidden feasible targets were identified and resolved. We also clarify how the final 2,400 tasks were selected from the initial 21,504 candidates.

  2. Added independent validation of evaluator reliability. To establish the reliability of our automated evaluation tools beyond verifying the feasibility of hidden targets, we conducted a score-blinded audit of 120 model outputs with two independent domain experts. The automated pipeline achieved 98.45% agreement at the individual-criterion level and 99.17% agreement at the overall task level, with a 0.00% false-acceptance rate among reviewed failures. We have also clarified how our evaluators account for molecular connectivity, stereochemistry, symmetry-related mappings, and donor boundaries.

  3. Added a trace-informed analysis of model failures. To provide deeper insights beyond aggregate pass/fail scores, we analyzed 600 erroneous model responses across 1D and 2D tasks, with failure causes annotated by two domain experts. We identified four recurring failure categories: task understanding and structural grounding errors, objective and property reasoning errors, plan-to-output implementation errors, and mixed reasoning and implementation errors. We further discuss their distributions and implications for different stages of molecular editing.

  4. Improved experimental reproducibility and protocol transparency. We have completed the reproduction path in the anonymous repository by providing the exact 480-task test-set manifests, model outputs, and scoring implementation. We have also clarified model identifiers and versions, input construction, inference procedures, output extraction, and RDKit canonicalization, making the experimental setup more transparent and reproducible.

  5. Added statistical robustness analyses and clarified modality comparisons. We independently evaluated each model three times on the same 480 tasks and report mean performance and paired confidence intervals. We further analyzed the representativeness of the test subset, model-ranking stability, and performance differences between 1D and 2D inputs. These analyses provide a more comprehensive assessment of the robustness and variability of the reported results.

  6. Clarified task definitions and refined the interpretation of experimental findings. We have added explicit explanations of annotation symbols in the model-facing prompts and clarified the operational definition of linker shortening through a task-level audit. We also conducted an additional analysis of 230 failed R2 responses to characterize different patterns of model interaction with reference molecules. Importantly, we have refined our claims to distinguish observable reference interaction from verified reference utilization and to avoid attributing performance differences to factors that our current experiments do not independently isolate.

We believe these revisions have substantially strengthened the manuscript by providing more rigorous benchmark validation, more transparent experimental procedures, and deeper insights into the capabilities and limitations of current language-based molecular editing models.

We sincerely appreciate the reviewers' time and valuable feedback, which have helped us improve the quality and clarity of this work. We hope that the revised manuscript adequately addresses their concerns.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.