acceptodds
Under review as a conference paper at ICLR 2027

Beyond Benchmark Accuracy: A Diagnostic Test for LLM Chemical Prediction

Abstract

Large language models have achieved strong performance on chemical prediction benchmarks. However, existing evaluation methods typically provide the identity information of the target entity directly, making it difficult to determine whether model predictions arise from transferable structure–property reasoning capabilities or rely on target-related contextual information or memory retrieval.We propose the Identity-Blinded Reasoning Diagnostic framework to evaluate chemical prediction models when target entity identity information is hidden. In this framework, the target is no longer directly defined by its entity identity, but instead by a known source entity and an explicit structural edit operation. This setting preserves the information required to determine the target while reducing the opportunity for models to access target identity cues.We evaluate this framework across six tasks spanning odor prediction, molecular property prediction, retrosynthesis, and reaction yield prediction. The results show that models generally achieve better prediction performance when the properties of the source entity and target entity are consistent. Providing explicit source entity labels further strengthens this dependence. Moreover, while keeping the target entity, structural change, and ground-truth label unchanged, replacing only the source entity label causes some model predictions to systematically shift toward the substituted source labels.These results indicate that high benchmark accuracy in chemical prediction tasks does not necessarily imply that large language models possess reliable structure–property reasoning capabilities. Our diagnostic framework provides a controlled evaluation approach for testing whether models can genuinely update predictions according to entity changes. The framework can be extended to other AI for Science tasks, particularly those where targets can be defined through transformations of known entities.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
LLM diagnostic evaluation, identity-blinded reasoning, large language models, chemical prediction, scientific reasoning, AI for Science

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_raHNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the Identity-Blinded Reasoning Diagnostic framework to investigate whether LLMs rely on transferable structure–property reasoning or contextual information when performing chemical prediction tasks. Instead of directly providing the target molecule's identity, the framework describes the target through a known source molecule and an explicit structural modification. The authors further examine the influence of source labels through controlled interventions and evaluate the framework across six tasks covering odor prediction, molecular properties, retrosynthesis, and reaction yield prediction. The findings reveal that model predictions can be strongly influenced by source-related information, suggesting that high benchmark accuracy does not necessarily indicate reliable chemical reasoning capabilities.

Critical strengths

The paper offers a valuable diagnostic perspective by examining the reliability of LLM-based chemical predictions beyond conventional benchmark accuracy. The identity-blinded evaluation design is a notable strength, as it reduces direct access to target identity information while retaining structural relationships. The controlled source-label interventions provide a systematic way to investigate the influence of contextual cues on model predictions. In addition, the evaluation across six tasks spanning different chemical domains broadens the relevance of the findings and highlights important limitations in current LLM prediction capabilities.

Critical weaknesses

Although the identity-blinded framework reduces direct access to target information, it does not completely eliminate the possibility of recovering target identity from the source structure and structural modifications. Therefore, the observed prediction patterns may not fully distinguish memorisation from genuine chemical reasoning. In addition, the dependence on source labels could partly reflect legitimate structure–property relationships rather than inappropriate reliance on contextual cues. Further analysis is needed to clarify these effects and establish the robustness of the diagnostic conclusions across different models and experimental conditions.

Questions for authors
  1. How do the authors ensure that the identity-blinded setting effectively prevents models from recovering target identity through the source structure and structural modifications?

  2. How can the authors distinguish inappropriate dependence on source labels from legitimate chemical relationships between source and target molecules?

  3. How consistent are the diagnostic findings across different LLMs, and could additional experiments help establish the robustness and generalisability of the proposed framework?

4 comments

W1 and Q1 Identity blinding and recovery of target identity

The reviewer's observation is correct. Identity blinding does not completely prevent models from recovering the target identity. In fact, if the target identity were completely hidden, the prediction task could not be defined. The design goal of this paper is not to prevent recovery but to avoid presenting the target's SMILES or name directly in the prompt, thereby removing the shortcut of direct retrieval while preserving all structural information needed for reasoning. Structure reconstruction experiments show that models reconstruct the target from the source molecule plus the edit with accuracy ranging from 86.05% to 100%.

It is precisely because the target can be recovered that this design can serve as a diagnostic. We demonstrate from two perspectives that the conclusions are not affected by whether or not the identity is recovered.

  1. The comparison between C0 and C1. If models first recover the target identity and then perform structure–property reasoning based on the target, the C1 results should be close to C0. However, on the same set of samples, CR>1.2 occurs in only 3/24 combinations under C0, with no systematic between-group gap. C1 merely changes the target presentation to source molecule plus edit without providing any labels, yet CR>1.2 rises to 14/24, with 10/24 reaching statistical significance, and C1 exceeding C0 in 20/24 combinations.

  2. When the target identity is directly provided, as shown in G6. When the target SMILES is directly provided in the prompt, the model's memory and knowledge about the target can be fully invoked, yet source-label dependence still persists. C2-direct has CR>1.2 in 15/15 combinations, and Claude still has 90.4% and 94.2% of predictions switching with the source label on BACE and HIV. Even when the model can access the target identity, it does not use the target identity to override the source label.

Therefore, regardless of whether the identity can be recovered, scores on standard benchmarks are insufficient to demonstrate that LLMs complete predictions based on structure–property relationships.

W1 and Q2 Distinguishing inappropriate dependence from legitimate structure–property relationships

We agree that the properties of a source molecule can serve as a reasonable reference. We distinguish reasonable reference from inappropriate dependence at three levels.

  1. Without labels. Neither C0 nor C1 provides source labels. On the same set of samples, there is no systematic consistent-group advantage under C0, with CR>1.2 in only 3/24 combinations. After switching to the identity-blinded presentation, C1 rises to 14/24. Legitimate structure–property relationships do not change when the target's presentation format changes, and structure reconstruction experiments show that models can recover the target identity under C1. Therefore, this change cannot be explained by legitimate structure–property relationships.

  2. With true labels, the magnitude exceeds the range that the structure–property relationship can support. Under C2, 22/24 combinations have CR>1.2. Claude achieves 100% accuracy in the consistent group and 0% in the inconsistent group on BACE and HIV, identical to the result of directly copying the source label. The Morgan fingerprint nearest-neighbor predictor in Table G5, which does not invoke any language model, represents the between-group gap that the structure–property relationship alone can produce. In 17/20 combinations, the LLM's C2 CR exceeds this reference. The complete comparison is provided in Table G5.

  3. When the target structure is visible, the dependence persists. In G6, the target SMILES is directly provided, and C2-direct still has CR>1.2 in 15/15 combinations. If the model reasonably treats the source label as a reference, it should rely more on the target itself when the target structure is fully visible, yet the actual label switching rates are essentially the same as under identity blinding.

In addition, Table G4 uses far-distance molecular pairs with source–target Tanimoto similarity between 0.20 and 0.55. The CR under C1 and C2 still increases relative to C0 in all 6/6 combinations, although the magnitude is weaker than with near-distance pairs. This indicates that the dependence is not limited to cases of high structural similarity.

Q3.1 Diagnostic consistency across models and framework robustness

Supplementary experiment Table G2b summarizes collapse ratios along the model dimension.

  • Source-label dependence is highly consistent across models. The C2 CR of all four models exceeds C0 CR on all six tasks, and C2 CR>1.2 covers 22/24 combinations.
  • After directly providing the target structure in G6, all 15/15 combinations across the three tested models still have CR>1.2.
  • The chemistry-specific models ChemDFM and ChemLLM have C2 CR>1.2 in 5/5 and 4/5 of the computable task combinations respectively, as shown in G1.
  • Source-structure dependence under C1 varies by task and model. DeepSeek-V4-Pro is the weakest with 2/6 significant, while Claude, GPT, and Gemini all reach 4/6, but the two olfactory tasks consistently appear across all four models.
ModelC1 sig.C2 sig.C1>C0C2>C1C2>C0C2>C1>C0
Claude-Opus-4.84/66/66/66/66/66/6
DeepSeek-V4-Pro2/64/64/65/66/63/6
GPT-5.54/66/65/66/66/65/6
Gemini-3.1-Pro4/66/65/65/66/64/6

Q3.2 Can additional experiments support robustness and generalizability?

Yes. The supplementary experiments we have added correspond to the following robustness dimensions.

  • Model type. Chemistry-specific models ChemDFM and ChemLLM, presented in G1.
  • Sample difficulty. Comparison of label-consistent and label-inconsistent groups under C0, presented in G3.
  • Structural distance. Far-distance molecular pairs with Tanimoto 0.20–0.55, presented in G4.
  • Structure–property relationship reference. Morgan fingerprint nearest-neighbor baseline, presented in G5.
  • Identity visibility. Source-label intervention with direct target SMILES, presented in G6.

Across all these dimensions, source-label dependence remains stable. Source-structure dependence is consistent across models on the olfactory tasks and varies by model on other tasks, which is consistent with the conclusions in the main text.

We have provided targeted responses to the reviewer's questions, addressing the recoverability of target identity, the distinction between legitimate structure–property reference and inappropriate source dependence, and the robustness of the findings across models and supplementary experiments. As the discussion period is drawing to a close, we would like to confirm whether these responses have adequately addressed the reviewer's concerns. We would be happy to provide further clarification on any remaining issues.

Thank you for your review. We have responded to each of your three questions in the rebuttal.

  1. Regarding whether target identity can be recovered, we clarified the design goal of identity blinding and directly provided the target SMILES in Appendix G6, confirming that the conclusions hold regardless of whether the identity can be recovered.
  2. Regarding distinguishing legitimate structure–property reference from inappropriate dependence, we added three groups of supplementary experiments including C0 group comparison, Morgan nearest-neighbor baseline, and far-distance molecular pairs, presented in Appendix G3, G5, and G4 respectively, confirming that the LLM's dependence on source labels exceeds the range that structural similarity alone can explain.
  3. Regarding consistency across different models, we added evaluations on chemistry-specific models and summarized the results along both the model and task dimensions, presented in Appendix G1 and G2 respectively, confirming that source-label dependence is consistent in direction across models and tasks.

The discussion period is drawing to a close. We kindly ask you to take a moment to review these responses and let us know whether they have addressed your concerns. If anything remains unclear, please point out the specific issue and we will provide further clarification immediately.

Reviewer_2dXVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes the Identity-Blinded Reasoning Diagnostic framework to evaluate chemical prediction models when target entity identity information is hidden. Compared with previous research, the paper provides a source entity and an explicit structural edit rather than a SMILES string or name. The paper evaluates this framework across six tasks spanning odor prediction, molecular property prediction, retrosynthesis, and reaction yield prediction.

Critical strengths
  1. The paired source-label test provides evidence of contextual influence. The C2–C3 comparison changes only the source label while holding the source structure, edit, and target constant. For example, 91.2% of Claude’s BACE predictions switch to the substituted label. Such a phenomenon also appears in set-valued and numerical outputs.
  2. The experiment of target structure reconstruction provides evidence that the source and edit can recover the target structure. Binary classification, descriptor, retrosynthesis, yield prediction, and other diagnostic tests provide additional evidence.
Critical weaknesses
  1. Label-consistent and label-inconsistent groups are not matched for difficulty. The paper deliberately chooses source and target molecules to be structurally related. Additionally, the substituted labels are presented as true source properties without any indication to the model that they are false. As a result, a predictor using related evidence can change its answer with correct structure–property reasoning. The paper might need stronger control (e.g. source copying and nearest-neighbor baselines and an otherwise matched condition in which the target structure is provided directly).
  2. Appendix C shows that the paper replaces the HIV task name with a generic description of enzyme inhibitory activity. However, the evaluation still uses the original HIV labels. This creates an alternative explanation for poor predictions and heavy reliance on source labels: the requested property may be underspecified.
Questions for authors
  1. What evidence distinguishes inappropriate label copying from reasonable reliance on information presented? What aspect of the existing results shows a failure to reason about the target, rather than sensitivity to changes in evidence?
  2. Appendix C says the HIV prompts replace a generic description of enzyme inhibitory activity because of Claude’s safety restrictions. Is this substitution applied to all four models, or only to Claude?
4 comments

W1.1 Difficulty mismatch between the consistent and inconsistent groups

Supplementary experiment Table G3 partitions the C0 results into the label-consistent and label-inconsistent groups. C0 directly provides the target SMILES and involves no source molecule or identity blinding.

  • No systematic consistent-group advantage exists under C0. CR>1.2 occurs in only 3/24 combinations, all from BACE, and the inconsistent group actually performs better on HIV, retrosynthesis, and yield.
  • After identity blinding under C1, CR>1.2 rises to 14/24.
  • After adding true source labels under C2, it rises to 22/24.
  • Strict monotonicity C2>C1>C0 holds in 18/24 combinations.

C0, C1, and C2 use the same set of samples and differ only in how the target is presented and whether a source label is provided. This gradient cannot be explained by inherent between-group difficulty differences.

W1.2 Intentional selection of structurally related source and target molecules

The main text selects structurally close molecular pairs to ensure that LLMs can reconstruct the target from the source molecule plus the edit. We have added a diagnostic with far-distance molecular pairs, presented in Table G4. The Morgan Tanimoto similarity between source and target ranges from 0.20 to 0.55, while still ensuring that the two differ by exactly one uniquely recoverable local edit. Because the structural difference is larger, exact reconstruction accuracy drops to 53.2%–58.8% for HIV and 34.3%–39.4% for olfaction. To exclude the confound of reconstruction failure, we report results only on samples where all three models exactly reconstruct the target.

Under the far-distance condition, the direction of the main findings is preserved, but the magnitude is weaker. Relative to C0, the CR under C1 and C2 increases in all 6/6 model–task combinations. Compared with the near-distance results, there are two findings.

  1. The collapse magnitude under C2 is less extreme than with near-distance molecular pairs, indicating that structural similarity does amplify model dependence on source labels to some extent, consistent with the reviewer's concern.

  2. C2 is no longer consistently higher than C1. For example, Gemini's C1 CR on HIV is 3.32 while its C2 CR is 2.17. This indicates that the relative contributions of source-structure dependence and source-label dependence shift under far-distance conditions.

Claude-Opus-4.8GPT-5.5Gemini-3.1-Pro
HIV C00.88×0.93×1.40×
HIV C11.20×1.60×3.32×
HIV C22.75×1.63×2.17×
Olfaction C01.22×1.18×1.42×
Olfaction C11.33×1.64×1.72×
Olfaction C22.15×1.99×1.98×

W1.3 and Q1 Distinguishing inappropriate label copying from reasonable dependence, and stronger controls

We first emphasize that the main body of evidence in this paper consists of C0, C1, and C2, none of which contains any false labels. C3 is a counterfactual intervention experiment that fixes the source structure, edit, and target and replaces only the source label, measuring how much of the prediction shift under C2 is caused by the source label value itself. Even without considering C3 at all, the gradient from C0 through C1 to C2 already holds.

The key to distinguishing "reasonable sensitivity to evidence changes" from "failure to reason about the target" lies in the inconsistent group of C2. In this group, the source label is true and is explicitly attributed to the source molecule, and the prompt also provides the structural edit between the target and the source molecule. Correct reasoning should recognize that the target's property has changed, yet the model still outputs the source molecule's property. This is not a reasonable response to incorrect evidence, because the evidence itself is not wrong. What is wrong is transferring the source molecule's property to the target. G6 further shows that this transfer persists even when the target SMILES is provided directly, as described in W1.4.

Source-copying baseline. Directly outputting the source label yields 100% in the consistent group, 0% in the inconsistent group, and CR=+∞. The results of Claude-Opus-4.8 under C2 on BACE and HIV are identical to the copying baseline. GPT-5.5 and Gemini-3.1-Pro fall between the nearest-neighbor baseline and the copying baseline.

For the two olfactory tasks, descriptor prediction is open-set, and the prior probability of predictions coincidentally matching the source label is extremely low. From C1 to C2, the only difference is the addition of a true label explicitly attributed to the source molecule, yet CR systematically increases, with structural change olfaction rising from 1.68–1.85 to 1.77–3.30 and chirality change olfaction rising from 1.33–2.56 to 3.65 and above.

Nearest-neighbor baseline. The Morgan fingerprint nearest-neighbor predictor in Table G5, which does not invoke any language model, represents the between-group gap that the structure–property relationship alone can produce. In 17/20 combinations, the LLM's C2 CR exceeds this reference.

Taskk-NN CRClaude C2GPT C2DeepSeek C2Gemini C2
BACE1.83×+∞3.69×1.45×2.48×
HIV1.89×+∞5.32×0.97×2.17×
Structural change olfaction1.25×3.30×1.89×2.69×1.77×
Chirality change olfaction2.84×+∞16.22×3.65×12.94×
Yield1.22×1.46×2.40×0.98×2.80×

W1.4 Matched condition with direct target structure

We have added this matched condition in Appendix G6. The source structure, source label, edit description, prediction target, and evaluation ground truth are kept identical to those in C2/C3, with the target SMILES additionally provided directly. We test on three models and five tasks.

  • C2-direct has CR>1.2 in 15/15 combinations, whereas the same samples have CR>1.2 in only 2/15 under C0.
  • The response to source label replacement is largely consistent with the identity-blinded setting. Claude's switching rate on BACE is 91.2% versus 90.4%, and on HIV is 94.6% versus 94.2%.

Complete results are provided in Table G6. When the target structure is fully visible and the model does not need to reconstruct the target, replacing a label that is explicitly attributed to the source molecule still causes over 90% of predictions to flip. This is not the behavior a predictor capable of correct structure–property reasoning would exhibit. Rather, the source label overrides reasoning about the target structure.

This result also clarifies the role of identity blinding. Source-label dependence exists equally when the target structure is provided directly, and it is a general behavior of LLMs whenever labels of similar molecules appear in the context. This makes the conclusion equally relevant to few-shot and retrieval-augmented chemical evaluations. The diagnostic value unique to identity blinding lies in the C0 versus C1 comparison, where neither provides any labels and the only difference is how the target is presented. On the same samples, CR>1.2 rises from 3/24 under C0 to 14/24 under C1, with C1 exceeding C0 in 20/24 combinations. This label-independent source-structure dependence is not observable under the standard benchmark setting.

W2 and Q2 HIV task description replacement

We thank the reviewer for raising this point. We did not explain this clearly in the paper. For fairness in cross-model comparison, the HIV task uses a generic description of enzyme inhibitory activity uniformly across all four models, not only for Claude. We understand the reviewer's concern that a generic description may make the predicted property less specific and thereby increase dependence on source labels. Even accepting this explanation, it applies to only the HIV task alone. The remaining five tasks all use the original property names and do not have this issue. After removing HIV, the remaining five tasks show 8/20 statistically significant consistent-group advantages under C1 and 19/20 under C2. The C0-to-C2 gradient also holds, with C1 exceeding C0 in 16/20, C2 exceeding C1 in 19/20, strict C2>C1>C0 in 15/20, and C2 exceeding C0 in 20/20. The conclusions are not driven by the HIV task alone and do not change when the HIV task is removed.

Thank you for the detailed responses. Most of my concerns have been addressed. I will raise my score to 3.

Reviewer_XizxOpenReview ↗
Rating
3(+1)
Confidence
2(−2)
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes the Identity-Blinded Reasoning Diagnostic, which specifies a target molecule through a source structure and a structural edit description without directly providing the target’s name or complete SMILES. The evaluation compares four conditions: direct target input, source-plus-edit input, source-plus-edit input with a true source label, and the same input with a false source label. Across six chemical prediction tasks and four models, the results reveal substantial source-label dependence in several settings: performance varies with source–target label consistency, and replacing the source label can systematically change predictions while the target remains fixed. The paper also investigates whether external tools providing property-relevant evidence improve prediction.

Critical strengths
  1. The fixed-target label interventions provide direct evidence of source-label dependence. C2 and C3 preserve the source structure, edit description, and target ground truth while changing only the source label. This provides more diagnostic information than aggregate benchmark accuracy alone
  2. The evaluation covers multiple output types and includes checks of structural recoverability. The six tasks span classification, set prediction, reactant generation, and numerical prediction, extending the analysis beyond an isolated task.
Critical weaknesses
  1. reasoning, or establish the additional diagnostic value of withholding direct target identity. A source molecule’s property can legitimately inform predictions for a similar target. When a model is uncertain about the target property, a source label presented as factual may reasonably influence its prediction. Sensitivity to a false label therefore establishes contextual dependence, but does not by itself identify a chemical reasoning failure. Moreover, the consistent and inconsistent groups are defined by their ground-truth property relationships and may differ in inherent prediction difficulty
Questions for authors

1 . Diagnostic validation: If the target SMILES is supplied directly together with the same true or false source labels, do comparable prediction shifts occur? What additional reasoning failure does the proposed diagnostic reveal relative to source-label-copying and structural-similarity baselines? Do the conclusions persist when restricting the analysis to correctly reconstructed targets?

7 comments

W1.1 Distinguishing reasonable reference from reasoning failure

We agree that the properties of a source molecule can serve as a reasonable reference for a structurally similar target, and that sensitivity to the source label alone only demonstrates contextual dependence. The criterion this paper uses to identify reasoning failure is whether the model makes use of the structural edit provided in the problem. A model that reasonably uses the source label would treat it as a prior and then adjust according to the edit, with the overall shift magnitude comparable to what the structure–property relationship itself supports. A model that fails to reason would ignore the edit and directly carry over the source molecule's properties. The existing results support the latter case.

  1. C2 uses true labels and explicitly attributes them to the source molecule. Claude-Opus-4.8 achieves 100% accuracy in the consistent group and 0% in the inconsistent group on BACE and HIV, meaning that it outputs the source molecule's label for every sample, and the edit has no effect on predictions. In the retrosynthesis task, the product structure has already changed, yet the model still carries over the reactant skeleton from the source reaction.

  2. The shift magnitude exceeds the range that the structure–property relationship can support. The Morgan fingerprint nearest-neighbor predictor in Table G5, which does not invoke any language model, represents the between-group gap that "structurally similar molecules tend to have similar properties" alone can produce. In 17/20 combinations, the LLM's C2 CR exceeds this reference, with some combinations reaching +∞ or above 5×, whereas the nearest-neighbor predictor reaches only 1.8–1.9×.

  3. The dependence persists when the target structure is fully visible, as shown in G6. After directly providing the target SMILES, C2-direct has CR>1.2 in 15/15 combinations, whereas the same samples have CR>1.2 in only 2/15 under C0. After replacing the source label, Claude still has 90.4% and 94.2% of predictions switch accordingly on BACE and HIV. At this point, the model faces no uncertainty about the target structure, yet over 90% of predictions are still determined by the source label.

Three layers of evidence indicate that the collapse under C2 is not a reasonable reference to the source label but rather the source label replacing reasoning about the target structure.

W1.2 Difficulty differences between the consistent and inconsistent groups

Supplementary experiment Table G3 partitions the C0 results into the label-consistent and label-inconsistent groups. C0 directly provides the target SMILES and involves no source molecule or identity blinding.

  • No systematic consistent-group advantage exists under C0. CR>1.2 occurs in only 3/24 combinations, all from BACE, and the inconsistent group actually performs better on HIV, retrosynthesis, and yield.
  • After identity blinding under C1, CR>1.2 rises to 14/24.
  • After adding true source labels under C2, it rises to 22/24.
  • Strict monotonicity C2>C1>C0 holds in 18/24 combinations.

C0, C1, and C2 use the same set of samples and differ only in how the target is presented and whether a source label is provided. If the between-group gap arose from inherent sample difficulty, it should have already appeared under C0. This gradient cannot be explained by between-group difficulty differences.

Q1.1 Providing target SMILES directly together with source labels

We have added this experiment in Appendix G6. The source structure, source label, edit description, prediction target, and evaluation ground truth are kept identical to those in C2/C3, with the target SMILES additionally provided directly. We test on Claude-Opus-4.8, GPT-5.5, Gemini-3.1-Pro, and five tasks.

Answer: comparable prediction shifts do occur. Directly providing the target structure does not reduce source-label dependence. C2-direct has CR>1.2 in 15/15 combinations, whereas the same samples have CR>1.2 in only 2/15 under C0, and there is no systematic weakening between C2-direct and C2. The response to source label replacement is also largely consistent between the two settings, with Claude's switching rate on BACE at 91.2% versus 90.4% and on HIV at 94.6% versus 94.2%. Complete results are provided in Table G6.

  1. Source-label dependence is a general behavior of LLMs whenever labels of similar molecules appear in the context. This makes the conclusion more relevant to standard evaluations, because few-shot and retrieval-augmented chemical evaluations place exactly such labels into the context.

  2. The additional diagnostic value of identity blinding lies in the C0 versus C1 comparison. Neither provides any labels, and the only difference is how the target is presented. On the same samples, CR>1.2 rises from 3/24 under C0 to 14/24 under C1, with 10/24 reaching statistical significance, and collapse ratio under C1 exceeding C0 in 20/24 combinations. This label-independent source-structure dependence is not observable under the standard benchmark setting of C0.

Q1.2 Source-label copying baseline

For binary classification tasks, the copying baseline directly outputs the source molecule's label, achieving 100% accuracy in the consistent group, 0% in the inconsistent group, and CR=+∞. Using this as a reference, the results of Claude-Opus-4.8 under C2 on BACE and HIV are identical to the copying baseline. The C2 CR values of GPT-5.5 and Gemini-3.1-Pro range from 2.17 to 5.32, falling between the nearest-neighbor baseline of 1.83–1.89 and the copying baseline, with most values closer to the copying side.

For the two olfactory tasks, descriptor prediction is open-set, the output space is far larger than binary classification, and the prior probability of predictions coincidentally matching the source label is extremely low. From C1 to C2, the source structure, edit description, and target structure remain completely unchanged, and the only difference is that C2 adds a true label explicitly attributed as the experimentally observed property of the source molecule. If the model reasons from the target structure, this information should not systematically alter predictions. However, the actual results show that the CR of structural change olfaction rises from 1.68–1.85 under C1 to 1.77–3.30 under C2, and chirality change olfaction rises from 1.33–2.56 to 3.65 and above. Complete data are provided in Table G3.

Q1.3 Structural similarity baseline

Supplementary experiment Table G5 constructs a Morgan fingerprint k-nearest-neighbor predictor that does not invoke any language model, representing the between-group gap that the structure–property relationship alone can produce. The nearest-neighbor predictor's CR is above 1 on all tasks, confirming that structurally similar molecules do tend to have similar properties. However, in 17/20 combinations, the LLM's C2 CR exceeds this reference. On BACE and HIV, some models reach +∞ or above 5×, whereas the nearest-neighbor predictor reaches only 1.83–1.89×. This indicates that LLM dependence on source labels under C2 exceeds the extent that the structure–property relationship alone can explain.

Taskk-NN CRClaude C2GPT C2DeepSeek C2Gemini C2
BACE1.83×+∞3.69×1.45×2.48×
HIV1.89×+∞5.32×0.97×2.17×
Structural change olfaction1.25×3.30×1.89×2.69×1.77×
Chirality change olfaction2.84×+∞16.22×3.65×12.94×
Yield1.22×1.46×2.40×0.98×2.80×

Q1.4 Restricting to correctly reconstructed targets

After retaining only the samples where the target structure is correctly reconstructed, the conclusions remain consistent. The C1 results indicate that failure to update predictions according to structural edits already exists without source labels but varies by task and model. The widespread between-group collapse under C2 persists after restricting to correctly reconstructed targets, with no reduction in magnitude.

TaskConditionClaudeDeepSeekGPTGemini
Structural olfactionC1 / C21.97 / 3.031.77 / 2.551.75 / 2.001.82 / 1.90
Chiral olfactionC1 / C21.56 / +∞2.56 / 3.651.33 / 16.222.20 / 12.94
BACEC1 / C21.93 / +∞0.94 / 1.571.51 / 3.611.34 / 2.49
HIVC1 / C21.75 / +∞1.10 / 0.970.38 / 5.311.34 / 2.13
RetrosynthesisC1 / C20.48 / 9.150.75 / +∞1.73 / +∞0.83 / +∞
Reviewer_rnagOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides an interesting methodology (an identity-blinded diagnostic) for verifying whether LLMs reason properly about molecules and their properties. Instead of providing the target molecule and asking the LLM for the property associated with it, the input consists of a different source molecule plus a structural edit that defines the target. The experiments test two settings, one that does not provide the source molecule's label and one that does. Results are split into pairs where the source and target share a label (consistent) and pairs where they differ (inconsistent). In a further experiment, only the source label is replaced with a false one, while the target and ground truth stay the same.

The experiments reveal that LLMs rely mostly on the source molecule and the provided contextual data rather than on the structural edit. This is why the consistent–inconsistent gap is so large when the source label is provided. In the most extreme case, Claude reaches 100% accuracy on consistent BACE pairs and 0% on inconsistent ones, which amounts to copying the source label. Swapping the source label also shifts predictions toward the false label.

Critical strengths
  1. A new methodology to hide the target. Defining the target as a source molecule plus a structural edit removes its name and SMILES while preserving the information needed to identify it. The authors show that models can reconstruct the target from this description 86–100% of the time. This provides a simple and convincing way to prevent direct retrieval of the target from memory. The design is applicable and can extend to other scientific domains where targets can be defined as edits of known entities.
  2. A controlled intervention. The label swap changes only the source label while keeping the target, edit, and ground truth fixed. Any change in prediction can therefore be attributed to the supplied label. This provides concrete evidence of the LLM's reliance on the label.
  3. Experiments on six tasks. The framework covers six tasks with binary, set-valued, generative (retrosynthesis), and continuous (yield) outputs and tests four frontier LLMs.
Critical weaknesses
  1. Limited novelty of the findings. It is well know that LLMs copy labels supplied in context (https://arxiv.org/abs/2303.03846). Similarly, the failure of models on structurally similar pairs with different labels is well known. It is therefore unclear which existing assumption the results challenge.
  2. Incorrect label as a fact. In experiment C3, the false source label may be presented to the model as fact. A model that follows it may simply be trusting the prompt rather than failing to reason about molecular structure. .
  3. LLMs are already weak. Table 3 shows that baselines are already weak (near chance). While this confirms the copying effect it undermines the underlying thesis of the paper on a need for diagnosis, and contradicts "LLMs score weel on benchmarks".
  4. Limited model scope. Only general-purpose LLMs are evaluated. Chemistry-specialized language models are trained on molecular structures and properties, and may rely less on the source label. Without them, it is unclear whether the conclusions apply to chemical language models in general or only to general-purpose LLMs.
Questions for authors
  1. I'm not sure I entirely understand the wrong source label experiment . Are you feeding the LLM incorrect information? Is that information available to them, e.g., "the label may not be accurate"? If you are feeding it incorrect information, are you really testing reasoning?

  2. Can you provide additional observations on which tasks are observed to better follow structure, and which are not (e.g. Figure 5), and add that as a key contribution?

7 comments

We thank the reviewer for raising these questions.

W1 On novelty

We thank the reviewer for pointing out the relevant literature. We agree that LLMs copying contextual labels is a known phenomenon. The assumption this paper challenges is not "whether LLMs copy labels" but rather the default assumption underlying chemical benchmarks, namely that accuracy reflects structure–property reasoning grounded in target structures. This paper differs from existing findings in two respects.

  1. Source-structure dependence that does not rely on any labels. Neither C0 nor C1 provides any labels, and the only difference is how the target is presented. On the same set of samples, CR>1.2 occurs in only 3/24 combinations under C0 but rises to 14/24 under C1, with 10/24 reaching statistical significance, and C1 exceeding C0 in 20/24 combinations. This phenomenon has not been observed in prior work.

  2. Implications for benchmark evaluation. In evaluations that directly provide the target molecule's SMILES, absolute metrics alone cannot determine how many predictions rely on structure–property reasoning and how many rely on shortcuts. Few-shot and retrieval-augmented chemical evaluations go further by placing structurally similar molecules and their labels into the context. Supplementary experiment Appendix G6 shows that even when the target structure is fully visible, model predictions remain primarily determined by the labels of similar molecules.

Therefore, the contribution of this paper is not in repeating the known phenomenon that "LLMs copy labels" but in proposing a quantifiable diagnostic method that reveals that standard benchmark accuracy cannot be entirely attributed to structure–property reasoning.

W2 and Q1 On the false source label experiment

Does C3 provide incorrect information? Yes. The source label in C3 is a false label, and the prompt does not indicate that "the label may be inaccurate." C3 itself does not test reasoning. It is a counterfactual intervention experiment that fixes the source structure, edit, and target on top of C2 and changes only the source label value, measuring the causal effect of the source label on predictions.

The reasoning conclusions do not depend on incorrect information. C0, C1, and C2 contain no false information, and C2 provides the true label. In the inconsistent group of C2, the source label is true and is explicitly attributed to the source molecule, and the model only needs to recognize from the edit that the property has changed. The result is that Claude-Opus-4.8 achieves 100% accuracy in the consistent group and 0% in the inconsistent group on both BACE and HIV, meaning that it outputs the source molecule's label for every sample. C0, C1, and C2 together establish the conclusions about LLM reasoning. C3, as a contrast to C2, serves as a causal intervention tool to quantify the causal contribution of the source label to predictions in C2.

W3 On weak baseline performance

"LLMs perform well on benchmarks" is the claim this paper tests, not a premise of this paper. Prior work has reported that LLMs achieve competitive performance on various chemical prediction tasks (Guo et al., 2023), matching or exceeding conventional methods in some few-shot settings (Baker et al., 2025). This paper tests whether such scores can be interpreted as reasoning based on structure–property relationships. In our tests, some tasks do show nontrivial performance. Retrosynthesis requires generating SMILES that exactly match the true reactants, and the exact match rates of the four models range from 34.9% to 48.8%. The two olfactory tasks involve open-vocabulary multi-label descriptor prediction, with F1 reaching 0.198–0.400. Both types of tasks are far beyond what random guessing can achieve.

Retaining weak tasks is an intentional design choice. We intentionally retained tasks spanning a range from strong C0 performance to near-random to below simple baselines, with the goal of testing whether source information dependence is related to a model's absolute performance. Table G2a shows that the two have no correspondence, with C1 exceeding C0 in 20/24 combinations and C2 showing significant collapse in 22/24 combinations. This demonstrates that regardless of benchmark score level, the score itself cannot tell us whether the model is reasoning from structure, which is precisely where the necessity of diagnosis lies.

W4 On chemistry-specific language models

We thank the reviewer for the suggestion. We have additionally evaluated ChemDFM-v2.0-14B and ChemLLM-7B-Chat. The results are consistent with the experimental findings in the main text. The C1 results vary across tasks and models, while C2 still produces pronounced intervention effects on multiple tasks, indicating that this phenomenon is not unique to general-purpose LLMs. +∞ indicates that no sample in the inconsistent group was successful, and 0/0 in the retrosynthesis task indicates that no successful samples were available for computing the ratio under that condition.

Model & ConditionBACEHIVStructural odorChiral odorRetrosynthesisYield
ChemDFM C11.460.141.651.00/00.83
ChemDFM C268+∞+∞+∞0/04.75
ChemLLM C10.861.291.380.360/00.84
ChemLLM C22.210.5614.838.30/02.9

Q2 On the degree of structure adherence across tasks

We have added summaries along the task and model dimensions, presented in Table G2a and G2b.

Table G2a Task-level summary

TaskC0 absolute performanceC1 CR > 1.2C2 CR > 1.2
Structural change olfactionF1 0.326–0.4004/44/4
Chirality change olfactionF1 0.198–0.2724/44/4
BACEAcc 51.5%–57.4%3/44/4
HIVAcc 31.4%–61.2%2/43/4
RetrosynthesisEM 34.9%–48.8%1/44/4
YieldMAE 32.34–56.460/43/4

The six tasks can be divided into three categories by C0 absolute performance, and they differ in their collapse patterns under C1 and C2.

  1. Olfactory and retrosynthesis tasks with C0 well above chance. The two olfactory tasks already show collapse in 4/4 models under C1, making them the least structure-adherent among the six tasks. Their C0 performance is well above random, yet after identity blinding the models already show clear bias toward the source molecule, indicating that higher benchmark scores do not mean the model is reasoning from structure. Olfactory labels are natural-language descriptors, and LLMs readily transfer the source molecule's descriptors directly to the target. Retrosynthesis shows collapse in only 1/4 under C1, but this jumps to 4/4 under C2 with the inconsistent group failing completely for three models. Once source reactant labels are provided, structural information is overridden.

  2. BACE and HIV with model-dependent C0 performance. C0 accuracy is near random level, with collapse in 3/4 and 2/4 under C1, rising to 4/4 and 3/4 under C2.

  3. Yield prediction with poor C0 performance. C1 shows 0/4 collapse, but here the CR near 1 reflects both groups failing equally. Under C2, 3/4 models exhibit collapse, indicating that although models cannot predict yield based on formulation changes, they directly reference the source reaction's yield.

These results show that there is no correspondence between C0 absolute performance and the degree of structure adherence. The olfactory tasks with the best C0 performance show the most widespread collapse, while the yield task with the worst C0 performance shows no collapse in the absence of labels. This demonstrates precisely that benchmark scores alone cannot determine whether a model is reasoning from structure.

I thank the authors for reporting additional results. I have a few remaining concerns:

  1. The discrepancy in C1 only appears in the classification tasks (BACE, HIV, and the two odor tasks) and is not detected in retrosynthesis or in reaction yield. Why is there no C1 effect in retrosynthesis and yield, and how far do the conclusions extend beyond classification-style benchmarks?

  2. What is the class composition of the label-consistent groups for BACE and HIV (how many pairs are both active or both inactive)?

  3. Why is accuracy reported for HIV and BACE, whereas F1 is often used for these imbalanced datasets (https://arxiv.org/pdf/2305.18365)? Could class bias alone result in the CR differences?

Also the results on the ChemLLM , show strongly reversed CRs on HIV between C1 and C2!

Comments on the paper

2 comments

We thank all reviewers for recognizing the importance of the research question, the source-plus-edit diagnostic design, the multi-task experimental coverage, and the label intervention and structure reconstruction analyses. The reviewers' suggestions have been very helpful and have enabled us to substantiate the claims of this paper more thoroughly.

Following the reviewers' suggestions, we have added six groups of experiments in Appendix G of the updated manuscript.

  1. Chemistry-specific language models (G.1). We evaluate ChemDFM and ChemLLM on all six tasks.

  2. Collapse ratio summary (G.2). We summarize collapse ratios along the task dimension (G2a) and the model dimension (G2b).

  3. C0 group comparison (G.3). Under the C0 condition, which directly provides only the target SMILES, we partition performance into the label-consistent and label-inconsistent groups to rule out inherent sample difficulty differences.

  4. Far-distance molecular pairs (G.4). We construct molecular pairs with Morgan Tanimoto similarity between 0.20 and 0.55 and run the full C0-through-C2 diagnostic on the HIV and structural change olfaction tasks.

  5. Structural nearest-neighbor baseline (G.5). We construct a Morgan fingerprint k-nearest-neighbor predictor that does not invoke any language model, quantifying the between-group gap that the structure–property relationship alone can explain.

  6. Source-label intervention with direct target structure (G.6). We retain the source structure, source label, and edit description while additionally providing the target molecule's SMILES directly (C2-direct / C3-direct).

Central claims of this paper

The hypothesis tested in this paper is that LLM accuracy on chemical prediction benchmarks reflects structure–property reasoning grounded in target structures. Several prior studies have raised doubts about this assumption (see Introduction), and this paper provides a systematic diagnostic method together with experimental validation. The diagnostic reveals two layers of dependence, each corresponding to a reliability concern for a different evaluation paradigm.

  1. Source-structure dependence. Neither C0 nor C1 provides any labels, and the only difference is how the target molecule is presented. The same set of samples shows no systematic between-group gap under C0, as shown in Table G3. After identity blinding, the collapse ratio under C1 exceeds that under C0 in 20/24 model–task combinations, with 10/24 reaching statistical significance. This layer of dependence directly challenges the standard benchmark evaluation approach, because absolute metrics alone cannot distinguish structure–property reasoning from shortcuts such as memorization.

  2. Source-label dependence. After providing source molecule labels, 22/24 combinations exhibit significant collapse, covering all six tasks and all four models. Even when the target structure is provided directly, this dependence does not diminish, as shown in Table G6. This layer of dependence challenges few-shot and retrieval-augmented chemical evaluations, which place similar molecules and their labels into the context. Table G6 demonstrates that model predictions remain primarily determined by source labels even when the target structure is fully visible.

Area_Chair_CxAaOpenReview ↗

Reminder to Reviewers: Please Engage in the Discussion

Dear Reviewers,

As the authors have now submitted their rebuttal, I would kindly ask you to read their responses and engage in the discussion.

Thank you!

Best regards,
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.