- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces the Identity-Blinded Reasoning Diagnostic framework to investigate whether LLMs rely on transferable structure–property reasoning or contextual information when performing chemical prediction tasks. Instead of directly providing the target molecule's identity, the framework describes the target through a known source molecule and an explicit structural modification. The authors further examine the influence of source labels through controlled interventions and evaluate the framework across six tasks covering odor prediction, molecular properties, retrosynthesis, and reaction yield prediction. The findings reveal that model predictions can be strongly influenced by source-related information, suggesting that high benchmark accuracy does not necessarily indicate reliable chemical reasoning capabilities.
Critical strengths
The paper offers a valuable diagnostic perspective by examining the reliability of LLM-based chemical predictions beyond conventional benchmark accuracy. The identity-blinded evaluation design is a notable strength, as it reduces direct access to target identity information while retaining structural relationships. The controlled source-label interventions provide a systematic way to investigate the influence of contextual cues on model predictions. In addition, the evaluation across six tasks spanning different chemical domains broadens the relevance of the findings and highlights important limitations in current LLM prediction capabilities.
Critical weaknesses
Although the identity-blinded framework reduces direct access to target information, it does not completely eliminate the possibility of recovering target identity from the source structure and structural modifications. Therefore, the observed prediction patterns may not fully distinguish memorisation from genuine chemical reasoning. In addition, the dependence on source labels could partly reflect legitimate structure–property relationships rather than inappropriate reliance on contextual cues. Further analysis is needed to clarify these effects and establish the robustness of the diagnostic conclusions across different models and experimental conditions.
Questions for authors
-
How do the authors ensure that the identity-blinded setting effectively prevents models from recovering target identity through the source structure and structural modifications?
-
How can the authors distinguish inappropriate dependence on source labels from legitimate chemical relationships between source and target molecules?
-
How consistent are the diagnostic findings across different LLMs, and could additional experiments help establish the robustness and generalisability of the proposed framework?
4 comments
Q3.1 Diagnostic consistency across models and framework robustness
Supplementary experiment Table G2b summarizes collapse ratios along the model dimension.
- Source-label dependence is highly consistent across models. The C2 CR of all four models exceeds C0 CR on all six tasks, and C2 CR>1.2 covers 22/24 combinations.
- After directly providing the target structure in G6, all 15/15 combinations across the three tested models still have CR>1.2.
- The chemistry-specific models ChemDFM and ChemLLM have C2 CR>1.2 in 5/5 and 4/5 of the computable task combinations respectively, as shown in G1.
- Source-structure dependence under C1 varies by task and model. DeepSeek-V4-Pro is the weakest with 2/6 significant, while Claude, GPT, and Gemini all reach 4/6, but the two olfactory tasks consistently appear across all four models.
| Model | C1 sig. | C2 sig. | C1>C0 | C2>C1 | C2>C0 | C2>C1>C0 |
|---|---|---|---|---|---|---|
| Claude-Opus-4.8 | 4/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 |
| DeepSeek-V4-Pro | 2/6 | 4/6 | 4/6 | 5/6 | 6/6 | 3/6 |
| GPT-5.5 | 4/6 | 6/6 | 5/6 | 6/6 | 6/6 | 5/6 |
| Gemini-3.1-Pro | 4/6 | 6/6 | 5/6 | 5/6 | 6/6 | 4/6 |
Q3.2 Can additional experiments support robustness and generalizability?
Yes. The supplementary experiments we have added correspond to the following robustness dimensions.
- Model type. Chemistry-specific models ChemDFM and ChemLLM, presented in G1.
- Sample difficulty. Comparison of label-consistent and label-inconsistent groups under C0, presented in G3.
- Structural distance. Far-distance molecular pairs with Tanimoto 0.20–0.55, presented in G4.
- Structure–property relationship reference. Morgan fingerprint nearest-neighbor baseline, presented in G5.
- Identity visibility. Source-label intervention with direct target SMILES, presented in G6.
Across all these dimensions, source-label dependence remains stable. Source-structure dependence is consistent across models on the olfactory tasks and varies by model on other tasks, which is consistent with the conclusions in the main text.
We have provided targeted responses to the reviewer's questions, addressing the recoverability of target identity, the distinction between legitimate structure–property reference and inappropriate source dependence, and the robustness of the findings across models and supplementary experiments. As the discussion period is drawing to a close, we would like to confirm whether these responses have adequately addressed the reviewer's concerns. We would be happy to provide further clarification on any remaining issues.
Thank you for your review. We have responded to each of your three questions in the rebuttal.
- Regarding whether target identity can be recovered, we clarified the design goal of identity blinding and directly provided the target SMILES in Appendix G6, confirming that the conclusions hold regardless of whether the identity can be recovered.
- Regarding distinguishing legitimate structure–property reference from inappropriate dependence, we added three groups of supplementary experiments including C0 group comparison, Morgan nearest-neighbor baseline, and far-distance molecular pairs, presented in Appendix G3, G5, and G4 respectively, confirming that the LLM's dependence on source labels exceeds the range that structural similarity alone can explain.
- Regarding consistency across different models, we added evaluations on chemistry-specific models and summarized the results along both the model and task dimensions, presented in Appendix G1 and G2 respectively, confirming that source-label dependence is consistent in direction across models and tasks.
The discussion period is drawing to a close. We kindly ask you to take a moment to review these responses and let us know whether they have addressed your concerns. If anything remains unclear, please point out the specific issue and we will provide further clarification immediately.
W1 and Q1 Identity blinding and recovery of target identity
The reviewer's observation is correct. Identity blinding does not completely prevent models from recovering the target identity. In fact, if the target identity were completely hidden, the prediction task could not be defined. The design goal of this paper is not to prevent recovery but to avoid presenting the target's SMILES or name directly in the prompt, thereby removing the shortcut of direct retrieval while preserving all structural information needed for reasoning. Structure reconstruction experiments show that models reconstruct the target from the source molecule plus the edit with accuracy ranging from 86.05% to 100%.
It is precisely because the target can be recovered that this design can serve as a diagnostic. We demonstrate from two perspectives that the conclusions are not affected by whether or not the identity is recovered.
The comparison between C0 and C1. If models first recover the target identity and then perform structure–property reasoning based on the target, the C1 results should be close to C0. However, on the same set of samples, CR>1.2 occurs in only 3/24 combinations under C0, with no systematic between-group gap. C1 merely changes the target presentation to source molecule plus edit without providing any labels, yet CR>1.2 rises to 14/24, with 10/24 reaching statistical significance, and C1 exceeding C0 in 20/24 combinations.
When the target identity is directly provided, as shown in G6. When the target SMILES is directly provided in the prompt, the model's memory and knowledge about the target can be fully invoked, yet source-label dependence still persists. C2-direct has CR>1.2 in 15/15 combinations, and Claude still has 90.4% and 94.2% of predictions switching with the source label on BACE and HIV. Even when the model can access the target identity, it does not use the target identity to override the source label.
Therefore, regardless of whether the identity can be recovered, scores on standard benchmarks are insufficient to demonstrate that LLMs complete predictions based on structure–property relationships.
W1 and Q2 Distinguishing inappropriate dependence from legitimate structure–property relationships
We agree that the properties of a source molecule can serve as a reasonable reference. We distinguish reasonable reference from inappropriate dependence at three levels.
Without labels. Neither C0 nor C1 provides source labels. On the same set of samples, there is no systematic consistent-group advantage under C0, with CR>1.2 in only 3/24 combinations. After switching to the identity-blinded presentation, C1 rises to 14/24. Legitimate structure–property relationships do not change when the target's presentation format changes, and structure reconstruction experiments show that models can recover the target identity under C1. Therefore, this change cannot be explained by legitimate structure–property relationships.
With true labels, the magnitude exceeds the range that the structure–property relationship can support. Under C2, 22/24 combinations have CR>1.2. Claude achieves 100% accuracy in the consistent group and 0% in the inconsistent group on BACE and HIV, identical to the result of directly copying the source label. The Morgan fingerprint nearest-neighbor predictor in Table G5, which does not invoke any language model, represents the between-group gap that the structure–property relationship alone can produce. In 17/20 combinations, the LLM's C2 CR exceeds this reference. The complete comparison is provided in Table G5.
When the target structure is visible, the dependence persists. In G6, the target SMILES is directly provided, and C2-direct still has CR>1.2 in 15/15 combinations. If the model reasonably treats the source label as a reference, it should rely more on the target itself when the target structure is fully visible, yet the actual label switching rates are essentially the same as under identity blinding.
In addition, Table G4 uses far-distance molecular pairs with source–target Tanimoto similarity between 0.20 and 0.55. The CR under C1 and C2 still increases relative to C0 in all 6/6 combinations, although the magnitude is weaker than with near-distance pairs. This indicates that the dependence is not limited to cases of high structural similarity.