acceptodds
Under review as a conference paper at ICLR 2027

Same molecule, different knowledge: Cross-identifier binding failures in LLMs for chemistry

Abstract

Chemical AI workflows routinely represent the same molecule through multiple co-referring identifiers, including trivial names, IUPAC names, and SMILES, implicitly assuming that these representations provide interchangeable access to an LLM's knowledge. To test this assumption, we first introduce a controlled benchmark and show that it does not hold. Across foundation models spanning 7B to 72B parameters, even chemistry-specialized models, co-referring molecular identifiers consistently expose different subsets of the model's knowledge. We then conduct a series of controlled behavioral, representational, and causal analyses to characterize this failure. We show that even when a model can identify a molecule from its SMILES and retrieve a fact through the corresponding name, the same fact can remain inaccessible directly from SMILES. We term this phenomenon cross-identifier binding failure. Further analysis of internal representations, conflicting identifiers, and parameter editing shows that neither improved chemical representation competence nor modification of individual facts is sufficient to establish reliable access across identifiers. Together, these results indicate that the core failure lies in the binding between an identifier and knowledge already present in the model. Motivated by this diagnosis, we introduce Binding Self-Distillation, a simple mitigation that uses lexical identity during training to transfer the model's existing knowledge access to the structural route. This enables factual retrieval directly from SMILES at inference time, without requiring a name lookup or training the underlying model from scratch.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
LLMs, chemistry, molecular representations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FY4ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tests an assumption that chemical AI pipelines implicitly rely on but that has not been systematically verified: whether co-referring molecular identifiers (trivial name, IUPAC name, SMILES) provide interchangeable access to an LLM's knowledge. The authors build ChemIDBench — 896 carefully deduplicated approved drugs from ChEMBL, each with three matched identifiers and 7 query types (entity-associated: identification, therapeutic indication; structure-computable: formula, benzene ring, nitrogen count, ring count, heavy atoms) — and evaluate seven 7B–72B models, including chemistry-specialized ChemDFM variants. The core finding is a systematic double dissociation: trivial names give far better access to entity-associated facts (indication accuracy collapses from 38.8% under the name to 2.1% under SMILES), while SMILES is better for structure-computable properties (benzene detection 76.7%→94.6%). Crucially, factual retrieval fails even when the model demonstrably recognizes the molecule from SMILES (both 2AFC and free-form criteria) — decoupling entity recognition, knowledge possession, and identifier-to-knowledge binding, which the authors name cross-identifier binding failure. Converging evidence is provided at three levels: behavioral (lexical bridge restores 91.8–99.4%; conflicting-identifier experiments), representational (Name–SMILES top-1 retrieval only 0.7–2.8% vs. up to 15% for IUPAC–SMILES in chemistry-specialized models), and causal (activation patching restores 76.4–138.4% of the name–SMILES gap with negative controls). Standard knowledge editors (ROME/FT) fail to reliably propagate edits across identifiers. Finally, Binding Self-Distillation distills the model's own name-conditioned predictive behavior into a LoRA adapter on the SMILES route via full-vocabulary KL, raising retrieved facts per entity from ~0.05 to 0.68–0.87 across all seven models without needing the name at inference. Appendix I replicates the phenomenon in the protein domain with UniProt accessions.

Critical strengths

1,Whether equivalent identifiers equivalently access knowledge is a real, overlooked reliability problem for LLMs in scientific workflows. The paper cleanly decouples recognition, knowledge possession, and access into three testable capabilities; "cross-identifier binding failure" is a crisp, falsifiable construct with direct implications for LLM4Science, knowledge editing, and agentic tool routing (resolve-then-query). 2,Behavioral (recognition controls, lexical bridge, conflicting identifiers) → representational (cross-identifier top-1 retrieval, chemistry-specialization contrasts) → causal (activation patching with different-molecule controls) — the three layers interlock into a coherent story. The patching controls (different molecule + different gold answer, all negative) make the causal argument more rigorous than typical work in this space. 3,Paired design (same molecule, same property, only the identifier changes) with McNemar tests (p<10⁻⁷); strict entity cleaning in benchmark construction (salts, multi-fragment, duplicate names/structures removed); 2AFC filtered across three independent decoy conditions; editing results corrected for locality leakage with bootstrap CIs; downstream evaluation prompts excluded from BSD training stems. Methodological care is above average.

Critical weaknesses

1,BSD trains a separate LoRA adapter per molecule (120 steps, rank 8) with no parameter sharing across molecules, and the authors themselves defer generalization to unseen entities to future work. As presented, it is an existence proof rather than a usable method: 896 molecules require 896 adapters, and transfer is only possible for molecules whose name route already answers indication correctly (leaving n=47 for ChemDFM-v1.5-8B). Meanwhile the lookup routing of Section 4.3 is the actually practical, training-free remedy. Suggestions: (a) include at least one shared-adapter variant and report cross-entity generalization; (b) tone down the "mitigation" framing in the main text. 2,BSD is only evaluated on molecules for which the model already retrieves the indication correctly via the trivial name. While logically necessary (there is no source behavior to transfer otherwise), this means: (a) reported gains are conditioned on a subset where knowledge exists and is name-reachable, and do not extrapolate to all 896 molecules; (b) selection uses retrieval success on the same relation (indication) as the gating criterion, risking circularity. Cross-relation transfer is reported (target 0→27.6%, MoA 0→5.0%, ATC 3.3→11.4%) but the latter two are quite low in absolute terms. A selected-vs-unselected contrast and more prominent reporting of absolute cross-relation numbers would strengthen the claim. 3,Even the name route only reaches 22–44% indication accuracy, and the SMILES route is near zero — the whole phenomenon sits on a substrate of weakly memorized facts. Approved ChEMBL drugs almost certainly appear in pretraining corpora, and name-route accessibility is plausibly correlated with drug popularity in the corpus (aspirin vs. obscure compounds). The paper neither stratifies by name frequency/popularity nor discusses contamination. This does not invalidate the conclusion, but bounds its scope: how much of binding failure is a restatement of "the knowledge is barely in the model at all"? A frequency-stratified analysis would help. 4,Activation patching replaces the residual stream at a single site (final prompt token) layer-by-layer; the peak depth varies across models (relative depth 0.50–0.83) and is acknowledged as merely descriptive. Recovery >100% (up to 138.4%) is left unexplained (what does overshoot mean?). No attention-head/MLP-level localization, and no account of what the lexical state actually carries. Adequate as causal evidence; preliminary as mechanistic explanation.

Questions for authors

see Weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer FY4R (1/2)

We thank the reviewer for their constructive comments and valuable feedback.

W1, BSD generalization.

Thank you for raising this point. Our original per-molecule BSD experiments were designed to establish whether existing factual knowledge could be made accessible through structural identifiers by adapting the model's parameters, rather than introducing new factual knowledge/train model from scratch. To directly address the reviewer's concerns about scalability and generalization, we conducted an additional experiment using a single shared adapter on Qwen2.5-7B-Instruct. Full experimental setup and training details are provided in Appendix H.8 of the revised manuscript.

RelationBefore BSDShared BSD
Indication (75)1.33%16.00%
Target (61)1.64%6.56%
MoA (65)0.00%1.54%
ATC (69)2.90%2.90%
Overall (270)1.48%7.04%

On unseen molecules, the shared adapter achieves a +5.56 percentage-point improvement, statistically significant at the molecule level (95% paired-bootstrap CI: [+2.77, +8.63] pp; paired randomization p = 0.000488). The largest improvement occurs in indication (1.33% → 16.00%), with an additional improvement in target retrieval. We have incorporated this experiment into Section 5 and Appendix H.8 of the revised manuscript. Following the reviewer's suggestion, we have toned down the mitigation framing in the revised submission.

W2, Selection bias.

Our original indication-based selection was intentional: BSD aims to transfer factual knowledge already accessible through a molecule's name to its structural representation, without external factual supervision. Selecting molecules with verified name-accessible knowledge therefore provides a controlled setting for testing whether this transfer is possible. To address the reviewer's concern about selection bias, we conducted an additional selected-versus-unselected experiment on Qwen2-72B-Instruct, extending the original 435 selected molecules (Table 8) with 100 previously excluded molecules. We divided the excluded molecules into two groups: 50 whose indications were incorrect based on their trivial names but who had at least one other correctly retrieved fact (target, MoA, or ATC), and 50 without verified name-accessible facts. All groups received the same molecule-specific BSD adaptation protocol. The experimental setup and full results are reported in Section 5 and Appendix H.9 of the revised manuscript.

SMILES-conditioned factual accuracy before → after BSD:

RelationOriginally selectedExcluded: other name-accessible factsExcluded: no verified facts
Indication0% → 17.0%0% → 2.0%0% → 0%
Target1.2% → 39.5%0% → 51.2%0% → 12.5%
MoA0% → 30.6%0% → 45.7%0% → 0%
ATC0.2% → 13.1%0% → 30.6%0% → 4.3%

Crucially, substantial cross-relation improvements occur even among molecules excluded by the original indication criterion. For molecules with other name-accessible facts, target accuracy increases from 0% to 51.2%, MoA from 0% to 45.7%, and ATC from 0% to 30.6%. Thus, BSD's benefits are not restricted to molecules whose indications were correctly answered through their names. We also investigated whether BSD's improvements depend on whether the model can already retrieve the specific fact being evaluated through the trivial name. For each relation (target, MoA, and ATC), we divided the original 435 molecules into two groups: those for which Qwen2-72B correctly answered that relation using the trivial name, and those for which it did not. We then measured the improvement in SMILES-conditioned accuracy after BSD for each group. The improvements were consistently larger when the corresponding fact was already accessible through the name: target +58.9 vs. +11.8 percentage points, MoA +49.3 vs. +26.0, and ATC +66.2 vs. +2.3 (name-correct vs. name-incorrect). These results show that BSD is particularly effective at improving access to facts the model can already retrieve through the trivial name, regardless of whether indication correctness was used as the original selection criterion.

W3, Molecular popularity.

To address the reviewer's concern that cross-identifier failures may primarily reflect limited factual memorization or differences in drug popularity, we conducted an additional frequency-stratified analysis across all 896 molecules and all seven models. We used PubMed exact-name mention frequency as a measure of molecular popularity and divided the molecules into low-, medium-, and high-popularity groups (299/298/299 molecules). The experimental setup and full results are reported in Section 2.3 and Appendix B.5 of the revised manuscript. See the next comment below.

AuthorsOpenReview ↗

Response to Reviewer FY4R (2/2)

ModelName accuracy (low → high)SMILES accuracy (high)SMILES failure among name-correct (high)
Qwen2.5-7B25.1% → 36.1%2.0%96.3%
Qwen2.5-14B30.1% → 40.5%2.0%95.9%
Qwen2-72B36.8% → 45.2%1.0%98.5%
ChemDFM-v1.5-8B1.0% → 10.4%0.3%96.8%
ChemDFM-v2.0-14B26.4% → 45.5%3.0%95.6%
Llama-3-8B28.4% → 43.5%1.0%97.7%
Llama-3.1-8B29.1% → 43.1%2.7%96.1%

Across all seven models, name-conditioned indication accuracy increases from the low- to the high-popularity group. This confirms that differences in drug familiarity are relevant to factual retrieval and should be distinguished from cross-identifier binding failure. However, the central finding is that greater name-conditioned factual accessibility does not translate into comparable SMILES-conditioned accessibility. Even in the highest-popularity group, SMILES accuracy remains between 0.3% and 3.0% across models. More importantly, among highly popular molecules whose indications are correctly retrieved through their trivial names, 95.6–98.5% still fail when queried through SMILES. Consequently, the observed failure cannot be attributed solely to the model not knowing the correct fact or to the molecule being relatively unfamiliar. These findings reinforce our central distinction between factual knowledge availability and cross-identifier knowledge access.

W4, Activation-patching.

To address the reviewer's concerns, we conducted additional controlled activation-patching experiments on Qwen2.5-7B-Instruct and ChemDFM-v1.5-8B.

1. What does the patched representation carry? Our original experiment patched the final prompt-token representation from a name-conditioned indication question. To distinguish molecule-specific information from computation associated with the indication question or answer, we introduced an indication-free source containing only the molecular name, with no indication question or answer, and patched its representation into the SMILES entity position. This intervention recovered 73.0% of the normalized factual log-probability gap on Qwen (layer 11) and 95.6% on ChemDFM (layer 8). Crucially, patching representations from a different molecule instead produced −60.3% and −51.6% recovery, respectively (both p < 10⁻⁶⁰). These controlled results show that substantial factual recovery is molecule-specific and does not require indication-conditioned information in the source, supporting the role of a question-independent, name-associated representation in enabling factual access. Full experimental details and results are provided in Section 4.1 and Appendix D.6 of the revised manuscript.

2. Where does recovery occur, and which components contribute? We extended the original residual-stream interventions to separately patch attention and MLP outputs at the entity position and final prompt token.

Model / positionFull residual stateAttention outputMLP output
Qwen / entity73.0% (L11)0.8%−0.3%
Qwen / final token104.9% (L21)79.7%54.3%
ChemDFM / entity95.6% (L8)1.3%8.8%
ChemDFM / final token76.4% (L16)58.8%17.9%

In both models, entity-position recovery peaks earlier than final-token recovery: relative depth 0.39 versus 0.75 for Qwen and 0.25 versus 0.50 for ChemDFM. At the entity position, patching the accumulated residual representation produces substantial recovery, whereas isolated component-output interventions have much smaller effects. At the final token, attention-output interventions alone recover 79.7% and 58.8% of the gap on Qwen and ChemDFM, respectively. Full experimental details and results are provided in Section 4.1 and Appendix D.7 of the revised manuscript. These interventions provide a more fine-grained picture: molecule-specific information is accessible in earlier entity-position representations, while later attention-mediated computation makes a substantial contribution to factual prediction.

3. Why can recovery exceed 100%? As already explained in Appendix D.4 of our original submission, normalized recovery measures the fraction of the original name–SMILES gold-answer log-probability gap closed by patching, rather than factual accuracy. Recovery exceeds 100% when the patched model assigns a higher log probability to the correct answer than the original name-conditioned model.

We hope our responses adequately address the reviewer's concerns.

Dear Reviewer FY4R,

Just a gentle reminder, as the discussion deadline is approaching. We hope our response and the additional clarifications/experiments addressed your concerns, and we would be grateful if you could reconsider your assessment in light of our revised response. We would also greatly appreciate any further feedback you may have before the deadline. Thank you again for your time and consideration.

Dear Reviewer FY4R,

As the discussion period is coming to an end and we did not receive your feedback on our rebuttal, we would like to briefly summarize the additional experiments and revisions we made to carefully address all four concerns raised in your review:

  1. BSD scalability and generalization: We introduced a shared-adapter variant trained on 250 molecules and evaluated on 75 unseen molecules. Indication accuracy increased from 1.33% to 16.00%, while overall factual accuracy improved from 1.48% to 7.04%. Following your suggestion, we also revised the Introduction and Abstract to tone down the mitigation claims, as reflected in the updated manuscript.

  2. Selection bias: We evaluated 100 previously excluded molecules and showed substantial cross-relation improvements even when the original indication-based selection criterion was not satisfied.

  3. Molecular popularity: We conducted a frequency-stratified analysis across all 896 molecules and seven models. Even among highly popular molecules whose indications are correctly retrieved through their names, 95.6–98.5% still fail through SMILES.

  4. Activation patching: We added indication-free, molecule-specific controls and attention/MLP-level interventions, providing further causal evidence and a more detailed understanding of the mechanism. We also clarified why normalized recovery can exceed 100%.

All these experiments and clarifications are incorporated into the revised manuscript.

We appreciate your recognition of the scientific motivation and experimental contributions of our work. We have carefully addressed each of your concerns and hope these additional results strengthen your assessment of the paper. We would be grateful if you could take these additional results into account in your final assessment and consider increasing your score.

Thank you again for your time.

Reviewer_AUsaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents an extensive set of systematic experiments demonstrating that LLMs, including chemistry-specialized LLMs, struggle to bind different chemical identifiers of the same molecule in a way that yields comparable performance on the same prediction task.

Critical strengths

The experimental design is systematic and careful. The experiments convincingly establish the existence of the problem, investigate its underlying causes, and explore three distinct mitigation strategies: lookup-based lexical routing, knowledge editing, and distillation-based training.

Critical weaknesses

The problem investigated is rather self-evident, and its existence is already close to a consensus within the field. The paper does establish this phenomenon more systematically and provides analyses of its causes and potential remedies. However, it offers limited additional insight or inspiration for the field and does not introduce a novel or superior solution. Its overall contribution to the field is therefore relatively limited.

Questions for authors

One of the paper’s most interesting findings is that, even within the subset of molecules for which the model successfully recognizes the correspondence between different identifiers, accuracy still differs substantially across representations (Table 2, particularly its last three columns). A natural question is whether these subsets would exhibit better performance and smaller cross-identifier gaps if, instead of being required to answer directly, the model were prompted to first generate the trivial name and then answer the question. More generally, would allowing the model to reason before answering, or evaluating reasoning-capable models, improve performance and reduce these gaps? At a broader level, could equipping models with reasoning capabilities be a viable approach to addressing the cross-identifier binding failure identified in this work?

4 comments
AuthorsOpenReview ↗

Response to Reviewer AUsa (1/2)

We thank the reviewer for their feedback.

W1, Novelty and scientific significance.

We respectfully disagree that the central problem investigated in our work is self-evident or already established in the literature. While previous research has examined how molecular representations affect LLM behavior, our work identifies and systematically investigates a more specific failure: an LLM can successfully recognize a molecule and retrieve factual knowledge through its name, yet fail to access that same knowledge through an equivalent structural identifier. This distinction is central to our contribution. Observing differences across molecular representations does not establish whether they arise from unsuccessful entity recognition, missing factual knowledge, or a failure to access knowledge already available to the model. Our experiments distinguish these possibilities and reveal consequences for factual retrieval and knowledge editing that go beyond the existing literature. The closest related works, Baker et al. (2025) and Yan et al. (2025), are already cited and discussed in our manuscript. Baker et al. investigate the effects of structure-based molecular representations (e.g., SMILES and IUPAC names) on zero- and few-shot property prediction, while Yan et al. study prediction consistency between SMILES and IUPAC names. In contrast, our work focuses on factual knowledge access across molecular identifiers, particularly between structural representations and commonly used drug names.

Research questionBaker et al.Yan et al.Ours
Representation-dependent prediction performance or consistency✓✓✓
Representation preferences across molecular property types✓—✓
Factual-access failure despite successful entity recognition——✓
Existing factual knowledge inaccessible across identifiers——✓
Causal investigation of cross-identifier factual access——✓
Recovering inaccessible facts through lexical routing——✓
Transferring existing factual access through self-distillation——✓

Our experiments establish why this distinction matters: First, recognition does not guarantee factual access. On ChemDFM-v2.0-14B, among molecules successfully recognized from SMILES, name-conditioned indication accuracy reaches 40.3%, whereas SMILES-conditioned accuracy remains at 3.2%, comparable to the 3.6% observed for unrecognized molecules. Second, the failure cannot be explained simply by molecular obscurity. Across all seven models, 95.6–98.5% of high-popularity molecules answered correctly through their names still fail through SMILES. Third, the problem extends to knowledge editing. Standard editors can successfully modify facts through molecular names without reliably transferring those edits to the corresponding SMILES. Fourth, our activation-patching experiments provide causal evidence: a molecule-specific representation obtained without an indication question substantially restores factual access through SMILES. Importantly, we do not stop at identifying the failure; we also investigate how to overcome it. Lookup-based lexical routing restores 83.6–98.8% of the aggregate name–SMILES factual-access gap across six models. We further propose Binding Self-Distillation (BSD), which transfers the model's own name-conditioned predictive behavior to its SMILES-conditioned route without external factual labels. This distinction is clarified in the revised Introduction and Appendix A. We believe these findings provide substantive scientific insights and concrete methodological directions for improving the reliability of LLMs across equivalent molecule identifiers.

AuthorsOpenReview ↗

Response to Reviewer AUsa (2/2)

Q1, Reasoning and explicit name generation.

Table 2 highlights the distinction underlying the reviewer's question: successful molecular recognition does not guarantee factual retrieval. For ChemDFM-v2.0-14B, among 62 molecules correctly recognized from SMILES, indication accuracy remains only 3.2% through SMILES versus 40.3% through their trivial names. Our original submission also investigates two complementary lexical-resolution strategies. Lookup-based routing (Section 4.3) supplies the correct molecular name and recovers 83.6–98.8% of the aggregate name–SMILES factual-access gap across six models. In contrast, self-resolution (Appendix F.5) requires the model to generate the name itself before factual retrieval, but correct identification remains limited (0.1–12.0% across six models). Importantly, explicitly generating the molecular name substantially improves factual retrieval on the recognized subset. Among the 62 molecules correctly recognized in our original free-form identification experiment, direct SMILES prompting retrieves only 3.2% of indications, whereas explicit self-resolution achieves 29.0% joint name-and-indication success (Appendix F.5, Table 24 of the revised paper). This confirms that explicitly establishing and using lexical identity can substantially improve factual retrieval.

To investigate whether reasoning can overcome this limitation, we additionally evaluate Qwen3-8B and Qwen3.8-27B on 200 molecules per model, comparing direct SMILES, name-first prompting (explicitly prompting the model to identify the name first before answering), and oracle-name access, with thinking enabled and disabled. The exact prompts, experimental settings, and full results are provided in Section 4.3 and Appendix F.6 of the revised manuscript.

ModelThinkingDirect SMILESName-firstOracle name
Qwen3-8BOff0.5%2.0%35.5%
Qwen3-8BOn1.5%2.5%39.5%
Qwen3.8-27BOff0.5%2.5%39.5%
Qwen3.8-27BOn4.5%4.5%41.0%

Thinking improves direct SMILES retrieval in some settings, particularly for Qwen3.8-27B. However, name-first prompting reaches at most 4.5%, compared with 35.5–41.0% under oracle-name access. Thus, explicit name generation and reasoning do not reliably close the overall factual-access gap in the evaluated settings. This reinforces our distinction between recognizing an entity and automatically using its identity for factual retrieval. It also motivates BSD, which learns to transfer existing name-conditioned factual access to SMILES without requiring name resolution at inference time.

We hope our responses adequately address the reviewer's concerns.

Thanks for your thoughtful and detailed response. Unfortunately, it has not persuaded me to revise my assessment of the problem studied in this paper. In my view, the specific failure you investigated and the proposed solution remain too incremental and trivial to warrant a standalone research paper. I therefore maintain my original score.

We respect the reviewer’s assessment, but we remain puzzled that the characterization of our contribution as “incremental and trivial” is not supported by specific scientific arguments or specific prior work. Our rebuttal clarified that representation-dependent predictions and failures to access existing factual knowledge despite successful entity recognition are distinct research questions. We also provided additional experiments on explicit name generation and reasoning-capable models, directly addressing the reviewer’s question. Importantly, the reviewer acknowledges that our experiments are systematic and convincingly establish the phenomenon, yet the latest response does not explain why the distinction we identify, its causal investigation, or the proposed mitigation lack sufficient novelty or significance. It would be very helpful if the Reviewer provides a more specific explanation of which aspects of our contribution are considered already established, ideally with references to prior research work. This would help us understand the remaining concern and try to address it.

Reviewer_2zoBOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the problem that LLMs cannot access relevant knowledge consistently when facing different identifiers of the same molecule (such as common names, IUPAC names and SMILES). The author constructed ChemIDBench, which contains 896 molecules, and found that, across seven models, common names are more conducive to retrieving factual knowledge such as drug indications, while SMILES is more conducive to answering questions about molecular structure. The author further revealed the phenomenon that even if the model can recognize molecules, it may not be able to retrieve known facts through its structural identifier, and called it Cross Identifier Binding Failure. Finally, the author proposes Binding Self Distribution (BSD), which uses the model's own knowledge access capability based on molecular names to improve the model's ability to retrieve facts directly from SMILES through knowledge distillation. Experiments show that this method can significantly improve the fact retrieval performance of multiple models.

Critical strengths
  1. From a very enlightening perspective, this paper studies whether a large language model can access its existing knowledge consistently through different identifiers pointing to the same entity. This problem not only has practical significance in the field of chemistry, but also may widely exist in other knowledge-intensive fields such as biomedicine, so it has strong research value and enlightenment significance.
  2. By constructing ChemIDBench, the author verified this phenomenon on a number of universal and chemical domain-specific big language models, and conducted in-depth research combining molecular recognition testing, conflict identifier experiments, representation analysis, causal intervention, and other methods. The overall experiment covers a wide range, providing sufficient empirical support for the core findings of the paper.
  3. The author not only analyzes the inconsistency of cross identifier knowledge access from the model output level, but also studies the impact of the internal state of the model on the fact retrieval by using Activation Patching and other methods, providing additional causal evidence for understanding this phenomenon.
Critical weaknesses
  1. Previous studies have explored the impact of different molecular representations on the performance and prediction consistency of large language models [1, 2], and found a similar phenomenon of representation dependency. Although the Cross Identifier Binding Failure proposed in this paper provides a new research perspective, the current experiments are not enough to fully prove that it is an independent internal mechanism, not just the specific performance of existing representation inconsistency problems in knowledge retrieval tasks.
  2. The BSD method proposed in this paper needs to train a LoRA adapter for each molecule separately, which lacks the verification of the generalization ability of unknown molecules. In addition, molecule by molecule training and maintenance of adapter may bring high computing and storage costs, which limits the scalability and practical application value of this method.
  3. By injecting the hidden state under the name condition into the model calculation under the SMILES condition, the author successfully restored part of the fact retrieval ability. However, this may only be because the injected hidden state contains information related to the correct answer, which cannot directly prove that the identifier binding mechanism has been repaired. It is suggested to add more fine-grained causal analysis and controlled experiments to distinguish the role of entity identity information and answer-related information.

[1]. George Arthur Baker, Mario Sanz-Guerrero, and Katharina von der Wense. 2025. Molecular String Representation Preferences in Pretrained LLMs: A Comparative Study in Zero- & Few-Shot Molecular Property Prediction. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 1071–1085, Suzhou, China. Association for Computational Linguistics.

[2]. Bing Yan, Angelica Chen, and Kyunghyun Cho. Inconsistency of llms in molecular representations. Digital Discovery, 4(10):2876–2892, 2025.

Questions for authors
  1. Can the author further explain the essential difference between Cross Identifier Binding Failure and existing representation inconsistency?
  2. In the Activation Patching experiment, how can the author exclude that the recovery of knowledge retrieval ability only comes from the information about answers contained in the injected hidden state?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 2zoB (1/2)

We thank the reviewer for their constructive comments and valuable feedback.

W1/Q1, Novelty beyond molecular representation dependence.

We agree that molecular representation dependence is established, and we do not claim otherwise. Both Baker et al. (2025) and Yan et al. (2025) are already discussed and cited in our paper. Baker et al. (2025) investigate the effects of different structure-based molecular representations (e.g., SMILES and IUPAC names) on zero- and few-shot property prediction, while Yan et al. (2025) examine prediction consistency between SMILES and IUPAC. In contrast, our work focuses on factual knowledge access across molecular identifiers, particularly between structural representations and commonly used drug names. Our contribution is not the observation that equivalent representations produce different outputs, but the identification and systematic investigation of a more fundamental problem: why factual knowledge already available in an LLM can remain inaccessible through an equivalent identifier, even when the molecule is successfully recognized. The distinction is summarized below:

Research questionBaker et al.Yan et al.Ours
Representation-dependent prediction performance or consistency✓✓✓
Representation preferences across molecular property types✓—✓
Factual-access failure despite successful entity recognition——✓
Existing factual knowledge inaccessible across identifiers——✓
Causal investigation of cross-identifier factual access——✓
Recovering inaccessible facts through lexical routing——✓
Transferring existing factual access through self-distillation——✓

Our experiments establish why this distinction matters: First, recognition does not guarantee factual access. On ChemDFM-v2.0-14B, among molecules successfully recognized from SMILES, name-conditioned indication accuracy reaches 40.3%, whereas SMILES-conditioned accuracy remains at 3.2%, comparable to the 3.6% observed for unrecognized molecules. Second, the failure cannot be explained simply by molecular obscurity. Across all seven models, 95.6–98.5% of high-popularity molecules answered correctly through their names still fail through SMILES. Third, the problem extends to knowledge editing. Standard editors can successfully modify facts through molecular names without reliably transferring those edits to the corresponding SMILES. Fourth, our activation-patching experiments provide causal evidence: a molecule-specific representation obtained without an indication question substantially restores factual access through SMILES. Importantly, we do not stop at identifying the failure; we also investigate how to overcome it. Lookup-based lexical routing restores 83.6–98.8% of the aggregate name–SMILES factual-access gap across six models. We further propose Binding Self-Distillation (BSD), which transfers the model's own name-conditioned predictive behavior to its SMILES-conditioned route without external factual labels. This distinction is clarified in the revised Introduction and Appendix A.

W2, BSD scalability and generalization.

Thank you for raising this point. Our original per-molecule BSD experiments were designed to establish whether existing factual knowledge could be made accessible through structural identifiers by adapting the model's parameters, rather than introducing new factual knowledge/train model from scratch. To directly address the reviewer's concerns about scalability and generalization, we conducted an additional experiment using a single shared adapter on Qwen2.5-7B-Instruct. Full experimental setup and training details are provided in Appendix H.8 of the revised manuscript.

RelationBefore BSDShared BSD
Indication (75)1.33%16.00%
Target (61)1.64%6.56%
MoA (65)0.00%1.54%
ATC (69)2.90%2.90%
Overall (270)1.48%7.04%

On unseen molecules, the shared adapter achieves a +5.56 percentage-point improvement, statistically significant at the molecule level (95% paired-bootstrap CI: [+2.77, +8.63] pp; paired randomization p = 0.000488). The largest improvement occurs in indication (1.33% → 16.00%), with an additional improvement in target retrieval. We have incorporated this experiment into Section 5 and Appendix H.8 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 2zoB (2/2)

W3/Q2, Distinguishing molecular identity from answer-related information.

To directly address the reviewer's concern, we conduct an additional activation-patching experiment. Unlike our original experiment, which used a name-conditioned indication question as the source, the new source contains only the molecular name, without any indication question or answer. We patch this representation into the SMILES entity position and compare it with a different-molecule control. Experimental details and results are provided in Section 4.1 and Appendix D.6 of the revised manuscript.

ModelPeak layerIndication-free recoveryDifferent-molecule control
Qwen2.5-7B1173.0%−60.3%
ChemDFM-v1.5-8B895.6%−51.6%

Crucially, substantial factual recovery persists even without an indication question or answer in the source. For comparison, our original indication-conditioned intervention at the final prompt token recovered 104.9% on Qwen (L21) and 76.4% on ChemDFM (L16). The new indication-free intervention operates at the entity position and peaks earlier (L11 and L8, respectively), showing that molecule-associated information can restore factual access before indication-specific processing at the final token. Furthermore, the strongly negative different-molecule controls demonstrate that recovery depends on the matched molecular identity rather than generic activation replacement. Thus, indication-conditioned answer information is not required in the source for substantial factual recovery.

We hope our responses adequately address the reviewer's concerns.

Dear Reviewer 2zoB,

Just a gentle reminder, as the discussion deadline is approaching. We hope our response and the additional clarifications/experiments addressed your concerns, and we would be grateful if you could reconsider your assessment in light of our revised response. We would also greatly appreciate any further feedback you may have before the deadline. Thank you again for your time and consideration.

Comments on the paper

1 comment
Area_Chair_LzzFOpenReview ↗

Reminder to Reviewers: Please Engage in the Discussion

Dear Reviewers,

As the authors have now submitted their rebuttal, I would kindly ask you to read their responses and engage in the discussion.

Thank you!

Best regards,
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.