- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper tests an assumption that chemical AI pipelines implicitly rely on but that has not been systematically verified: whether co-referring molecular identifiers (trivial name, IUPAC name, SMILES) provide interchangeable access to an LLM's knowledge. The authors build ChemIDBench — 896 carefully deduplicated approved drugs from ChEMBL, each with three matched identifiers and 7 query types (entity-associated: identification, therapeutic indication; structure-computable: formula, benzene ring, nitrogen count, ring count, heavy atoms) — and evaluate seven 7B–72B models, including chemistry-specialized ChemDFM variants. The core finding is a systematic double dissociation: trivial names give far better access to entity-associated facts (indication accuracy collapses from 38.8% under the name to 2.1% under SMILES), while SMILES is better for structure-computable properties (benzene detection 76.7%→94.6%). Crucially, factual retrieval fails even when the model demonstrably recognizes the molecule from SMILES (both 2AFC and free-form criteria) — decoupling entity recognition, knowledge possession, and identifier-to-knowledge binding, which the authors name cross-identifier binding failure. Converging evidence is provided at three levels: behavioral (lexical bridge restores 91.8–99.4%; conflicting-identifier experiments), representational (Name–SMILES top-1 retrieval only 0.7–2.8% vs. up to 15% for IUPAC–SMILES in chemistry-specialized models), and causal (activation patching restores 76.4–138.4% of the name–SMILES gap with negative controls). Standard knowledge editors (ROME/FT) fail to reliably propagate edits across identifiers. Finally, Binding Self-Distillation distills the model's own name-conditioned predictive behavior into a LoRA adapter on the SMILES route via full-vocabulary KL, raising retrieved facts per entity from ~0.05 to 0.68–0.87 across all seven models without needing the name at inference. Appendix I replicates the phenomenon in the protein domain with UniProt accessions.
Critical strengths
1,Whether equivalent identifiers equivalently access knowledge is a real, overlooked reliability problem for LLMs in scientific workflows. The paper cleanly decouples recognition, knowledge possession, and access into three testable capabilities; "cross-identifier binding failure" is a crisp, falsifiable construct with direct implications for LLM4Science, knowledge editing, and agentic tool routing (resolve-then-query). 2,Behavioral (recognition controls, lexical bridge, conflicting identifiers) → representational (cross-identifier top-1 retrieval, chemistry-specialization contrasts) → causal (activation patching with different-molecule controls) — the three layers interlock into a coherent story. The patching controls (different molecule + different gold answer, all negative) make the causal argument more rigorous than typical work in this space. 3,Paired design (same molecule, same property, only the identifier changes) with McNemar tests (p<10⁻⁷); strict entity cleaning in benchmark construction (salts, multi-fragment, duplicate names/structures removed); 2AFC filtered across three independent decoy conditions; editing results corrected for locality leakage with bootstrap CIs; downstream evaluation prompts excluded from BSD training stems. Methodological care is above average.
Critical weaknesses
1,BSD trains a separate LoRA adapter per molecule (120 steps, rank 8) with no parameter sharing across molecules, and the authors themselves defer generalization to unseen entities to future work. As presented, it is an existence proof rather than a usable method: 896 molecules require 896 adapters, and transfer is only possible for molecules whose name route already answers indication correctly (leaving n=47 for ChemDFM-v1.5-8B). Meanwhile the lookup routing of Section 4.3 is the actually practical, training-free remedy. Suggestions: (a) include at least one shared-adapter variant and report cross-entity generalization; (b) tone down the "mitigation" framing in the main text. 2,BSD is only evaluated on molecules for which the model already retrieves the indication correctly via the trivial name. While logically necessary (there is no source behavior to transfer otherwise), this means: (a) reported gains are conditioned on a subset where knowledge exists and is name-reachable, and do not extrapolate to all 896 molecules; (b) selection uses retrieval success on the same relation (indication) as the gating criterion, risking circularity. Cross-relation transfer is reported (target 0→27.6%, MoA 0→5.0%, ATC 3.3→11.4%) but the latter two are quite low in absolute terms. A selected-vs-unselected contrast and more prominent reporting of absolute cross-relation numbers would strengthen the claim. 3,Even the name route only reaches 22–44% indication accuracy, and the SMILES route is near zero — the whole phenomenon sits on a substrate of weakly memorized facts. Approved ChEMBL drugs almost certainly appear in pretraining corpora, and name-route accessibility is plausibly correlated with drug popularity in the corpus (aspirin vs. obscure compounds). The paper neither stratifies by name frequency/popularity nor discusses contamination. This does not invalidate the conclusion, but bounds its scope: how much of binding failure is a restatement of "the knowledge is barely in the model at all"? A frequency-stratified analysis would help. 4,Activation patching replaces the residual stream at a single site (final prompt token) layer-by-layer; the peak depth varies across models (relative depth 0.50–0.83) and is acknowledged as merely descriptive. Recovery >100% (up to 138.4%) is left unexplained (what does overshoot mean?). No attention-head/MLP-level localization, and no account of what the lexical state actually carries. Adequate as causal evidence; preliminary as mechanistic explanation.
Questions for authors
see Weaknesses
4 comments
Response to Reviewer FY4R (2/2)
| Model | Name accuracy (low → high) | SMILES accuracy (high) | SMILES failure among name-correct (high) |
|---|---|---|---|
| Qwen2.5-7B | 25.1% → 36.1% | 2.0% | 96.3% |
| Qwen2.5-14B | 30.1% → 40.5% | 2.0% | 95.9% |
| Qwen2-72B | 36.8% → 45.2% | 1.0% | 98.5% |
| ChemDFM-v1.5-8B | 1.0% → 10.4% | 0.3% | 96.8% |
| ChemDFM-v2.0-14B | 26.4% → 45.5% | 3.0% | 95.6% |
| Llama-3-8B | 28.4% → 43.5% | 1.0% | 97.7% |
| Llama-3.1-8B | 29.1% → 43.1% | 2.7% | 96.1% |
Across all seven models, name-conditioned indication accuracy increases from the low- to the high-popularity group. This confirms that differences in drug familiarity are relevant to factual retrieval and should be distinguished from cross-identifier binding failure. However, the central finding is that greater name-conditioned factual accessibility does not translate into comparable SMILES-conditioned accessibility. Even in the highest-popularity group, SMILES accuracy remains between 0.3% and 3.0% across models. More importantly, among highly popular molecules whose indications are correctly retrieved through their trivial names, 95.6–98.5% still fail when queried through SMILES. Consequently, the observed failure cannot be attributed solely to the model not knowing the correct fact or to the molecule being relatively unfamiliar. These findings reinforce our central distinction between factual knowledge availability and cross-identifier knowledge access.
W4, Activation-patching.
To address the reviewer's concerns, we conducted additional controlled activation-patching experiments on Qwen2.5-7B-Instruct and ChemDFM-v1.5-8B.
1. What does the patched representation carry? Our original experiment patched the final prompt-token representation from a name-conditioned indication question. To distinguish molecule-specific information from computation associated with the indication question or answer, we introduced an indication-free source containing only the molecular name, with no indication question or answer, and patched its representation into the SMILES entity position. This intervention recovered 73.0% of the normalized factual log-probability gap on Qwen (layer 11) and 95.6% on ChemDFM (layer 8). Crucially, patching representations from a different molecule instead produced −60.3% and −51.6% recovery, respectively (both p < 10⁻⁶⁰). These controlled results show that substantial factual recovery is molecule-specific and does not require indication-conditioned information in the source, supporting the role of a question-independent, name-associated representation in enabling factual access. Full experimental details and results are provided in Section 4.1 and Appendix D.6 of the revised manuscript.
2. Where does recovery occur, and which components contribute? We extended the original residual-stream interventions to separately patch attention and MLP outputs at the entity position and final prompt token.
| Model / position | Full residual state | Attention output | MLP output |
|---|---|---|---|
| Qwen / entity | 73.0% (L11) | 0.8% | −0.3% |
| Qwen / final token | 104.9% (L21) | 79.7% | 54.3% |
| ChemDFM / entity | 95.6% (L8) | 1.3% | 8.8% |
| ChemDFM / final token | 76.4% (L16) | 58.8% | 17.9% |
In both models, entity-position recovery peaks earlier than final-token recovery: relative depth 0.39 versus 0.75 for Qwen and 0.25 versus 0.50 for ChemDFM. At the entity position, patching the accumulated residual representation produces substantial recovery, whereas isolated component-output interventions have much smaller effects. At the final token, attention-output interventions alone recover 79.7% and 58.8% of the gap on Qwen and ChemDFM, respectively. Full experimental details and results are provided in Section 4.1 and Appendix D.7 of the revised manuscript. These interventions provide a more fine-grained picture: molecule-specific information is accessible in earlier entity-position representations, while later attention-mediated computation makes a substantial contribution to factual prediction.
3. Why can recovery exceed 100%? As already explained in Appendix D.4 of our original submission, normalized recovery measures the fraction of the original name–SMILES gold-answer log-probability gap closed by patching, rather than factual accuracy. Recovery exceeds 100% when the patched model assigns a higher log probability to the correct answer than the original name-conditioned model.
We hope our responses adequately address the reviewer's concerns.
Dear Reviewer FY4R,
Just a gentle reminder, as the discussion deadline is approaching. We hope our response and the additional clarifications/experiments addressed your concerns, and we would be grateful if you could reconsider your assessment in light of our revised response. We would also greatly appreciate any further feedback you may have before the deadline. Thank you again for your time and consideration.
Dear Reviewer FY4R,
As the discussion period is coming to an end and we did not receive your feedback on our rebuttal, we would like to briefly summarize the additional experiments and revisions we made to carefully address all four concerns raised in your review:
-
BSD scalability and generalization: We introduced a shared-adapter variant trained on 250 molecules and evaluated on 75 unseen molecules. Indication accuracy increased from 1.33% to 16.00%, while overall factual accuracy improved from 1.48% to 7.04%. Following your suggestion, we also revised the Introduction and Abstract to tone down the mitigation claims, as reflected in the updated manuscript.
-
Selection bias: We evaluated 100 previously excluded molecules and showed substantial cross-relation improvements even when the original indication-based selection criterion was not satisfied.
-
Molecular popularity: We conducted a frequency-stratified analysis across all 896 molecules and seven models. Even among highly popular molecules whose indications are correctly retrieved through their names, 95.6–98.5% still fail through SMILES.
-
Activation patching: We added indication-free, molecule-specific controls and attention/MLP-level interventions, providing further causal evidence and a more detailed understanding of the mechanism. We also clarified why normalized recovery can exceed 100%.
All these experiments and clarifications are incorporated into the revised manuscript.
We appreciate your recognition of the scientific motivation and experimental contributions of our work. We have carefully addressed each of your concerns and hope these additional results strengthen your assessment of the paper. We would be grateful if you could take these additional results into account in your final assessment and consider increasing your score.
Thank you again for your time.
Response to Reviewer FY4R (1/2)
We thank the reviewer for their constructive comments and valuable feedback.
W1, BSD generalization.
Thank you for raising this point. Our original per-molecule BSD experiments were designed to establish whether existing factual knowledge could be made accessible through structural identifiers by adapting the model's parameters, rather than introducing new factual knowledge/train model from scratch. To directly address the reviewer's concerns about scalability and generalization, we conducted an additional experiment using a single shared adapter on Qwen2.5-7B-Instruct. Full experimental setup and training details are provided in Appendix H.8 of the revised manuscript.
On unseen molecules, the shared adapter achieves a +5.56 percentage-point improvement, statistically significant at the molecule level (95% paired-bootstrap CI: [+2.77, +8.63] pp; paired randomization p = 0.000488). The largest improvement occurs in indication (1.33% → 16.00%), with an additional improvement in target retrieval. We have incorporated this experiment into Section 5 and Appendix H.8 of the revised manuscript. Following the reviewer's suggestion, we have toned down the mitigation framing in the revised submission.
W2, Selection bias.
Our original indication-based selection was intentional: BSD aims to transfer factual knowledge already accessible through a molecule's name to its structural representation, without external factual supervision. Selecting molecules with verified name-accessible knowledge therefore provides a controlled setting for testing whether this transfer is possible. To address the reviewer's concern about selection bias, we conducted an additional selected-versus-unselected experiment on Qwen2-72B-Instruct, extending the original 435 selected molecules (Table 8) with 100 previously excluded molecules. We divided the excluded molecules into two groups: 50 whose indications were incorrect based on their trivial names but who had at least one other correctly retrieved fact (target, MoA, or ATC), and 50 without verified name-accessible facts. All groups received the same molecule-specific BSD adaptation protocol. The experimental setup and full results are reported in Section 5 and Appendix H.9 of the revised manuscript.
SMILES-conditioned factual accuracy before → after BSD:
Crucially, substantial cross-relation improvements occur even among molecules excluded by the original indication criterion. For molecules with other name-accessible facts, target accuracy increases from 0% to 51.2%, MoA from 0% to 45.7%, and ATC from 0% to 30.6%. Thus, BSD's benefits are not restricted to molecules whose indications were correctly answered through their names. We also investigated whether BSD's improvements depend on whether the model can already retrieve the specific fact being evaluated through the trivial name. For each relation (target, MoA, and ATC), we divided the original 435 molecules into two groups: those for which Qwen2-72B correctly answered that relation using the trivial name, and those for which it did not. We then measured the improvement in SMILES-conditioned accuracy after BSD for each group. The improvements were consistently larger when the corresponding fact was already accessible through the name: target +58.9 vs. +11.8 percentage points, MoA +49.3 vs. +26.0, and ATC +66.2 vs. +2.3 (name-correct vs. name-incorrect). These results show that BSD is particularly effective at improving access to facts the model can already retrieve through the trivial name, regardless of whether indication correctness was used as the original selection criterion.
W3, Molecular popularity.
To address the reviewer's concern that cross-identifier failures may primarily reflect limited factual memorization or differences in drug popularity, we conducted an additional frequency-stratified analysis across all 896 molecules and all seven models. We used PubMed exact-name mention frequency as a measure of molecular popularity and divided the molecules into low-, medium-, and high-popularity groups (299/298/299 molecules). The experimental setup and full results are reported in Section 2.3 and Appendix B.5 of the revised manuscript. See the next comment below.