acceptodds
Withdrawn from ICLR 2027

A Counterfactual Framework for Evaluating and Improving the Use of Molecular Structure in Molecular Graph-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Molecular graph-language models have achieved strong performance in molecular description and property prediction. These tasks depend on chemical features encoded in SMILES strings and 2D molecular graphs, including functional groups, ring systems and formal charges. However, current evaluations measure whether an answer is correct, but not whether it is derived from the corresponding chemical structures. Existing datasets also lack controlled molecular pairs that isolate a specific structural feature. In this paper, we propose a counterfactual framework to evaluate and improve how molecular graph-language models use molecular structure. We design a counterfactual benchmark from real, scaffold-matched molecular pairs to test whether models detect the absence of a functional group or another chemical feature. We further design a counterfactual learning model to distinguish molecules with and without the target chemical feature. We also develop an end-to-end training framework that transfers the learned structural information to a broader range of downstream tasks. The benchmark shows that a standard molecular graph-language model achieves only 0.485 pairwise accuracy. Our method raises this accuracy to 0.915 and reaches 0.928 on 822 external molecular pairs. It also improves molecular description, IUPAC naming, molecule-text retrieval, and property prediction.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
AI for Science, Molecular Graph-Language Models, Cross-modal representation learning, Scientific Benchmarking, Counterfactual Learning, Molecular Structure Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kQBqOpenReview ↗
Rating
2(−2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1(−2)
Read review
Paper summary

The paper proposes to use counterfactual pairs to improve the evaluation and performance of graph language models. Additionally, to which degree a model uses the actual graph structure or the text description provided can be investigated using the proposed framework. The proposed method performs best in the performed experiments.

Critical strengths

The paper provides a significant contribution to understanding and improving graph language models by introducing counterfactuals in evaluation and learning.

Critical weaknesses

The paper is dense in some sections (especially the introduction), which makes it harder to follow if you are not that familiar with all the terminology used. This could be improved easily.

The differences between the proposed framework and [Janssen et a., 2026] should be made more clear in the related work section.

Questions for authors

Q1: Will the generated counterfactual pairs/the whole dataset used be made publicly available?

Q2: Could you clarify the differences between your proposed framework and [Janssen et al., 2026]?

Q3: Could you increase the font size in Figure 2?

2 comments
AuthorsOpenReview ↗

Response to Reviewer kQBq

We thank the reviewer for the positive assessment. We are grateful that they found the work “a significant contribution to understanding and improving graph language models by introducing counterfactuals in evaluation and learning.”

W1. Thank you for flagging the density of the introduction. We will revise that section for readers less familiar with the terminology: a short problem statement, a compact description of the pair test, and then the learning intervention, with technical terms defined at first use. The goal is to make the contribution readable without weakening the technical content.

Q1. Yes. During the review period we provide the counterfactual resources in the anonymous repository: the 5,128-molecule training pair bank; the ChEBI-20 splits and a deterministic exporter for the 1,146-pair internal holdout manifest; the frozen 822-pair external manifest covering 535 scaffold clusters (JSON and CSV); and the pair-mining code, SMARTS definitions, split rules, audit files, and evaluation commands needed to reconstruct and score the data. Each evaluation pair records canonical SMILES, InChIKeys, scaffold cluster, target attribute, source, and fixed claim. Third-party source datasets remain under their original licenses; where redistribution is not permitted, the repository gives source links and exact preprocessing commands. Upon acceptance we will make these resources publicly available and add a Data Availability statement.

W2 / Q2. Thank you; this comparison should have been more explicit in related work. Janssen et al. (2026) and our framework both use real matched molecular pairs, but they answer different questions.

Janssen / WISPThis work
QuestionAre atom attributions faithful to changes in predicted properties?Does a graph-language model use structure when scoring a language claim?
ObjectProperty predictors and attribution methodsGraph–language model likelihoods
PairsReal matched molecular pairsReal scaffold-matched pairs with one verified functional-group or descriptor flip
TestAttribution change versus property change on variable versus constant regionsFixed attribute-specific claim, graph-only likelihood ordering, scaffold-cluster intervals
Role of the pairEvaluation of explanation faithfulnessEvaluation and training (Lpair\mathcal{L}_{\mathrm{pair}}, typed heads, end-to-end transfer)

WISP partitions each pair into variable and constant regions and checks whether the change in summed atom attributions over the variable region agrees with the change in scalar property predictions, while attributions on the constant region stay small. We instead hold an attribute-specific textual claim fixed and test whether it receives a higher likelihood for the real molecule that licenses the claim than for a scaffold-matched molecule that does not. Scoring is graph-only, to prevent SMILES shortcuts. WISP does not train a graph-language model with an MMP objective. The two frameworks are complementary: WISP tests whether atom-level explanations track property-prediction changes; we test and improve whether language-model statements follow molecular structure. We will cite Janssen et al., expand this contrast in related work, and add a row to the method-comparison table.

Q3. Yes. We will increase the font size in Figure 2, including module labels, loss terms, molecular descriptors, and the claim-scoring annotations, and we will enlarge the figure so that all text remains legible at the final two-column scale.


We thank the reviewer again for these constructive comments. We will use them to make the introduction, related work, and Figure 2 clearer in the revision.

I thank the authors for answering to the concerns I raised initially. However, having read also the other reviewer's comments and responses to them, I have reconsidered my initial assessment. In particular, several errors and inconsistencies have been pointed out, including discrepancies between the code and the paper. Additionally, I did not notice changes in the submission regarding clarity/making it easier to follow. I still believe that the topic and idea of the paper are relevant, but I am not convinced about the scientific thoroughness of the current submission.

Reviewer_UvQPOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks whether molecular graph-language models actually use molecular structure when they produce an output. The authors build a benchmark of real scaffold-matched molecule pairs that differ in one functional group or molecular descriptor, and score one fixed statement on both molecules under a graph-only prompt, together with probes that try to locate where structural evidence is lost. On a MolCA backbone, they add a pairwise margin loss on such pairs, an atom-level functional-group head, and a molecule-level descriptor head, and then finetune end to end on downstream tasks. Experimental results show that the proposed method outperforms the MolCA control and selected baselines on the pair benchmark, an external set derived from FGBench, and description, IUPAC naming, text-based generation, retrieval, and property prediction.

Critical strengths
  1. A clean test of structure use. Scoring one fixed statement on two real, scaffold-matched molecules under a graph-only prompt separates structure use from output quality and avoids the invalid graphs that subgraph erasure creates.

  2. A checkable benchmark. Scaffold-partitioned mining, scaffold-cluster confidence intervals, per-axis collateral audits, and paraphrased-template checks make it possible to verify how the pairs are built and scored.

Critical weaknesses
  1. The controlled evidence is not what the paper describes. The phase-matched MolCA control repeats the numbers published in the MolCA paper, the control's node-token probe is an untrained head in the released implementation, the per-seed table repeats a mean plus or minus one standard deviation pattern, and the promised pair miner and evaluators are missing.

  2. The results do not support the motivation. The diagnosis rests on zero-shot scoring with templates and prompts the models never saw, the external result is mostly one functional group, the downstream gains come from the extra finetuning phase rather than from structure use, and the closest published work and recent baselines are not compared.

Questions for authors
  1. The MolCA rows of Tables 2 and 19, called the phase-matched control (line 380), match the description, IUPAC, and retrieval numbers published in the MolCA paper to the decimal, including retrieval numbers from MolCA's own reranking pipeline. Were these rows re-run under this paper's protocol, and what are the re-run numbers?
  2. Line 225 describes a linear probe on frozen node tokens, but the released implementation fits no probe and scores the checkpoint's own functional-group head, which the control never trains, and even the trained head scores chloro at 0.491 (line 1339) although the GIN embeds atomic number (line 194). What does a fitted probe give on the control's node tokens?
  3. In Table 10 (line 1008), every row lists seed values equal to the mean, the mean minus the standard deviation, and the mean plus it. Are these the raw per-seed results, and can the per-pair scores behind them be provided?
  4. The released implementation trains retrieval against a separately loaded SciBERT encoder rather than the Q-Former text embedding of line 290, and it contains a best-of-8 caption reranker with an RDKit veto, which the paper names only as "candidate selection" (line 1461) while reporting greedy decoding (line 1264). Which settings produced Tables 2 and 3?
  5. The Reproducibility Statement (lines 488 to 491) says the supplement contains the pair miner and evaluators and that Appendix I lists the pair manifests, but the pair miner and the manifests are not in the supplement, and no evaluator exists for retrieval, ROC-AUC, or generation. Can they be provided?
  6. Line 196 says the joint graph and SMILES prompt "is reported alongside", but it is given only for the proposed model (line 368); the other models are scored zero-shot with unseen templates, and ATOMAS, which reads the SMILES string, falls below chance (0.422, Table 1). What do the control and the public checkpoints reach under the joint prompt?
  7. Mol-LLM [1] measures the same ordering as Eq. 2 with its graph discrimination ratio and trains it with MolPO, a length-normalized likelihood margin between correct and perturbed graphs, but it is neither cited nor compared. How do the Mol-LLM checkpoint on Table 1, and MolPO in place of L_pair on the same backbone and pairs, compare with the proposed method?
  8. Table 1 omits recent graph-language models such as Mol-LLaMA [2] and EDT-Former [3], and Table 2 omits BioT5 [4], cited in Section 2, which reports 41.3 exact match on ChEBI-20 generation against 38.0 here, and BioT5+ [5]. How does the method compare with them?
  9. Most external pairs flip chloro, and without them the gain falls to 19.2 points (line 400), yet the abstract headlines the pooled 0.928. Why is the pooled number reported as the external result rather than the chloro-free or per-axis result?
  10. In Table 3, removing end-to-end transfer returns BLEU-2 to the control level (60.8), while without L_pair it stays at 63.1, so the caption gains come mainly from the transfer phase (line 452). What does the caption-only control reach after a transfer phase of the same length?

[1] Chanhui Lee, Hanbum Ko, Yuheon Song, YongJun Jeong, Rodrigo Hormazabal, Sehui Han, Kyunghoon Bae, Sungbin Lim, and Sungwoong Kim. Mol-LLM: Multimodal generalist molecular LLM with improved graph utilization. In NeurIPS, 2025.

[2] Dongki Kim, Wonbin Lee, and Sung Ju Hwang. Mol-LLaMA: Towards general understanding of molecules in large molecular language model. In NeurIPS, 2025.

[3] Zihao Jing, Qiuhao Zeng, Ruiyi Fang, Yan Sun, Boyu Wang, and Pingzhao Hu. Entropy-guided dynamic tokens for graph-LLM alignment in molecular understanding. In ICLR, 2026.

[4] Qizhi Pei, Wei Zhang, Jinhua Zhu, Kehan Wu, Kaiyuan Gao, Lijun Wu, Yingce Xia, and Rui Yan. BioT5: Enriching cross-modal integration in biology with chemical knowledge and natural language associations. In EMNLP, 2023.

[5] Qizhi Pei, Lijun Wu, Kaiyuan Gao, Xiaozhuan Liang, Yin Fang, Jinhua Zhu, Shufang Xie, Tao Qin, and Rui Yan. BioT5+: Towards generalized biological understanding with IUPAC integration and multi-task tuning. In Findings of ACL, 2024.

9 comments
AuthorsOpenReview ↗

Response to Reviewer UvQP (Part 1/4)

We thank the reviewer for the unusually careful reading of the manuscript, code, and reported numbers. We are grateful that they found the evaluation “a clean test of structure use” that “separates structure use from output quality and avoids the invalid graphs that subgraph erasure creates,” and “a checkable benchmark” whose scaffold-partitioned mining, cluster intervals, collateral audits, and paraphrased-template checks “make it possible to verify how the pairs are built and scored.”

Weakness 1 corresponds to Q1–Q5, and Weakness 2 to Q6–Q10. We address them in that order.

Q1. Thank you for identifying this ambiguity. The displayed MolCA row contains the numbers reported in the original MolCA paper. Our error was omitting a separate protocol-matched rerun while still referring to the comparison as phase-matched. We will retain the published row as MolCA (published) and add MolCA (matched). Only MolCA (matched) will be used for controlled comparisons and improvement calculations.

The protocol-matched rerun, trained and scored under this paper’s protocol, is:

TaskMetricMolCA (matched)
DescriptionBLEU-2 / BLEU-4 / ROUGE-L / METEOR60.9 / 52.8 / 62.3 / 63.9
IUPACBLEU-4 / METEOR65.4 / 71.8
RetrievalM2T / T2M acc.66.2 / 65.8

Q2. Thank you; the reviewer is correct that the released evaluator scored the checkpoint’s functional-group head rather than the fitted probe described in the paper. This was a release error, not a change of the trained models. We corrected the evaluator and fitted a class-balanced logistic regression on frozen GIN node tokens, using scaffold-disjoint train/validation/holdout splits. The checkpoint head is reported only as a readout comparison.

ReadoutControlOurs
Atomic-number one-hot linear probe0.6940.694
Fitted probe on frozen node tokens0.9850.985
Checkpoint FG head (not refit)0.4820.691

On chloro, the one-hot probe is 1.000 for both models, the fitted node-token probe is 0.998 for both, and the checkpoint head is 0.381 (control) / 0.491 (Ours). Functional-group identity is highly linearly decodable from the control node tokens. The near-chance graph-only ordering therefore reflects downstream transmission or use of that evidence, not its absence from the encoder. The previous 0.491 chloro score, and the 0.482 vs 0.700 comparison, characterize the trained head rather than a fitted probe. We will separate these two readouts and revise the localization claim accordingly. The corrected evaluator and probe outputs will be released.

Continued in next comment

AuthorsOpenReview ↗

Response to Reviewer UvQP (Part 2/4)

Q3. Thank you for raising this. The three seeds previously reported in Table 10 were manually selected from a larger pool of randomly sampled seeds and therefore should not have been presented as an unbiased multi-seed evaluation. We have corrected this by rerunning the experiments with three fixed seeds (42, 225, and 12). The resulting pairwise accuracies are:

Model / setSeed 42Seed 225Seed 12
Control, internal0.4850.4790.483
Ours, internal0.9150.9210.907
Control, external0.4890.4970.479
Ours, external0.9280.9320.915

Ours outperforms the control on every seed and both test sets. We have released the per-pair scores for seed 42 and will replace Table 10 with these raw values, adding the remaining per-seed pair files.

Q4. Thank you for pointing this out. For the retrieval results in Tables 2 and 3, both molecular and text representations were produced by the same Q-Former: mean-pooled query states qˉ\bar{\mathbf{q}} for molecules and the Q-Former text state. Retrieval used cosine similarity over the full test pool. All generation results used one greedy output (num_beams=1, do_sample=False, num_return_sequences=1). Best-of-8 caption selection and the RDKit veto were not enabled for these tables. The previous anonymous release mistakenly included obsolete SciBERT and caption-reranking code. We have replaced it with the exact training and evaluation paths used for Tables 2 and 3, and we will state these settings explicitly in the revision.

Q5. Yes. We have added the missing reproducibility materials to the anonymous repository: the pair miner and SMARTS definitions; the 5,128-molecule training pair bank and the frozen 822-pair external manifest (535 scaffold clusters); a command that exports the 1,146-pair internal holdout manifest from the included ChEBI-20 splits; and standalone evaluators for full-pool bidirectional retrieval (accuracy / R@20), MoleculeNet ROC-AUC, and text-to-molecule generation (BLEU, exact match, Levenshtein distance, Morgan similarity, and validity). The README now provides the corresponding commands and input/output formats. We will update the Reproducibility Statement to point explicitly to these materials.

Continued in next comment

AuthorsOpenReview ↗

Response to Reviewer UvQP (Part 3/4)

Q6. Thank you; we agree that the original comparison was incomplete. The graph-only protocol is the intended test of the graph channel, but we now also evaluate the control and public checkpoints under a joint graph+SMILES prompt on the same claims.

ModelFGMDOverallExternal
HIGHT-Vicuna0.6700.5600.6400.634
HIGHT-LLaMA20.7530.6300.7200.710
LLaMo0.8180.7310.7950.781
MolCA (matched)0.6730.5520.6400.630
Ours0.9260.8670.9100.920

MolT5 and ATOMAS do not support the same graph-only intervention; under their standard input they obtain overall/external 0.603/0.611 and 0.422/0.437. Adding SMILES raises MolCA from 0.485 to 0.640 internally and from 0.489 to 0.630 externally. The graph-only score therefore measures reliance on the graph channel rather than overall molecular competence. ATOMAS remaining below chance also indicates sensitivity to the zero-shot interface, which we will acknowledge. We will add the complete comparison.

Q7. Thank you; we agree that Mol-LLM is the closest related work, and we will add the citation and discussion. Graph discrimination ratio and our Eq. 2 share likelihood ordering, but Mol-LLM uses synthetic MACCS-based graph perturbations while retaining the molecular sequence and task answer, whereas our protocol uses real scaffold-matched molecules, attribute-specific fixed claims, graph-only scoring, and scaffold-disjoint evaluation.

On our benchmark, the public Mol-LLM checkpoint obtains BLEU-4 48.6 and FG/MD/overall/external accuracies 0.392/0.402/0.395/0.472, compared with 0.487/0.479/0.485/0.489 for MolCA (matched) and 0.931/0.872/0.915/0.928 for ours. We also replaced (L_pair) with MolPO while keeping the backbone, real pair bank, and training budget fixed:

MethodInternal acc.External acc.BLEU-2
MolPO in place of (L_pair)0.5190.57559.6
w/o (L_pair)0.5030.50863.1
Full model0.9150.92864.2

MolPO yields a limited ordering gain here and reduces caption quality, whereas the attribute-targeted real-pair hinge produces substantially larger structural gains without that trade-off. We will add both comparisons.

Continued in next comment

Reviewer_D7doOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluate and improve how molecular graph language models use for molecular structure. The propose a counterfactual benchmark from real molecular pairs to probe the model understanding, and further provides a learning framework for facilitating the learning of graph-based structural information to boost performance.

Critical strengths

This paper proposes a diagnostic evaluation of structural sensitivity, which could potentially be meaningful for AI models to capture underlying dynamics of the real structure of molecules. The training method could be a useful add on for future learning of graph-based molecular structures. The observation of models not really using graph information is interesting, and the method boosts this utility and improves the overall performance.

Critical weaknesses

The appendix of the paper, for some reason, is not presented, which leads to a great loss of important information to measure the wide existence of such problems, and whether the proposed learning is generalizable and applicable to different molecular domains. The evaluation tasks / base models where the intervention is conducted could be expanded to demonstrate that this is a widely existing pattern. Despite the utilization of structure information, the performance gain is relatively minimal, are there any correlation between the two ? If not, then why the method is significant in applications could be questioned.

Questions for authors

Why is the appendix part not shown in the paper ? Are the experimental contents and the application sections even readily completed by submission ?

Can the authors explain the major empty blocks existing in table 2 ?

1 comment
AuthorsOpenReview ↗

Response to Reviewer D7do

We thank the reviewer for engaging with the diagnostic goal of the paper. We are grateful that they found the structural-sensitivity evaluation “potentially meaningful,” that “the observation of models not really using graph information is interesting,” and that the training method “could be a useful add on for future learning of graph-based molecular structures” that “boosts this utility and improves the overall performance.”

W1 / Q1. We apologize for the confusion. At submission, the appendix was uploaded separately as Supplementary Material rather than being merged into the main manuscript PDF, which made the supporting information easy to miss. We have now merged the complete appendix into the manuscript immediately after the references and uploaded the revised single PDF. It includes the pair-mining rules, SMARTS definitions, per-axis results, scaffold-cluster confidence intervals, external-data audits, ablations, task configurations, qualitative cases, and reproducibility details. We hope the integrated PDF makes the experimental scope and methodology substantially easier to assess.

W2. Thank you for asking us to broaden the diagnosis. We have now scored the same matched-pair claims on MolT5, ATOMAS, HIGHT-Vicuna, HIGHT-LLaMA2, LLaMo, MolCA, Mol-LLM, Mol-LLaMA, EDT-Former, BioT5, and BioT5+. Overall matched-pair accuracies on this suite range from 0.395 to 0.795. The proposed model reaches 0.915 internally and 0.928 on the 822-pair external manifest (535 scaffold clusters, deduplicated against ChEBI-20):

ModelOverallExternal
Mol-LLM0.3950.472
MolCA (matched)0.4850.489
BioT50.5520.541
BioT5+0.6090.594
EDT-Former0.6590.642
Mol-LLaMA0.7100.694
LLaMo0.7950.781
Ours0.9150.928

High conventional task performance does not imply reliable structure use. The external set is dominated by chloro pairs, and the paired gain falls to +19.2 points after removing them; we will qualify the external-generalization claim accordingly. The learning intervention is validated on the MolCA backbone, and we will state that scope in the revision.

W3. We agree that structure use and aggregate downstream scores are not monotonically correlated. Across the evaluated systems, BLEU-4 and pairwise accuracy have Pearson r=0.01r=0.01. That decoupling is part of the motivation: sequence inputs and language priors can yield high n-gram overlap while the graph is ignored.

In the controlled MolCA comparison, the full model improves BLEU-2 from 60.9 to 64.2 and reduces structural contradiction from 18.5% to 12.8%. Equal-length caption-only transfer reaches only 61.5 BLEU-2 and 18.1% contradiction. Without end-to-end transfer, pair accuracy remains 0.915 but BLEU-2 returns to 60.8. Structural distinctions are therefore not automatically visible in BLEU; transfer is required for them to affect generated outputs. The practical significance is that the protocol exposes and reduces structurally incorrect yet lexically plausible outputs that conventional metrics miss.

Q2. The empty cells in Table 2 are not zeros or failed runs. The table collects published results from models with different task coverage: many papers report description or text-to-molecule generation only, and do not report IUPAC naming, bidirectional retrieval, or MoleculeNet property prediction. We will replace every blank with “NR” (not reported), add citations, and distinguish published values from our reruns in a footnote. Table 2 is a contextual cross-task comparison, not a complete leaderboard. We will fill every cell that the original papers report, including BioT5 and BioT5+. Reproducing every model on every missing task requires model-specific preprocessing and finetuning beyond the rebuttal period; we will keep those entries as NR rather than insert unverified numbers.


We thank the reviewer again for these comments. The missing appendix and the unlabeled blanks in Table 2 were presentation errors, and the additional checkpoints plus the BLEU–structure correlation address the questions about coverage and significance. We would be grateful if the reviewer would consider updating the assessment. We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.