- The MolCA rows of Tables 2 and 19, called the phase-matched control (line 380), match the description, IUPAC, and retrieval numbers published in the MolCA paper to the decimal, including retrieval numbers from MolCA's own reranking pipeline. Were these rows re-run under this paper's protocol, and what are the re-run numbers?
- Line 225 describes a linear probe on frozen node tokens, but the released implementation fits no probe and scores the checkpoint's own functional-group head, which the control never trains, and even the trained head scores chloro at 0.491 (line 1339) although the GIN embeds atomic number (line 194). What does a fitted probe give on the control's node tokens?
- In Table 10 (line 1008), every row lists seed values equal to the mean, the mean minus the standard deviation, and the mean plus it. Are these the raw per-seed results, and can the per-pair scores behind them be provided?
- The released implementation trains retrieval against a separately loaded SciBERT encoder rather than the Q-Former text embedding of line 290, and it contains a best-of-8 caption reranker with an RDKit veto, which the paper names only as "candidate selection" (line 1461) while reporting greedy decoding (line 1264). Which settings produced Tables 2 and 3?
- The Reproducibility Statement (lines 488 to 491) says the supplement contains the pair miner and evaluators and that Appendix I lists the pair manifests, but the pair miner and the manifests are not in the supplement, and no evaluator exists for retrieval, ROC-AUC, or generation. Can they be provided?
- Line 196 says the joint graph and SMILES prompt "is reported alongside", but it is given only for the proposed model (line 368); the other models are scored zero-shot with unseen templates, and ATOMAS, which reads the SMILES string, falls below chance (0.422, Table 1). What do the control and the public checkpoints reach under the joint prompt?
- Mol-LLM [1] measures the same ordering as Eq. 2 with its graph discrimination ratio and trains it with MolPO, a length-normalized likelihood margin between correct and perturbed graphs, but it is neither cited nor compared. How do the Mol-LLM checkpoint on Table 1, and MolPO in place of L_pair on the same backbone and pairs, compare with the proposed method?
- Table 1 omits recent graph-language models such as Mol-LLaMA [2] and EDT-Former [3], and Table 2 omits BioT5 [4], cited in Section 2, which reports 41.3 exact match on ChEBI-20 generation against 38.0 here, and BioT5+ [5]. How does the method compare with them?
- Most external pairs flip chloro, and without them the gain falls to 19.2 points (line 400), yet the abstract headlines the pooled 0.928. Why is the pooled number reported as the external result rather than the chloro-free or per-axis result?
- In Table 3, removing end-to-end transfer returns BLEU-2 to the control level (60.8), while without L_pair it stays at 63.1, so the caption gains come mainly from the transfer phase (line 452). What does the caption-only control reach after a transfer phase of the same length?
[1] Chanhui Lee, Hanbum Ko, Yuheon Song, YongJun Jeong, Rodrigo Hormazabal, Sehui Han, Kyunghoon Bae, Sungbin Lim, and Sungwoong Kim. Mol-LLM: Multimodal generalist molecular LLM with improved graph utilization. In NeurIPS, 2025.
[2] Dongki Kim, Wonbin Lee, and Sung Ju Hwang. Mol-LLaMA: Towards general understanding of molecules in large molecular language model. In NeurIPS, 2025.
[3] Zihao Jing, Qiuhao Zeng, Ruiyi Fang, Yan Sun, Boyu Wang, and Pingzhao Hu. Entropy-guided dynamic tokens for graph-LLM alignment in molecular understanding. In ICLR, 2026.
[4] Qizhi Pei, Wei Zhang, Jinhua Zhu, Kehan Wu, Kaiyuan Gao, Lijun Wu, Yingce Xia, and Rui Yan. BioT5: Enriching cross-modal integration in biology with chemical knowledge and natural language associations. In EMNLP, 2023.
[5] Qizhi Pei, Lijun Wu, Kaiyuan Gao, Xiaozhuan Liang, Yin Fang, Jinhua Zhu, Shufang Xie, Tao Qin, and Rui Yan. BioT5+: Towards generalized biological understanding with IUPAC integration and multi-task tuning. In Findings of ACL, 2024.
Response to Reviewer kQBq
We thank the reviewer for the positive assessment. We are grateful that they found the work “a significant contribution to understanding and improving graph language models by introducing counterfactuals in evaluation and learning.”
W1. Thank you for flagging the density of the introduction. We will revise that section for readers less familiar with the terminology: a short problem statement, a compact description of the pair test, and then the learning intervention, with technical terms defined at first use. The goal is to make the contribution readable without weakening the technical content.
Q1. Yes. During the review period we provide the counterfactual resources in the anonymous repository: the 5,128-molecule training pair bank; the ChEBI-20 splits and a deterministic exporter for the 1,146-pair internal holdout manifest; the frozen 822-pair external manifest covering 535 scaffold clusters (JSON and CSV); and the pair-mining code, SMARTS definitions, split rules, audit files, and evaluation commands needed to reconstruct and score the data. Each evaluation pair records canonical SMILES, InChIKeys, scaffold cluster, target attribute, source, and fixed claim. Third-party source datasets remain under their original licenses; where redistribution is not permitted, the repository gives source links and exact preprocessing commands. Upon acceptance we will make these resources publicly available and add a Data Availability statement.
W2 / Q2. Thank you; this comparison should have been more explicit in related work. Janssen et al. (2026) and our framework both use real matched molecular pairs, but they answer different questions.
WISP partitions each pair into variable and constant regions and checks whether the change in summed atom attributions over the variable region agrees with the change in scalar property predictions, while attributions on the constant region stay small. We instead hold an attribute-specific textual claim fixed and test whether it receives a higher likelihood for the real molecule that licenses the claim than for a scaffold-matched molecule that does not. Scoring is graph-only, to prevent SMILES shortcuts. WISP does not train a graph-language model with an MMP objective. The two frameworks are complementary: WISP tests whether atom-level explanations track property-prediction changes; we test and improve whether language-model statements follow molecular structure. We will cite Janssen et al., expand this contrast in related work, and add a row to the method-comparison table.
Q3. Yes. We will increase the font size in Figure 2, including module labels, loss terms, molecular descriptors, and the claim-scoring annotations, and we will enlarge the figure so that all text remains legible at the final two-column scale.
We thank the reviewer again for these constructive comments. We will use them to make the introduction, related work, and Figure 2 clearer in the revision.