- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Ampere, a framework for molecular property prediction using direct and indirect experimental evidence from scientific literature. It combines a learned property-transfer model with submodular context optimization to select evidence for LLM-based analogical reasoning. Experiments on six ADME and safety tasks across two benchmarks show improvements over conventional molecular predictors and similarity-based retrieval baselines.
Critical strengths
- The paper addresses the practical challenge of using indirect experimental evidence from related molecules. I find this direction interesting, especially for properties where direct measurements are limited.
- The evaluation covers six molecular properties, two benchmarks, and three LLM backbones. The improvements over conventional predictors and the component ablations suggest that the proposed evidence selection is useful.
Critical weaknesses
- The paper does not experimentally compare with related methods such as MolRAG. Its main matched retrieval baseline uses fingerprint similarity, which is relatively simple. With the strongest backbone, the improvements over this baseline are only 2.4 and 4.5 macro-F1 points on the two benchmarks. While these gains are promising, I find the current comparisons insufficient to establish a convincing advantage over stronger retrieval-based methods.
- The literature-derived benchmark and retrieval corpus are constructed using the same automated extraction pipeline, making independent validation of data quality important. More importantly, Appendix E.2 states that the test cohorts were evaluated in earlier experiments. It is unclear whether these evaluations influenced method development. The relatively small TDC test sets and the lack of uncertainty estimates for single-rollout LLM predictions further weaken confidence in the reported gains.
Questions for authors
- How does Ampere compare with stronger retrieval-based methods such as MolRAG, or a simpler approach that ranks evidence directly using the learned transfer scores, under matched experimental settings?
- Were the previously evaluated test cohorts used to guide method development or hyperparameter choices?
- Can the authors provide repeated-run results or confidence intervals to establish the reliability of the reported improvements?
2 comments
Dear Reviewer,
Thank you again for your thoughtful feedback. We’ve included new experimental results in our rebuttal to address the questions you raised. When you have a chance, we’d welcome your thoughts and would be happy to discuss any remaining concerns.
We thank the reviewer for the thoughtful review. We've performed new experiments to incorporate your feedback and address some of the concerns you raised:
How does Ampere compare with stronger retrieval-based methods such as MolRAG, or a simpler approach that ranks evidence directly using the learned transfer scores, under matched experimental settings?
Our naive LLM baseline follows MolRAG's core RAG workflow: retrieving structurally similar labeled molecules and prompting an LLM. However, it uses a different prompt and larger retrieval budget. For a more direct comparison, we reproduced MolRAG's best-reported 4-shot Sim-CoT configuration. As no official code is available, we reconstructed it from the paper, preserving Morgan/Dice retrieval, four distinct references, and Sim-CoT reasoning. All other settings, including LLM backbones, are matched:
On all three LLMs, Ampere performs significantly better than MolRAG on both benchmarks.
Beyond the MolRAG baseline, we also have ablations in Table 1. The transfer tool is the largest individual contributor to our system, resulting in a 1.7 F1 reduction in performance across literature sourced datasets when removed.
Were the previously evaluated test cohorts used to guide method development or hyperparameter choices?
We apologize for the lack of clarity and poor writing in this appendix section. In all cases, we optimized hyperparameters exclusively on development data. Across all data settings, test sets were used to sanity check generalization before submission fewer than 5 times, and no model optimization or development took place on it. We will revise the description in the Appendix section accordingly to clarify.
Can the authors provide repeated-run results or confidence intervals to establish the reliability of the reported improvements? We thank the reviewer for this valuable suggestion. To assess the reproducibility and reliability of our reported improvements, we conducted three independent runs for all stochastic ML baselines and DeepSeek-based methods, reporting the mean ± standard deviation (SD) across runs. In general, the SD of performance of our models across both literature and TDC evaluation sets was less than 1 point, indicating our models reliably outperform the baselines we have considered.
The repeated-run results on both benchmarks are summarized below: