- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper asks whether pretraining-corpus composition is a lever on the diversity–activity trade-off of a molecular generator. An LLM agent (Qwen-3.5-27B, autoresearch-style loop plus a "Grill-me" clarification step) proposes mixtures over ChEMBL/ZINC/COCONUT/HMDB and sampling operators. A 2.8M-parameter SMILES GPT is pretrained on 500K molecules, fine-tuned on SYK inhibitors, and scored by S = scaffold entropy × mean predicted pIC50 of the top-50 generated molecules (QSAR). Over ~22 iterations the best mixture is "ZINC molecules farthest from ChEMBL". It is reported as +1.6% over a 500K-ChEMBL baseline, attributed to higher scaffold entropy, and said to rival architectural scaling.
Critical strengths
- Code and agent instructions are released, agent failure modes are disclosed (out-of-space proposals, wrong epoch selection), and the selected and baseline configurations are re-run over five seeds. Seeds 1–4 are independent of the selection run.
- The question is reasonable, and the observation that moderate ChEMBL/ZINC/natural-product mixes do not help is a useful negative data point. AI involvement is disclosed; my concerns are verifiability and scope, not authorship.
Critical weaknesses
- The headline is not robust and the conclusions exceed the evidence.
- "+1.6%" is seed 0 vs seed 0 (9.093 vs 8.949), the run used for selection. Five-seed means are 9.035 vs 8.956 (+0.9%), and about 9.021 vs 8.957 (+0.7%) on seeds 1–4. Seed 0 is an outlier (entropy 0.970 vs 0.947–0.951; validity 0.506 vs 0.635–0.652).
- Fig. 5 shows the reported scores are per-run peaks. The "BEST" run spikes at epoch 2 (≈9.09), but at the authors' own recommended stop (epoch 3) it is ≈8.88, below the ChEMBL baseline (≈8.95) and, by my reading, the other mixtures shown. Taking the maximum over epochs on the objective itself is optimistic and favours spiky curves. The epoch used in Tables 3–4 is not stated.
- Under the authors' Bonferroni correction only the composite remains significant (p=0.0026; entropy p=0.048, pIC50 p=0.27, validity p=0.16). The text nevertheless asserts "diversity-driven" and quotes p<0.05 for entropy.
- The QSAR model is both objective and evaluator. Validity drops (0.62 vs 0.67; 0.51 in the selected run). There is no uniqueness, novelty, SA, or distance-to-training-ligands analysis, and the N behind the top-50 is unspecified.
- No scaling experiment supports "rival/more impactful than architectural scaling".
- The agent's contribution is untested, and figures/tables are internally inconsistent.
- The search space is tiny (4 corpora, mixture weights, 7 operators). The best configuration (found ~iteration 6, Fig. 3) amounts to "take the molecules farthest from ChEMBL", a one-line human hypothesis.
- There is no random/grid/BO baseline and no repeated agent runs or LLM variants. Roughly half of the ~22 trials sit at 8.97–9.03, close to seed noise (baseline SD ≈0.02). Picking the best of ~22 noisy trials invites winner's curse.
- Fig. 2: n=23 vs 16 mixtures in Table 2. R²=0.86 is largely arithmetic (S=H×pIC50, with pIC50 only 9.29–9.62). Entropy and pIC50 trade off (best point 9.375 vs baseline 9.482). The SYK reference point lies on the fitted line and, if included in n=23, adds leverage.
- Fig. 4: the farthest-ZINC subset is a compact island in UMAP, possibly homogeneous. Internal diversity and size/charge distributions are not reported, and under-adaptation to the target is not excluded.
- Table 2 lists three different mixtures with identical pIC50/scaffold-count/validity (9.455/31/0.633), and two rows with the same label but different results (8.637 vs 9.021).
- Text vs appendix: 9.025 vs 9.0354; entropy SD 0.001 vs 0.0095; top-50 vs top-1000; vocabulary 165 vs 512.
Questions for authors
-
Which epoch is reported in Tables 2–4 for each arm and seed? Provide per-seed trajectories and a fixed-epoch comparison.
-
Explain the identical Table 2 rows and the duplicate label, and release raw logs for all runs.
-
Add random/grid/BO baselines with the same trial budget, and repeat the agent search ≥3 times or with other LLMs.
-
Define scaffold entropy (which set, normalization) and N generated. Does the QSAR training set overlap the SYK fine-tuning set? What fraction of the top-50 are near-duplicates of SYK ligands?
-
(suggestion) Add a tuned ChEMBL-only baseline (epochs, temperature), more targets, and a real model-size scaling comparison if parity is claimed.