- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies overlap between molecular benchmarks and candidate pretraining corpora, arguing that exact matching is too narrow (misses salt-parent, stereochemical, and tautomeric relations) while generic similarity is too broad (retrieves neighbors without establishing which corpus record supports a declared relation). CHEMTRACE computes four deterministic, relation-specific normalization keys (exact canonical isomeric SMILES; largest-fragment + uncharge parent; stereochemistry-stripped connectivity; canonical tautomer) under a fixed RDKit snapshot, builds relation-specific hash indices over an inspectable corpus, and emits versioned JSON "provenance certificates" binding code / schema versions, input digests, and typed retrieval evidence — no model access required. Validation: (i) independently defined ChEMBL36 registry and Tautobase relations give 0.913-0.992 primary-relation recall with precision >= 0.996, and leave-one-route-out ablations show the three non-exact keys are non-redundant; (ii) under controlled injections into a fixed 50k-record corpus across six MoleculeNet datasets, CHEMTRACE achieves complete changed-only recall (1.000 over 78 cells) vs a descriptive oracle mean of ~0.84; (iii) under a 13-route common protocol with dose-matched decoys, it attains S of 0.99 vs best comparators at S around 0.88 with higher incidental retrieval; (iv) controlled pretraining / fine-tuning with a label-adjusted difference-in-differences estimator over 51 cells shows every bootstrap interval crossing zero — no consistent downstream performance inflation, so retrieval-confirmed exposure and model effect should be reported separately.
Critical strengths
- A genuinely novel, well-validated attribution primitive. Relation-typed, record-level provenance (which corpus record supports a declared parent / stereo / tautomer relation, in a versioned certificate) is not provided by exact deduplication or similarity-based leakage tools, and I verified against the literature that no prior work fills this gap. The keys are validated against independently defined external relations (ChEMBL36 hierarchy, Standard InChI stereo layers, curated Tautobase pairs) with high precision and recall, and leave-one-route-out ablations show each non-exact key contributes non-redundant coverage. This is consequential because it converts "the benchmark overlaps the corpus" from an unattributable similarity score into auditable, reproducible evidence — the property that makes downstream evaluation claims interpretable.
- A common-protocol comparison with honest negative controls. All 13 routes run on identical splits, intervention records, and corpora; dose-matched decoys are sampled with no exclusion criteria so incidental retrieval is measured rather than assumed away; the 500 deliberately separated registry negatives and the explicitly-labeled conditional hard-neighborhood bank separate construction checks from specificity claims. This is consequential because it is the difference between a method paper that demonstrates its coverage-specificity trade-off under one protocol and one that merely asserts it — and the paper's own caveats (oracle is descriptive; bank precision is a construction property) show the authors are not overclaiming.
Critical weaknesses
-
Evidence accessibility cannot be verified at review time. The Reproducibility Statement promises an anonymized supplementary code package (implementation, fixed retrieval inputs, tables underlying the results, SHA-256 manifest, regeneration commands), but no URL or link to code or data appears anywhere in the PDF. Every reported number — including the headline S >= 0.99 and all 51 downstream cells — therefore rests on the paper's own tables. This is consequential because the contribution is a reproducibility / provenance tool: if the package is missing or non-functional, the paper's central promise (regenerable certificates) is unverified. It appears fixable, which is why it does not by itself drive a rejection — but it must be resolved before acceptance.
-
Practical significance is supported only by synthetic injections and a small real-corpus proxy. The downstream analysis uses controlled injections into a 50k-record corpus with ~681k-parameter models on 10% training subsets, and the only real-corpus result is a ZINC15 proxy audit (24 typed matches among 67,228 records) that shows the method runs but not that it changes any real evaluation claim. The null RQ4 result is underpowered, so the paper cannot demonstrate that detected exposure actually inflates real model performance. This is consequential because it leaves the use case — auditing actual molecular foundation-model pretraining corpora — plausible but unproven end-to-end. The authors name this gap explicitly, which keeps it a weakness of evidence rather than of framing.
Questions for authors
- Where is the anonymized code / data package referenced in the Reproducibility Statement? The PDF contains no link. Please confirm it is attached to this submission and that the documented commands regenerate the retrieval analyses (Tables 1-2, C.1) and the downstream tables (F.2-F.3). If it is not accessible at review time, please say so — this directly affects my soundness assessment.
- The ChemRxiv preprint "When Is a Molecule a Duplicate? Identity Policy Determines What a Benchmark Audit Finds" (10.26434/chemrxiv.15009099, posted 20 Sept 2026) audits MoleculeNet benchmarks under a four-level identity hierarchy (canonical graph, fragment parent, charge-normalised parent, tautomer-normalised parent) closely resembling your four keys. It is not cited in Related Work. How does CHEMTRACE's attribution problem and certificate output differ from that work, and will you discuss it in revision?
- In Table F.2/F.3, primary cells use 4 or 12 paired seeds with median CI half-width 0.0443, and practical equivalence holds for only 2-3 of 51 cells. Given this power, what is the smallest loss reduction your design could have detected at reasonable probability, and do you consider "no consistent inflation" a claim about the tested regime only?
- In my view, an audit of a real molecular foundation-model pretraining corpus, with the certificate attached to an actual evaluation claim (e.g., re-reporting a published benchmark result with and without attributed exposure), would close the practical-significance gap. Is that possible to address this during rebuttal?
- Would you agree that powered downstream analysis (more paired seeds, or a pre-registered equivalence design with an explicit margin) would let RQ4 distinguish "no inflation" from "inflation too small to detect"?