acceptodds
Under review as a conference paper at ICLR 2027

ChemTrace: Relation-Typed Provenance for Molecular Pretraining-Corpus Audits

Abstract

Molecular models are often pretrained on large chemical corpora and evaluated on reused public benchmarks. Potential overlap between benchmark molecules and pretraining records complicates the interpretation of model evaluation: exact matching misses standardized-parent, stereochemical, and tautomeric relations, whereas generic similarity cannot establish which candidate-corpus record supports a declared molecular relation. Our goal is to recover attributable molecular exposure—an auditable typed relation between a benchmark molecule and an inspectable candidate-corpus record—without requiring model access. We introduce CHEMTRACE, a provenance framework that indexes canonical identity, standardized parent, stereochemistry-stripped connectivity, and canonical tautomer form. It emits provenance certificates, versioned JSON records that bind fixed corpus, split, and intervention specifications to relation-typed retrieval evidence. Independently defined ChEMBL36 registry relations and curated Tautobase tautomer pairs yield 0.913–0.992 primary-relation recall across the three non-exact relations. CHEMTRACE recovers every changed non-exact exposure in the controlled audit, emits no certificate on 500 deliberately separated registry negatives, and remains silent across a 3,948-pair hard-neighborhood bank explicitly conditioned to be typed-relation-negative. Across six MoleculeNet datasets and 13 same-protocol audit routes, CHEMTRACE is the only evaluated method combining complete changed-only recall, a 0.0036 ± 0.0032 background hit rate, and zero decoy hits. A two-stage scaffold/seed bootstrap finds no consistently positive downstream inflation across 51 primary-estimator cells. Together, CHEMTRACE gives benchmark auditors a reproducible, record-level way to identify source-linked molecular exposure without treating chemical-neighborhood similarity as provenance; downstream model effects remain a separate empirical question.

Primary area:
datasets and benchmarks
Keywords:
Molecular machine learning, pretraining-corpus auditing, data provenance, benchmark contamination, chemical structure standardization, relation-typed retrieval, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wdhgOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies overlap between molecular benchmarks and candidate pretraining corpora, arguing that exact matching is too narrow (misses salt-parent, stereochemical, and tautomeric relations) while generic similarity is too broad (retrieves neighbors without establishing which corpus record supports a declared relation). CHEMTRACE computes four deterministic, relation-specific normalization keys (exact canonical isomeric SMILES; largest-fragment + uncharge parent; stereochemistry-stripped connectivity; canonical tautomer) under a fixed RDKit snapshot, builds relation-specific hash indices over an inspectable corpus, and emits versioned JSON "provenance certificates" binding code / schema versions, input digests, and typed retrieval evidence — no model access required. Validation: (i) independently defined ChEMBL36 registry and Tautobase relations give 0.913-0.992 primary-relation recall with precision >= 0.996, and leave-one-route-out ablations show the three non-exact keys are non-redundant; (ii) under controlled injections into a fixed 50k-record corpus across six MoleculeNet datasets, CHEMTRACE achieves complete changed-only recall (1.000 over 78 cells) vs a descriptive oracle mean of ~0.84; (iii) under a 13-route common protocol with dose-matched decoys, it attains S of 0.99 vs best comparators at S around 0.88 with higher incidental retrieval; (iv) controlled pretraining / fine-tuning with a label-adjusted difference-in-differences estimator over 51 cells shows every bootstrap interval crossing zero — no consistent downstream performance inflation, so retrieval-confirmed exposure and model effect should be reported separately.

Critical strengths
  1. A genuinely novel, well-validated attribution primitive. Relation-typed, record-level provenance (which corpus record supports a declared parent / stereo / tautomer relation, in a versioned certificate) is not provided by exact deduplication or similarity-based leakage tools, and I verified against the literature that no prior work fills this gap. The keys are validated against independently defined external relations (ChEMBL36 hierarchy, Standard InChI stereo layers, curated Tautobase pairs) with high precision and recall, and leave-one-route-out ablations show each non-exact key contributes non-redundant coverage. This is consequential because it converts "the benchmark overlaps the corpus" from an unattributable similarity score into auditable, reproducible evidence — the property that makes downstream evaluation claims interpretable.
  2. A common-protocol comparison with honest negative controls. All 13 routes run on identical splits, intervention records, and corpora; dose-matched decoys are sampled with no exclusion criteria so incidental retrieval is measured rather than assumed away; the 500 deliberately separated registry negatives and the explicitly-labeled conditional hard-neighborhood bank separate construction checks from specificity claims. This is consequential because it is the difference between a method paper that demonstrates its coverage-specificity trade-off under one protocol and one that merely asserts it — and the paper's own caveats (oracle is descriptive; bank precision is a construction property) show the authors are not overclaiming.
Critical weaknesses
  1. Evidence accessibility cannot be verified at review time. The Reproducibility Statement promises an anonymized supplementary code package (implementation, fixed retrieval inputs, tables underlying the results, SHA-256 manifest, regeneration commands), but no URL or link to code or data appears anywhere in the PDF. Every reported number — including the headline S >= 0.99 and all 51 downstream cells — therefore rests on the paper's own tables. This is consequential because the contribution is a reproducibility / provenance tool: if the package is missing or non-functional, the paper's central promise (regenerable certificates) is unverified. It appears fixable, which is why it does not by itself drive a rejection — but it must be resolved before acceptance.

  2. Practical significance is supported only by synthetic injections and a small real-corpus proxy. The downstream analysis uses controlled injections into a 50k-record corpus with ~681k-parameter models on 10% training subsets, and the only real-corpus result is a ZINC15 proxy audit (24 typed matches among 67,228 records) that shows the method runs but not that it changes any real evaluation claim. The null RQ4 result is underpowered, so the paper cannot demonstrate that detected exposure actually inflates real model performance. This is consequential because it leaves the use case — auditing actual molecular foundation-model pretraining corpora — plausible but unproven end-to-end. The authors name this gap explicitly, which keeps it a weakness of evidence rather than of framing.

Questions for authors
  1. Where is the anonymized code / data package referenced in the Reproducibility Statement? The PDF contains no link. Please confirm it is attached to this submission and that the documented commands regenerate the retrieval analyses (Tables 1-2, C.1) and the downstream tables (F.2-F.3). If it is not accessible at review time, please say so — this directly affects my soundness assessment.
  2. The ChemRxiv preprint "When Is a Molecule a Duplicate? Identity Policy Determines What a Benchmark Audit Finds" (10.26434/chemrxiv.15009099, posted 20 Sept 2026) audits MoleculeNet benchmarks under a four-level identity hierarchy (canonical graph, fragment parent, charge-normalised parent, tautomer-normalised parent) closely resembling your four keys. It is not cited in Related Work. How does CHEMTRACE's attribution problem and certificate output differ from that work, and will you discuss it in revision?
  3. In Table F.2/F.3, primary cells use 4 or 12 paired seeds with median CI half-width 0.0443, and practical equivalence holds for only 2-3 of 51 cells. Given this power, what is the smallest loss reduction your design could have detected at reasonable probability, and do you consider "no consistent inflation" a claim about the tested regime only?
  4. In my view, an audit of a real molecular foundation-model pretraining corpus, with the certificate attached to an actual evaluation claim (e.g., re-reporting a published benchmark result with and without attributed exposure), would close the practical-significance gap. Is that possible to address this during rebuttal?
  5. Would you agree that powered downstream analysis (more paired seeds, or a pre-registered equivalence design with an explicit margin) would let RQ4 distinguish "no inflation" from "inflation too small to detect"?
Reviewer_ofChOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Authors introduce a new way for auditing chemical datasets that are used for ML models pre-training to identify test data leakage. They argue that methods such as generic similarity and exact matching are not good enough for reliable cross-split overlap detection. They show that parent, stereo, and tautomer relations linkage between test and train molecules is a better for chemical training data de-contamination.

Critical strengths

Authors highlight the important problem of test data leakage and propose a new way to mitigate it in chemistry domain.

Critical weaknesses
  • The paper introduces its own evaluation metrics and does not rely on existing evaluation protocols for fair comparison of their method and at the same time don't provide a conclusive evidence that their decontamination method has any advantage for fairer chemical model evaluation ("downstream model effects remain a separate empirical question").
  • The paper content-wise is more related to chemistry-related venues rather than Machine Learning.
Questions for authors
  • Why the authors selected their own metrics (at least there's no reference to other works using them) such as coverage, incidental retrieval (table 2)? For example in [1] (fig. 5) authors use RMSE/ROC AUC of different models with and without injected noise (including contamination). This metrics is more illustrative of the specific decontamination methods, it's simple and easy to understand and on top of this, it would allow direct comparison with the literature.
  • In this work [2], authors use clustering based on ECFP4 Tanimoto similarity (≥ 0.95) and , why the authors use Tanimoto ≥ 0.80
  • Why was only one dataset used for audit (MoleculeNet)?

References

  1. Schuh, Maximilian G., Aleksandra Daniluk, and Stephan A. Sieber. "Auditing widely used biomolecular benchmarks reveals systematic data inconsistencies." Chemical science 17.36 (2026): 17455-17467.
  2. Parrondo-Pizarro, Raquel, et al. "Enhancing molecular property prediction through data integration and consistency assessment." Journal of cheminformatics 17.1 (2025): 163.
Reviewer_3T8hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses training-evaluation overlap in molecular pretraining corpora without relying on brittle exact-string matching or overly broad fingerprint/scaffold similarity. The authors introduce ChemTrace, a corpus-side auditing framework that indexes candidate pretraining sets across four deterministic RDKit normalization routes: canonical identity, standardized parent, stereochemistry-stripped connectivity, and canonical tautomer. It represents matches as a relation-typed bipartite graph linking benchmark queries to specific corpus records and outputs versioned JSON provenance certificates for reproducibility. For validation, the authors test the normalization keys against independent positives from ChEMBL36 and Tautobase, separated registry negatives, and a 3,948-pair hard-neighborhood stress bank, alongside a 13-route comparison and a ZINC15 proxy audit. Finally, through controlled pretraining interventions across six MoleculeNet datasets (using Transformer, BiGRU, and GIN encoders against length-matched decoys), a label-adjusted difference-in-differences analysis finds no consistent downstream performance inflation across 51 evaluation cells, leading the authors to argue that corpus provenance and downstream model effects should be evaluated separately.

Critical strengths
  1. The paper draws a clear, practical distinction between identifying a chemical relationship in a corpus and claiming benchmark contamination. Exact matches, salt/charge forms, stereoisomers, and tautomers have different implications for an audit, and tracking these relation types alongside their source records is much more informative than a single overlap score. The authors are also careful not to overclaim in the manuscript.

  2. The retrieval keys are validated against external reference data (ChEMBL36 and Tautobase) rather than relying purely on synthetic transformations generated by RDKit itself, achieving strong recall across parent (99.2%), stereo (96.15%), and tautomer (91.34%) relations. The downstream intervention study is also cleanly controlled via fixed corpus sizes, length-matched decoys, and paired seeds, and the paper carefully separates statistically inconclusive intervals from actual practical equivalence.

Critical weaknesses
  1. It is not yet clear how much practical value the four separate routes add over a standard all-in-one standardization pipeline like RDKit’s SuperParent (currently tucked away in Appendix A.5). In Table A.3, SuperParent actually achieves higher external recall than ChemTrace (97.84% vs. 96.54%) while matching its 100% changed-only recall. Because ChemTrace evaluates four atomic routes independently, it naturally misses composite transformations (e.g., a salt that also differs in stereochemistry or tautomerism) that SuperParent captures. Understanding where and why these two methods disagree is central to justifying the proposed framework, and the SuperParent baseline should really belong in the main text.

  2. The downstream experiments provide limited evidence about the size of exposure effects, even within the tested setting. The primary setup pretrains a 0.68M-parameter Transformer for just 4 epochs on 50,000 molecules, and several non-exact cells contain very few changed test molecules (as few as 5 to 41 entries in Appendix F.1). While all 51 confidence intervals cross zero, only 2 out of 51 fall within the paper’s own ±0.01\pm 0.01 practical-equivalence margin (and 3 within ±0.02\pm 0.02). Most cells could therefore just be statistically noisy. If the experimental setup cannot reliably detect a positive effect even under an extreme exposure condition (e.g., high-dose exact repetition or a larger capacity regime), we cannot distinguish genuine robustness to chemical exposure from a testbed with low sensitivity. As a result, these experiments say little about what happens when overlaps occur in large-scale molecular foundation models.

Questions for authors
  1. Could you inspect a sample of pairs where SuperParent triggers a match but ChemTrace does not? Specifically, how many of those cases are valid composite relationships (e.g., salt + stereoisomer/tautomer differences) versus spurious matches that would mislead an auditor?

  2. The ZINC15 proxy audit identifies 24 matches (4 exact, 18 stereo, 2 tautomer). Could you walk through a few concrete examples to illustrate what an auditor does differently given ChemTrace’s typed labels and certificates compared to simply matching SuperParent structures with source IDs?

  3. How small of an effect can the current downstream setup actually detect, and is there a positive-control setting (e.g., higher injection dose or longer pretraining) where this pipeline consistently shows a performance gain? This would help clarify whether the 51 null intervals reflect true invariance or low statistical power.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.