- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors have argued that the current generative designers for molecules and others are not properly evaluated based on best of N yield since they show that design yield is a tail functional.
Critical strengths
Theoretical proofs on design yields.
Critical weaknesses
- Assuming the proofs are correct (which can be done by AI in 2026), the theorems only show trivial conclusions. It is well-known that best-of-N yields are not sufficient but it is practically relevant and fair for people to compete for. In particular, in the domain of drug molecule generation, there is not any infinite resource where people can test all generated molecules.
- From the molecular perspective, although molecular data are static, it is generally acknowledged that there exist flexibilities and dynamics in molecule formation in actual 3D space. Therefore, distributional generation is actually coherent with the underlying domain knowledge.
- The study is lack of comprehensive evaluations.
- The study is lack of practical implications on specific domains.
Questions for authors
The submission could be more suitable for AISTATS than ICLR.
2 comments
Thank you for the comments. I have read your comments and raised the score to 2.
Response to Reviewer GfbP
We thank the reviewer for the comments. Below we clarify two key distinctions regarding our problem setup and summarise the empirical and practical evidence in the revised manuscript.
Best-of-N as the deployment target. We agree that a campaign has a finite budget and that best-of-N is the fair and practically relevant quantity. The paper does not argue against it. It argues that the quantities generators are ranked by before a campaign (validity, uniqueness, novelty, FID-like scores, MMD, precision/recall, the GuacaMol distribution-learning score) do not order generators by what a best-of-N campaign delivers. Theorem 1 and Corollary 2 make that exact: for an objective unbounded above, these metrics can agree to within any ε while best-of-N yield differs by any amount, at any fixed budget. The finite budget sharpens the statement: the Lipschitz constant is BN for a bounded score, so the guarantee weakens as the budget grows, and it disappears entirely when the score is unbounded.
Training objective versus benchmark evaluation. We agree that modelling the data distribution is well motivated chemically, and nothing here argues against distribution learning as a training objective. The claim concerns evaluation: a number computed from the generator's whole output distribution does not rank generators by campaign yield, and the two metrics that are sensitive to the tail, W1 and the forward KL to the data, respond with the opposite sign, because the tail that raises yield is the tail that lies away from the data. Proposition 4 gives a family in which the rank correlation between distributional quality and yield is exactly −1, and Section 5.1 measures ρ=−0.700 and −0.745 on real chemistry.
Empirical scale and statistical rigour (Section 5.2). The measured study covers 48 trained samplers spanning three architectures, six or three checkpoints per training run and four decoding settings; 27 clear the validity floor. Each is evaluated on five objectives, two unbounded (logP, penalised logP) and three bounded and conventional (QED, negated synthetic accessibility, Tanimoto similarity to a target), at two deployment budgets, with the yield measured on draws held out from every predictor and with the campaign accounting that charges for invalid draws reported alongside. Furthermore, the revision adds (i) a two-level cluster bootstrap over training runs, which leaves the two tail statistics positive with intervals excluding zero in all 20 objective-by-budget cells, and (ii) an independent replication on 37 samplers from five runs trained from scratch with separate random seeds, on which the distribution-learning score reproduces its sign pattern (negative on logP, penalised logP and −SA; positive on QED and similarity) and the tail statistics stay positive on all five objectives.
Practical takeaway for molecular benchmarks (Sections 3 and 7). The paper proposes a concrete and cheap change to how a generative-design benchmark reports. On logP at N=1000, the sampler closest to the data by the GuacaMol distribution-learning score yields 7.43 while the best designer in the same population yields 9.63, a gap of 2.20 against a mean bootstrap standard error of 0.021. A benchmark that ranked by the distributional score would have put the wrong model first. The remedy needs no new sampling: the sample 99th percentile and a peaks-over-threshold estimate of E[MN] are computed from the draws a benchmark already makes, and the revision gives the exact best-of-N expectation of the peaks-over-threshold fit, an interval procedure and its measured coverage, together with a comparison against the observed maximum and the plug-in expected maximum at budgets on both sides of the sample size.
Relevance to ICLR. The object studied here is how generative models are evaluated and selected, measured on trained models, which is why we submitted it to the generative-models area. Section 7 states the practical recommendation as a reporting rule for benchmarks.