acceptodds
Under review as a conference paper at ICLR 2027

Design yield is a tail functional: why distributional metrics cannot rank generative designers

12 views

Abstract

Generative models for molecules, binders, and materials are commonly evaluated with distributional metrics and then deployed by drawing candidates and retaining the best. We study the mismatch between these two procedures. Best-of- yield depends on a generator only through the upper tail of its pushforward score distribution. For scores bounded in , yield is Lipschitz in total variation with sharp constant , so the guarantee weakens linearly with the deployment budget. For scores unbounded above, yield has no modulus of continuity: metrics built from finitely many bounded test functions, including validity, uniqueness, novelty, FID-like scores, MMD, and precision/recall, can remain arbitrarily close while yield differs arbitrarily. The two metrics that respond to this construction, and forward KL to the data, penalise the same tail mass that improves design yield. We give a family with rank correlation between distributional quality and yield, and construct generators with identical forward divergence and identical moments to any prescribed order whose sharp yield gap is determined by a linear program. We also evaluate a peaks-over-threshold estimator of and measure its calibration. Across 27 trained samplers from three architectures and five objectives, with yield held out from every predictor, the standard distribution-learning score is negatively correlated with design yield on three objectives. Two tail statistics are positively correlated with yield on all five, with bootstrap intervals excluding zero. On , the sampler closest to the data trails the highest-yield sampler by about a hundred bootstrap standard errors. These quantities can be computed from samples that a generative benchmark already draws.

Primary area:
generative models
Keywords:
generative models, generative design, molecular design, model evaluation, best-of-N sampling, extreme value theory, distributional metrics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GfbPOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors have argued that the current generative designers for molecules and others are not properly evaluated based on best of N yield since they show that design yield is a tail functional.

Critical strengths

Theoretical proofs on design yields.

Critical weaknesses
  1. Assuming the proofs are correct (which can be done by AI in 2026), the theorems only show trivial conclusions. It is well-known that best-of-N yields are not sufficient but it is practically relevant and fair for people to compete for. In particular, in the domain of drug molecule generation, there is not any infinite resource where people can test all generated molecules.
  2. From the molecular perspective, although molecular data are static, it is generally acknowledged that there exist flexibilities and dynamics in molecule formation in actual 3D space. Therefore, distributional generation is actually coherent with the underlying domain knowledge.
  3. The study is lack of comprehensive evaluations.
  4. The study is lack of practical implications on specific domains.
Questions for authors

The submission could be more suitable for AISTATS than ICLR.

2 comments
AuthorsOpenReview ↗

Response to Reviewer GfbP

We thank the reviewer for the comments. Below we clarify two key distinctions regarding our problem setup and summarise the empirical and practical evidence in the revised manuscript.

Best-of-NN as the deployment target. We agree that a campaign has a finite budget and that best-of-NN is the fair and practically relevant quantity. The paper does not argue against it. It argues that the quantities generators are ranked by before a campaign (validity, uniqueness, novelty, FID-like scores, MMD, precision/recall, the GuacaMol distribution-learning score) do not order generators by what a best-of-NN campaign delivers. Theorem 1 and Corollary 2 make that exact: for an objective unbounded above, these metrics can agree to within any ε\varepsilon while best-of-NN yield differs by any amount, at any fixed budget. The finite budget sharpens the statement: the Lipschitz constant is BNBN for a bounded score, so the guarantee weakens as the budget grows, and it disappears entirely when the score is unbounded.

Training objective versus benchmark evaluation. We agree that modelling the data distribution is well motivated chemically, and nothing here argues against distribution learning as a training objective. The claim concerns evaluation: a number computed from the generator's whole output distribution does not rank generators by campaign yield, and the two metrics that are sensitive to the tail, W1W_1 and the forward KL to the data, respond with the opposite sign, because the tail that raises yield is the tail that lies away from the data. Proposition 4 gives a family in which the rank correlation between distributional quality and yield is exactly −1-1, and Section 5.1 measures ρ=−0.700\rho=-0.700 and −0.745-0.745 on real chemistry.

Empirical scale and statistical rigour (Section 5.2). The measured study covers 48 trained samplers spanning three architectures, six or three checkpoints per training run and four decoding settings; 27 clear the validity floor. Each is evaluated on five objectives, two unbounded (log⁡P\log P, penalised log⁡P\log P) and three bounded and conventional (QED, negated synthetic accessibility, Tanimoto similarity to a target), at two deployment budgets, with the yield measured on draws held out from every predictor and with the campaign accounting that charges for invalid draws reported alongside. Furthermore, the revision adds (i) a two-level cluster bootstrap over training runs, which leaves the two tail statistics positive with intervals excluding zero in all 20 objective-by-budget cells, and (ii) an independent replication on 37 samplers from five runs trained from scratch with separate random seeds, on which the distribution-learning score reproduces its sign pattern (negative on log⁡P\log P, penalised log⁡P\log P and −-SA; positive on QED and similarity) and the tail statistics stay positive on all five objectives.

Practical takeaway for molecular benchmarks (Sections 3 and 7). The paper proposes a concrete and cheap change to how a generative-design benchmark reports. On log⁡P\log P at N=1000N=1000, the sampler closest to the data by the GuacaMol distribution-learning score yields 7.43 while the best designer in the same population yields 9.63, a gap of 2.20 against a mean bootstrap standard error of 0.021. A benchmark that ranked by the distributional score would have put the wrong model first. The remedy needs no new sampling: the sample 99th percentile and a peaks-over-threshold estimate of E[MN]\mathbb{E}[M_N] are computed from the draws a benchmark already makes, and the revision gives the exact best-of-NN expectation of the peaks-over-threshold fit, an interval procedure and its measured coverage, together with a comparison against the observed maximum and the plug-in expected maximum at budgets on both sides of the sample size.

Relevance to ICLR. The object studied here is how generative models are evaluated and selected, measured on trained models, which is why we submitted it to the generative-models area. Section 7 states the practical recommendation as a reporting rule for benchmarks.

Thank you for the comments. I have read your comments and raised the score to 2.

Reviewer_WkFQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a generator that closely matches a reference dataset will also produce better designs. It defines yield as the average best score from N generated candidates, over repeated trials. The authors show that generators can look similar under measures of their output distributions yet have different yields. They also show how the number of candidates, affects this difference.

The paper proposes estimating yield from the highest scores in a sample and tests this idea on artificial examples and molecules. The study of trained models covers 27 generator settings and 5 objectives. It uses an initial sample to predict yield and a separate sample to estimate the yield achieved. In this study, predictors based on high scores rank generators more successfully. Greater similarity to the reference can go with either higher or lower yield. The central finding is useful, but some mathematical claims need correction, and the proposed estimator needs clearer details and stronger tests.

Critical strengths
  1. The theory explains why reference similarity need not predict design yield. Generators can match reference-based measures, including moments such as the mean and variance, yet differ in their expected best score. Independent calculations support the central examples. The dependence on the number of generated candidates makes the finding relevant to evaluation budgets.

  2. The molecular study shows that this ranking problem can occur in practice. The reference-based score sometimes favors lower-yield generators, while high-score predictors correlate positively with yield on all five objectives. Predictors and yield use separate generated samples, and an extra analysis counts invalid outputs against the budget. These controls strengthen the case-study evidence.

Critical weaknesses
  1. Some mathematical claims exceed what is proved. Proposition 5's condition does not guarantee a peak at an intermediate optimization strength. Consider two outcomes: probability 0.99, true score 1, proxy score 2; and probability 0.01, true score 6, proxy score 1. Optimizing the proxy makes the better outcome less likely, so true yield falls throughout for every N. Corollary 2 also needs extra assumptions or separate arguments for some named metrics. The broader wording about a "sharp gap" should retain Theorem 3’s asymptotic qualification and the implementation’s restriction to a chosen score grid. These issues need correction, although the central separation result survives.

  2. The proposed yield estimator needs clearer details and stronger comparisons. The peaks-over-threshold (POT) method lacks a complete expected-maximum formula and interval-adjustment procedure. Reported 90% intervals cover only 50–83% of true values. The main trained-model table omits simple baselines, including the observed maximum and direct expected-maximum estimates. Failed fits and prediction errors need comparison on the same cases. The 27 generator settings are also related, limiting what their uncertainty estimates establish about new models. Missing code, raw scores, and details of the held-out yield calculation prevent a full check of the practical recommendation.

Questions for authors
  1. Can you correct Proposition 5 in light of the counterexample above and provide the exact probabilities and scores for the paper's three-outcome example?

  2. What are the exact POT yield formula and interval-adjustment procedure? How often do the adjusted intervals contain the true yield with 1000 observed scores?

  3. How does POT compare with the observed maximum and direct expected-maximum estimates on the same cases and budgets, including N larger than the observed sample size? How are failed fits handled?

  4. Can you state the assumptions or separate arguments that justify applying Corollary 2 to FID, MMD, and the implemented GuacaMol score?

  5. Can you release the code, splits, scoring rules, linear-programming inputs, and individual scores? Please identify shared training runs and specify the held-out yield and uncertainty calculations.

5 comments
AuthorsOpenReview ↗

Response to Reviewer WkFQ (part 1/2)

(1/2)

We thank the reviewer for the detailed technical critique and the insightful two-atom counterexample. We have incorporated every mathematical correction, closed-form derivation, and baseline comparison into the revised manuscript and supplementary code.

Q1 (Proposition 5 and the three-outcome instance). Your counterexample is right: continuity plus exceeding the limit does not give an interior maximum. The revised Proposition 5 replaces that step with the derivative itself. With qλ∝p eλs^q_\lambda\propto p\ e^{\lambda \hat s}, FλF_\lambda the law of the true score under qλq_\lambda, and GNλ(z)=∫smin⁡zFλ(u)N−1duG^\lambda_N(z)=\int_{s_{\min}}^z F_\lambda(u)^{N-1}du, it states that λ↦Eqλ[s^]\lambda\mapsto\mathbb{E}_{q_\lambda}[\hat s] is strictly increasing; that ddλE[MN(qλ)]=N Covqλ(s^(x),GNλ(s(x)))\frac{d}{d\lambda}\mathbb{E}[M_N(q_\lambda)]=N\ \mathrm{Cov}_{q_\lambda}(\hat s(x),G^\lambda_N(s(x))); that E[MN(qλ)]→E[MN(pA)]\mathbb{E}[M_N(q_\lambda)]\to\mathbb{E}[M_N(p_A)] with A=arg⁡max⁡s^A=\arg\max\hat s; that a positive slope at 00, together with that limit lying below E[MN(p)]\mathbb{E}[M_N(p)], gives an interior maximum strictly above both endpoints, while a negative slope at 00 means true yield falls from the start; and that for s^=s\hat s=s the slope is non-negative at every λ\lambda and NN. Appendix A.4 now proves all five parts, the derivative by differentiating FλF_\lambda and integrating by parts in E[MN]=smax⁡−∫FλN\mathbb{E}[M_N]=s_{\max}-\int F_\lambda^N, and the interior maximum by compactness on [0,Λ][0,\Lambda].

The paper's instance is p=(0.70,0.20,0.10)p=(0.70,0.20,0.10), s=(0,1,6)s=(0,1,6), s^=(0,4,1.5)\hat s=(0,4,1.5). At N=10N=10 the slope at λ=0\lambda=0 is +1.33+1.33, E[M10]\mathbb{E}[M_{10}] rises from 4.234.23 to 4.364.36 at λ⋆=0.19\lambda^\star=0.19 and falls to 1.001.00, a 77% loss from the optimum, while the surrogate mean rises from 0.950.95 to 4.004.00. Your instance, p=(0.99,0.01)p=(0.99,0.01), s=(1,6)s=(1,6), s^=(2,1)\hat s=(2,1), has slope −0.45-0.45 at N=10N=10 and −1.83-1.83 at N=100N=100, and true yield decreases from λ=0\lambda=0; it is now in the paper as the second case of part (iv). The derivative formula is checked against finite differences to a worst relative error of 9.5×10−69.5\times10^{-6} over 60 random instances at three budgets and three tilt strengths, and the s^=s\hat s=s claim over 600 cases, in the shipped verification suite (72 checks, 0 failures).

Q4 (assumptions behind Corollary 2 for FID, MMD and GuacaMol). Corollary 2 now has two parts. Part (a) is the population statement. Part (b) is the one a benchmark meets: for any statistic D^=T(x1,…,xn)\hat D=T(x_1,\dots,x_n) of nn i.i.d. draws taking values in an interval of length CC, ∣EqnD^−Eq′nD^∣≤C n dTV(q,q′)|\mathbb{E}_{q^n}\hat D-\mathbb{E}_{q'^n}\hat D|\le C\ n\ d_{\mathrm{TV}}(q,q'), by subadditivity of total variation over product measures. With the constant stated this covers validity, uniqueness, novelty, precision, recall and coverage (C=1C=1); density (C=M/kC=M/k for MM reference points and kk neighbours); the GuacaMol distribution-learning score, which is a mean of exp⁡(−KL)\exp(-\mathrm{KL}) terms and so lies in [0,1][0,1] whatever the descriptor histograms are; and the biased MMD$^2$ estimate with a kernel in [0,K][0,K], which lies in [0,2K][0,2K]. FID is covered by (a), or by (b) when its feature map is bounded on the design space, which holds for any extractor with saturating or clipped outputs. The bound is verified numerically on the distinct fraction over 2,000 random pairs.

Theorem 3 (wording of the sharp gap). The limit is now stated as asymptotic in NN and on a fixed finite score grid, where Tm⋆T^\star_m is the value of a linear feasibility problem, with the finite-NN gap reported from the exact best-of-NN expectation. The abstract, the contributions and the Figure 1 caption carry the same qualification.

Continued in part 2/2 below.

AuthorsOpenReview ↗

Response to Reviewer WkFQ (part 2/2)

(2/2)

Q2 (the exact POT formula, the interval, and its coverage at n=1000n=1000). The fitted model is the empirical body below the threshold uu plus a generalised Pareto above it carrying mass ζ\zeta. Its best-of-NN expectation is closed form, Eq. (4) in the revision: E[MN]=BN+u(1−(1−ζ)N)+σξ[NζξB(1−ξ,N)Iζ(1−ξ,N)−(1−(1−ζ)N)]\mathbb{E}[M_N]=B_N+u(1-(1-\zeta)^N)+\frac{\sigma}{\xi}[N\zeta^{\xi}\mathrm{B}(1-\xi,N)I_\zeta(1-\xi,N)-(1-(1-\zeta)^N)], with BNB_N the exact body term and a separate exponential branch at ξ=0\xi=0. It is finite exactly when ξ<1\xi< 1, which is the condition for the fitted law to have a mean. It agrees with Monte Carlo wherever Monte Carlo is reliable, ∣z∣≤1.6|z|\le1.6 over 40 cells with ξ≤0.4\xi\le0.4, and stays exact at ξ=0.8\xi=0.8, where the fitted law has infinite variance and the Monte Carlo average does not settle.

On the interval, you identified the cause correctly: at n=1000n=1000 the error is dominated by threshold bias, which resampling a single fit cannot detect. We therefore take the envelope over $q\in\lbrace 0.90,0.925,0.95,0.975\rbrace $ of the per-threshold percentile intervals. Measured on 7 score laws by 5 budgets with 200 pools of 1,000 scores each, one threshold gives median coverage 0.80 with a minimum of 0.64 and misses the nominal 0.90 in 25 of 35 cells; the envelope gives 0.88 with a minimum of 0.74 at 1.4 times the width and misses in 5 cells; the envelope widened by one global factor of 1.6 gives 0.94 with a minimum of 0.88 at 2.25 times the width and misses in none.

Q3 (against the observed maximum and the plug-in, including NN beyond the pool). All four predictors see the same draws. The observed maximum is biased low by 0.49 to 3.16 in relative terms at N=10N=10, and beyond the pool it cannot move at all, since no order statistic exceeds the largest score drawn; the plug-in expected maximum under the empirical law has the same ceiling. At N=104N=10^4 and 10510^5 against a pool of 10310^3, the tail fit has the lower median error in 12 of 12 parametric law-by-budget cells, by a median factor of 1.37. Within the pool the three agree. Fits are reported with status rather than dropped: the two failure modes are fewer than 20 exceedances and ξ^≥1\hat\xi\ge1, which leaves the fitted law without a mean, and in this study 100% of fits returned a finite prediction. Figure 2(b,c) and Table A1 give the per-law numbers.

Q5 (release, shared training runs, and the held-out calculation). The supplementary material contains the scoring code, the per-sampler draw streams with their cached per-molecule scores, the linear-programming inputs, the splits and the analysis scripts. The held-out calculation is specified in Section 5.2: each sampler's draw stream is split at its 1,000th valid molecule, every predictor is computed before the cut, and E[MN]\mathbb{E}[M_N] is measured after it, with percentile bootstrap intervals. The population's 27 samplers come from 4 training runs, which the text now states and uses. A two-level bootstrap over runs leaves the tail statistics positive with intervals excluding zero in 20 of 20 objective-by-budget cells, and within each run with enough samplers the sign matches the pooled sign in 60 of 60 cells. We also repeated the study on independently trained generators, with different seeds and the same recipe: on 37 samplers from 5 independent runs the distribution-learning score keeps its sign pattern and the tail statistics stay positive on all five objectives, with run-clustered intervals excluding zero in 19 of 20 cells.

We hope these mathematical corrections, closed-form derivations, and multi-run replications resolve your concerns, and we welcome any further questions during the discussion period.

Thank you again for your detailed review. Before the discussion period closes, we wanted to check whether the revised Proposition 5 (which now covers your two-atom counterexample, with the full derivative proof in Appendix A.4), the closed-form POT expectation and threshold-envelope coverage study (Eq. 4, Figure 2b,c, Table A1), Corollary 2(b), and the multi-run replications, released with the code, data splits and generated samples, resolve the questions in your review. We would be glad to clarify any remaining points.

Reviewer_ZhrXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper demonstrates the mismatch between distributional evaluation metrics and the actual performance of generative models in design applications, where only the candidates with the top scores are selected. The authors proposed to formulate the design yield by expected maximum score and show that it is primarily governed by the upper tail of the generated score distribution. They establishes a sharp continuity bound for bounded scoring functions and show that, for unbounded objectives, arbitrarily small differences in total variation can correspond to arbitrarily large differences in design yield. They also investigate properties for KL divergence and matching score moments, as well as extreme value theory to predict design yield at a sampling budgets.

Critical strengths
  1. The paper identifies the difference between distribution learning and deployment. The training objective may not reflect the need of selecting the best candidate from a finite sampling budget.
  2. The continuity bound in Theorem 1 provides a quantitative characterization of how distributional differences affect best-of-N yield. Theorem 3 demonstrates that even matching KL divergence and finitely many score moments does not uniquely determine extreme outcomes. The math looks sound.
Critical weaknesses
  1. Molecular objectives are not equivalent to experimental design success. For example, a very high logP score is not necessarily desirable for drug discovery. Thus, a model that generates an extremely high logP molecule does not guarantee it generated a better therapeutic candidate.
  2. Proposition 5 seems to need further justification or additional assumptions. In paritcular, the authors states "since E[MN(qλ)]\mathbb{E}[M_N(q_\lambda)]is continuous in λ\lambda and exceeds its limit somewhere, it is non-monotone and attains an interior maximum." However, a function can be continuous, exceed its limiting value, and still be monotonically decreasing, e.g. a+be−kλa + be^{-k\lambda} for k>0,b>0k>0, b>0.
Questions for authors
  1. Can the author make a better justification on proposition 5?
  2. How should design yield be interpreted when the scoring function is imperfect or misaligned with practical design success?
  3. The trained-generator experiment has multiple samplers derived from checkpoints and decoding settings of the same underlying models. Could the authors clarify how they handle dependence among these samplers in the bootstrap confidence intervals? Would the results remain consistent with independently trained generators or uncertainty estimates that account for shared training runs?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ZhrX

We thank the reviewer for catching the gap in the original argument for Proposition 5 and for raising the questions on objective alignment and sampler dependence. All three points are addressed in the revised manuscript.

Q1 (justification for Proposition 5). As your example shows, continuity plus exceeding the limit does not give an interior maximum, and we have replaced the argument with an exact one. For pp with finite support, qλ∝p eλs^q_\lambda \propto p\ e^{\lambda \hat s}, FλF_\lambda the law of the true score ss under qλq_\lambda, and GNλ(z)=∫smin⁡zFλ(u)N−1duG^\lambda_N(z)=\int_{s_{\min}}^z F_\lambda(u)^{N-1}du, the revised Proposition 5 states that λ↦Eqλ[s^]\lambda\mapsto\mathbb{E}_{q_\lambda}[\hat s] is strictly increasing; that ddλE[MN(qλ)]=N Covqλ(s^(x),GNλ(s(x)))\frac{d}{d\lambda}\mathbb{E}[M_N(q_\lambda)]=N\ \mathrm{Cov}_{q_\lambda}(\hat s(x),G^\lambda_N(s(x))); that E[MN(qλ)]→E[MN(pA)]\mathbb{E}[M_N(q_\lambda)]\to\mathbb{E}[M_N(p_A)] as λ→∞\lambda\to\infty, where pAp_A is pp conditioned on A=arg⁡max⁡s^A=\arg\max\hat s; that a positive slope at λ=0\lambda=0, together with that limit lying below E[MN(p)]\mathbb{E}[M_N(p)], gives an interior maximum strictly above both endpoints, while a negative slope at λ=0\lambda=0 means true yield falls from the start; and that for s^=s\hat s=s the slope is non-negative at every λ\lambda and every NN. Appendix A.4 gives the complete proof of all five parts.

The derivative formula is checked against finite differences to a worst relative error of 9.5×10−69.5\times10^{-6} over 60 random instances at three budgets and three tilt strengths, and the no-harm claim for s^=s\hat s=s over 600 random cases, in the shipped verification suite.

In the paper's instance p=(0.70,0.20,0.10)p=(0.70,0.20,0.10), s=(0,1,6)s=(0,1,6), s^=(0,4,1.5)\hat s=(0,4,1.5), the slope at λ=0\lambda=0 is +1.33+1.33 at N=10N=10 and true yield rises from 4.234.23 to a maximum of 4.364.36 at λ⋆=0.19\lambda^\star=0.19 before falling to 1.001.00. The two-atom instance p=(0.99,0.01)p=(0.99,0.01), s=(1,6)s=(1,6), s^=(2,1)\hat s=(2,1) has slope −0.45-0.45 at N=10N=10 and −1.83-1.83 at N=100N=100, and true yield decreases from λ=0\lambda=0 at every budget. This second case shows that the initial-slope condition in part (iv) cannot be dropped: without it, tilting can lower true yield from the start.

Q2 (a high log⁡P\log P is not a better therapeutic candidate). Agreed, and the paper's claim is conditional on the scoring rule rather than on log⁡P\log P being the right one: for a fixed scoring rule ss, a benchmark that ranks generators by a distributional metric does not rank them by what a best-of-NN campaign under ss delivers.

The case where ss is an imperfect proxy is the subject of Proposition 5, which treats optimisation against a surrogate s^\hat s that misranks a rare true winner. The corrected statement says when that is harmful and shows the harm cannot be detected from the surrogate's own statistics. Section 4 now states this link explicitly: when the benchmark score is itself a proxy for a downstream property, as log⁡P\log P is for therapeutic value, Proposition 5 applies with the benchmark score in the role of s^\hat s.

The measured study also does not rest on log⁡P\log P. It uses five objectives, three of them bounded and chemically conventional (QED, negated synthetic accessibility, Tanimoto similarity to a target), and the conclusion is the same on all five: the distribution-learning score is anti-correlated with yield on log⁡P\log P, penalised log⁡P\log P and −-SA, correlated on QED and similarity, while the two tail statistics are positively correlated on all five.

Q3 (dependence among samplers in the bootstrap). The conclusion survives two stricter treatments, both new in the revision.

Resampling training runs. The 27 samplers come from four training runs: two for the transformer, one each for the small transformer and the recurrent model. A two-level bootstrap that resamples runs with replacement and then samplers within each drawn run gives, for the two tail statistics, intervals that exclude zero in all 20 objective-by-budget cells, for example the sample 99th percentile on log⁡P\log P at N=1000N=1000 at ρ=+0.81\rho=+0.81, [+0.36,+0.95][+0.36,+0.95]. Inside each training run with enough samplers to compute a rank correlation, the sign agrees with the pooled sign in 60 of 60 cells, so the association arises within a single training trajectory rather than from comparing runs.

Independently trained generators. We trained a second population of eight runs from different seeds, four each for the small transformer and the recurrent model, sharing only the recipe: three checkpoints per run, the same four decoding settings, the same data split and tokeniser. On the 37 samplers that clear the same validity floor, drawn from five of these runs, the distribution-learning score keeps the same sign pattern, negative on log⁡P\log P, penalised log⁡P\log P and −-SA and positive on QED and similarity, and the tail statistics are positive on all five objectives with run-clustered intervals excluding zero in 19 of 20 cells.

We hope the revision fully address your concerns.

Thank you again for your review. Before the discussion period closes, we wanted to check whether the exact derivative characterisation and proof for Proposition 5 (Section 4, Appendix A.4), the explicit link to imperfect proxy scores, and the two-level run-clustered bootstrap together with the replication on 37 samplers from five independently trained runs (Section 5.2) address all three of your questions. We would be happy to clarify any remaining points.

Dear Reviewer,

This is a friendly reminder that the ICLR 2027 discussion phase will end on November 18 at 11:59 PM AoE.

Please take some time to review author's responses before the deadline. If you have any questions or concerns, please feel free to raise them in the discussion thread so that the authors have an opportunity to respond.

Thank you for your time and effort in reviewing the papers.

Best regards,

Area Chair

Comments on the paper

1 comment
AuthorsOpenReview ↗

summary of revision

We thank all reviewers for their rigorous feedback. Based on the reviews, we have strengthened the mathematical statements in Proposition 5 and Corollary 2, derived the exact closed-form POT expectation with calibrated intervals, and validated our empirical findings under run-clustered bootstraps and independently trained generator populations. The revised PDF is uploaded, and the changes are summarised below with where to find them.

Proposition 5. Reviewers WkFQ and ZhrX both showed that a continuous function may exceed its limit and still decrease. The revision replaces the continuity argument with the exact slope of true yield along a surrogate tilt, ddλE[MN(qλ)]=N Covqλ(s^,GNλ(s))\frac{d}{d\lambda}\mathbb{E}[M_N(q_\lambda)]=N\ \mathrm{Cov}_{q_\lambda}(\hat s,G^\lambda_N(s)), together with the limit as λ→∞\lambda\to\infty and a sufficient condition for an interior optimum, and Appendix A.4 now proves all five parts. Both behaviours appear in the paper: the three-atom instance has an interior optimum at λ⋆=0.19\lambda^\star=0.19 with a 77% loss beyond it, and WkFQ's two-atom instance has a negative initial slope and no interior optimum. The formula agrees with finite differences to a worst relative error of 9.5×10−69.5\times10^{-6} (code/s1_exact.py, 72 checks, 0 failures; Section 4, Figure 2a, Appendix A.4).

Corollary 2 now covers the finite-sample form a benchmark reports. Any statistic of nn draws taking values in an interval of length CC moves by at most C n dTVC\ n\ d_{\mathrm{TV}}, which gives the constant explicitly for validity, uniqueness, novelty, precision/recall/coverage, density, the GuacaMol score and the biased MMD$^2$ estimate, and states the condition under which FID is covered (Section 2.1).

The sharp gap of Theorem 3 now carries its qualifications, asymptotic in NN and on a fixed finite score grid, in the theorem, the abstract, the contributions and the Figure 1 caption, with the finite-NN gap reported from the exact best-of-NN expectation.

The tail predictor is specified exactly and compared with the alternatives. Section 3 gives the closed form for E[MN]\mathbb{E}[M_N] under the fitted peaks-over-threshold model, finite exactly when ξ<1\xi< 1; an interval built from the spread of the estimate across thresholds, since threshold choice dominates the error at these sample sizes; and a comparison against the observed maximum and the plug-in expected maximum on the same draws, at budgets on both sides of the pool. Measured on seven score laws with 200 pools of 1,000 scores, resampling one threshold covers 0.80 against a nominal 0.90, the envelope across thresholds covers 0.88, and the envelope widened by 1.6 covers 0.94 with a minimum of 0.88. Beyond the pool the tail fit has the lower median error in 12 of 12 cells (Figure 2b,c; Table A1).

The population analysis now accounts for shared training runs. The 27 samplers come from four training runs. A bootstrap that resamples runs and then samplers within them leaves the two tail statistics positive with intervals excluding zero in all 20 objective-by-budget cells, and inside each run with enough samplers the sign matches the pooled sign in 60 of 60 cells. We also trained a second population of independent runs with different seeds and the same recipe: on 37 samplers from 5 independent runs the distribution-learning score keeps its sign pattern (negative on log⁡P\log P, penalised log⁡P\log P and −-SA; positive on QED and similarity) and the tail statistics stay positive on all five objectives, with run-clustered intervals excluding zero in 19 of 20 cells (Section 5.2, Figure 3).

The code, the cached per-molecule scores for every draw stream, the splits, the linear-programming inputs and the analysis scripts are in the supplementary material, so every number above can be regenerated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.