acceptodds
Under review as a conference paper at ICLR 2027

Reject-to-Protect: Dynamic Decision-Aware Adversarial Training for Selective Molecular Prediction

Abstract

Selective prediction pairs each model output with an accept–defer decision, but most pipelines attach rejection rules to predictors trained independently for accuracy. In molecular property prediction, where structurally similar compounds can have very different prediction harms, such post-hoc strategies still accept risky analogues or reject reliable ones. We introduce Reject-to-Protect (R2P), a decision-aware adversarial training framework that directly optimizes the composed prediction–acceptance policy. R2P builds a frozen, structure-only candidate graph of chemically plausible neighbors. This restricts optimization to the local neighborhoods where selective errors are most likely to occur, and the search stays finite and auditable. Within this graph, a discrete inner maximization searches for two complementary failure modes. Unsafe acceptance (UA) occurs when a harmful prediction is accepted despite its error, while over-rejection (OR) defers a reliable prediction to expert review. An outer minimization then uses these fixed endpoints to train a risk head, and optionally the predictor, under an explicit target-coverage constraint. The risk head is supervised by scaffold out-of-fold prediction harm, and retrieval, supervision, and thresholding are each confined to their own data splits to prevent leakage. On nine MoleculeNet benchmarks at 80% coverage, R2P achieves the lowest mean selective risk on five of six classification tasks and two of three regression tasks, with up to a 33.6% reduction relative to full acceptance.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Molecular property prediction, Selective prediction, Adversarial training, Drug discovery

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bdSHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reject-to-Protect (R2P), a framework for selective molecular prediction that jointly optimizes predictive performance and accept/defer decisions. R2P constructs a frozen candidate graph based on molecular structural similarity and identifies Unsafe Acceptance (UA) and Over-Rejection (OR) endpoints within local neighborhoods. These endpoints are used for decision-aware adversarial training to jointly update the molecular predictor and risk head. The framework also employs scaffold-based out-of-fold (OOF) prediction harm for risk supervision. Experiments on nine MoleculeNet datasets show that R2P achieves the lowest mean selective risk on seven datasets at 80% coverage.

Critical strengths
  1. Clear problem formulation: The paper addresses the objective mismatch between prediction-accuracy-oriented training and selective decision-making, explicitly defining two types of decision failure: UA and OR.

  2. Decision-aware adversarial training: The framework identifies challenging accept/defer decisions among structurally similar, observed molecules and jointly updates the predictor and risk head.

  3. Empirical performance: R2P achieves the lowest mean selective risk on seven of nine datasets and provides analyses across different coverage levels as well as individual molecular case studies.

Critical weaknesses
  1. Insufficient isolation of the contribution of adversarial training: R2P(Base) does not undergo additional training after initialization, whereas the UA/OR variants receive further optimization using clean prediction, risk-supervision, and coverage losses. Therefore, it is unclear whether the improvements result from the adversarial endpoint loss itself or from additional training. This distinction is particularly important because R2P(Base) already learns prediction harm through OOF supervision and a ranking loss; the incremental benefit of the UA/OR objectives requires direct validation.

  2. Insufficient evidence for the necessity of structural locality: A central component of R2P is mining UA/OR endpoints within structural neighborhoods, but the paper does not compare this strategy against global hard-example mining or random endpoint selection. It is therefore unclear whether the improvements are due to structural locality or hard-example mining in general.

  3. Limited strong baselines: The paper does not compare against existing joint selective learning methods such as SelectiveNet or Deep Gamblers. Thus, although improvements over the evaluated post-hoc methods are reported, R2P's relative advantages in joint selective optimization remain insufficiently established.

  4. Gap between the theoretical analysis and practical optimization objective: Equation (14) relates global UA/OR scores to selective risk under fixed prediction harm and coverage, whereas Equation (15) uses a local source-relative squared hinge loss. The theoretical relationship therefore does not directly guarantee that optimizing the practical objective reduces selective risk. In particular, the motivation for choosing a source-relative formulation over an absolute endpoint score, and evidence of its benefits, remain limited.

Questions for authors

Clarification Request 1: Rationale for the Relative Endpoint Loss

Equation (14) establishes a relationship between global UA/OR scores and selective risk, whereas the actual training objective in Equation (15) uses a source-relative squared hinge loss. Why did the authors choose a source-relative formulation instead of an absolute endpoint loss? Could the authors explain its advantages, particularly regarding optimization stability or improvements in selective risk?

Clarification Request 2: Interaction Between UA and OR Objectives

The ablation results show that UA+OR underperforms UA-only or OR-only on some datasets. How do the authors interpret this behavior, given that the two objectives are designed to correct different decision failures? In particular, could gradient conflicts or optimization trade-offs between the objectives explain these results?

Additional Comparison Request 1: Independent Contribution of Adversarial Training

R2P(Base) is trained using OOF harm supervision and a ranking loss, whereas the UA/OR variants undergo additional optimization using clean prediction, risk-supervision, and coverage losses. Do the authors have results for a baseline that receives the same additional training budget but excludes the UA/OR endpoint loss? Such a comparison is important for determining whether the gains come from the proposed adversarial objective rather than additional training alone.

Additional Comparison Request 2: Contribution of Structural Locality

The paper's central methodological choice is to mine UA/OR endpoints within local neighborhoods of structurally similar molecules. Do the authors have comparisons of local endpoint mining against global hard-example mining or random endpoint selection under otherwise identical training conditions? These comparisons would distinguish the benefit of structural locality from that of hard-example mining more generally.

Additional Comparison Request 3: Joint Selective Learning Baselines

The paper compares R2P with post-hoc selective prediction approaches, including confidence-based and conformal methods, but does not include methods such as SelectiveNet or Deep Gamblers that jointly optimize prediction and selection. Do the authors have results comparing these joint selective learning methods under the same backbone and evaluation conditions? Such evidence would help establish whether R2P's improvements go beyond the benefits of joint optimization itself and stem from the proposed decision-aware adversarial training.

5 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

We thank the reviewer for the detailed and constructive requests. These comments helped us clarify the independent contribution of the endpoint objectives, the respective roles of hard-example mining and structural locality, and the scope of the theoretical motivation. We added matched training controls, local/global and hard/random mining comparisons, joint selective-learning baselines, and absolute-loss variants. The new results support an incremental contribution from the proposed endpoint objectives, while also showing that additional training and hard-example mining account for part of the observed gains. We have revised our claims accordingly.

Request 1 (Additional Comparison 1) — Independent contribution of adversarial training

We agree that the original comparison did not fully isolate the effect of the endpoint objectives, because R2P(Base) was frozen while the dynamic variants received additional optimization. To address this confound, we added a matched Base_op control. Base_op retains the clean prediction loss, OOF risk supervision, and coverage penalty, while matching the optimizer updates, OOF refresh budget, checkpoint-selection procedure, and per-seed Base initialization of the UA/OR variants. Only the UA/OR endpoint losses are removed.

Table bdSH-1. Matched-budget attribution. The last column is the difference between the two means, Δ\Delta = R2P(UA+OR) − Base_op, in the metric's own units; per-seed differences carry their own spread, which is not shown in this column.

DatasetBaseBase_opR2P(UA)R2P(OR)R2P(UA+OR)Δ\Delta
BACE0.1812 ± 0.01450.1759 ± 0.01410.1707 ± 0.01370.1691 ± 0.02010.1637 ± 0.0124−0.0122
BBBP0.0893 ± 0.00270.0891 ± 0.01860.0762 ± 0.00890.0824 ± 0.00700.0887 ± 0.0194−0.0004
ClinTox0.0450 ± 0.00180.0452 ± 0.00210.0466 ± 0.00360.0453 ± 0.00370.0450 ± 0.0018−0.0002
SIDER0.1653 ± 0.00190.1647 ± 0.00280.1653 ± 0.00190.1661 ± 0.00370.1632 ± 0.0030−0.0015
Tox210.0724 ± 0.00150.0722 ± 0.00250.0714 ± 0.00060.0732 ± 0.00180.0737 ± 0.0024+0.0015
ToxCast0.0574 ± 0.00130.0566 ± 0.00410.0574 ± 0.00130.0574 ± 0.00130.0549 ± 0.0043−0.0017
ESOL0.6544 ± 0.02910.6455 ± 0.02720.6233 ± 0.04270.6245 ± 0.01380.6247 ± 0.0239−0.0208
FreeSolv1.2334 ± 0.12541.2235 ± 0.16841.2334 ± 0.12541.2170 ± 0.14231.2003 ± 0.1627−0.0232
Lipophilicity0.6095 ± 0.01320.6064 ± 0.01980.6022 ± 0.01730.6073 ± 0.01340.5992 ± 0.0211−0.0072

Base_op improves over Base on eight of the nine datasets, by 0.2–2.9%; on ClinTox, its mean risk is 0.4% higher (0.0452 against 0.0450). Thus, additional optimization accounts for a real part of the improvement over the original frozen Base. At the same training budget, the joint UA+OR model further lowers mean selective risk relative to Base_op on eight of the nine datasets, by 0.4–6.9%. Tox21 is the exception, where the joint model increases risk relative to both Base_op and Base. Some incremental differences, particularly on BBBP and ClinTox, are small relative to the seed variability.

These results therefore support an incremental and dataset-dependent contribution from the endpoint objectives, rather than attributing the entire Base-to-R2P improvement to adversarial training. They also do not imply that the joint objective uniformly dominates the individual branches: UA-only has a lower mean on BBBP, Tox21, and ESOL. We will revise the component analysis to report this attribution explicitly.

AuthorsOpenReview ↗

Rebuttal (2/3)

Request 2 (Additional Comparison 2) — Structural locality versus global hard-example mining

We agree that the original experiments did not distinguish the effect of structural locality from that of hard-example mining. We therefore added four matched-budget variants:

  • Local-hard is the default R2P configuration.
  • Local-random samples endpoints uniformly from the same local query set.
  • Global-hard-Q performs hard endpoint selection over size-matched candidate lists drawn from label-compatible Train molecules without structural constraints.
  • Global-random-Q samples endpoints uniformly from the corresponding global candidate lists.

All variants share the same Base initialization, clean-batch order, loss formulation, update count, OOF refresh budget, and query budget.

Table bdSH-2. Endpoint mining: local versus global, hard versus random, at equal query budget.

DatasetLocal-hardLocal-randomGlobal-hard-QGlobal-random-Q
BACE0.1637 ± 0.01240.1684 ± 0.01130.1665 ± 0.00740.1678 ± 0.0153
BBBP0.0887 ± 0.01940.0906 ± 0.00860.0898 ± 0.01400.0909 ± 0.0102
ClinTox0.0450 ± 0.00180.0464 ± 0.00660.0438 ± 0.00920.0457 ± 0.0056
SIDER0.1632 ± 0.00300.1640 ± 0.00060.1638 ± 0.00090.1642 ± 0.0003
Tox210.0737 ± 0.00240.0746 ± 0.00450.0741 ± 0.00110.0748 ± 0.0021
ToxCast0.0549 ± 0.00430.0558 ± 0.00090.0553 ± 0.00080.0559 ± 0.0006
ESOL0.6247 ± 0.02390.6320 ± 0.01220.6192 ± 0.03190.6284 ± 0.0096
FreeSolv1.2003 ± 0.16271.2145 ± 0.21601.1937 ± 0.16891.2062 ± 0.1820
Lipophilicity0.5992 ± 0.02110.6044 ± 0.00430.5999 ± 0.01130.6008 ± 0.0069

The results support two distinct conclusions.

First, hard-example mining is the most consistent source of improvement. Both local-hard and global-hard-Q achieve lower mean risk than both random controls on all nine datasets. Thus, the benefit of endpoint mining is not explained by merely introducing additional training examples.

Second, structural locality provides an additional but dataset-dependent benefit. Local-hard achieves lower mean risk than global-hard-Q on six of the nine datasets and is the lowest-mean arm among the four variants on those six datasets. Global-hard-Q performs better on ClinTox, ESOL, and FreeSolv, by 0.5–2.7%.

We therefore do not interpret these results as showing that structural locality is uniformly necessary. Rather, local retrieval improves over unconstrained global hard mining on most, but not all, evaluated benchmarks. The local graph additionally provides a finite, structure-based, and auditable search space: retrieval and ranking use no label values, predictions, harms, or risk scores, while observed-label availability is used only to determine endpoint eligibility.

We will revise the manuscript to distinguish clearly between the general benefit of hard-example mining and the incremental, dataset-dependent benefit of structural locality.

Request 3 (Additional Comparison 3) — Joint selective learning baselines

We agree that comparisons with joint selective-learning methods were missing. We therefore added SelectiveNet and Deep Gamblers as joint selective-learning baselines. Both methods use the same CMPNN backbone, data splits, seeds, and fixed-acceptance evaluation protocol as R2P, while retaining their original selection mechanisms. Deep Gamblers is evaluated on the six classification benchmarks.

Table bdSH-3. SelectiveNet and Deep Gamblers under the same backbone and evaluation protocol.

DatasetR2PSelectiveNetDeep Gamblers
BACE0.1637 ± 0.01240.1689 ± 0.01720.1956 ± 0.0205
BBBP0.0887 ± 0.01940.0914 ± 0.00360.1084 ± 0.0177
ClinTox0.0450 ± 0.00180.0478 ± 0.01070.0512 ± 0.0098
SIDER0.1632 ± 0.00300.1643 ± 0.00060.1655 ± 0.0008
Tox210.0737 ± 0.00240.0731 ± 0.00310.0787 ± 0.0017
ToxCast0.0549 ± 0.00430.0566 ± 0.00090.0592 ± 0.0016
ESOL0.6247 ± 0.02390.7135 ± 0.0207n/a
FreeSolv1.2003 ± 0.16271.5445 ± 0.0827n/a
Lipophilicity0.5992 ± 0.02110.6140 ± 0.0520n/a

R2P achieves lower mean selective risk than SelectiveNet on eight of the nine datasets and lower mean risk than Deep Gamblers on all six classification benchmarks. The exception is Tox21, where SelectiveNet obtains 0.0731 compared with 0.0737 for R2P. We explicitly report this as a dataset-specific advantage for SelectiveNet rather than claiming uniform superiority for R2P.

These comparisons strengthen the empirical evidence that the benefits of R2P are not explained solely by jointly optimizing prediction and selection. Nevertheless, we limit our conclusion to the evaluated methods and benchmarks rather than making a broader state-of-the-art claim.

AuthorsOpenReview ↗

Rebuttal (3/3)

Request 4 (Clarification 1) — Rationale for the relative endpoint loss

For classification, the per-molecule summed UA/OR score is

aihi+(1−ai)(1−hi),a_i h_i+(1-a_i)(1-h_i),

and Eq. (14) states that its global sum equals

2m SelRisk+∑i(1−hi)−m.2m\,\mathrm{SelRisk}+\sum_i(1-h_i)-m.

Consequently, under fixed prediction harms, fixed acceptance mass mm, and a global sum over molecules, the summed score and selective risk have the same minimizers. This relation establishes the intended direction of the two decision scores.

We agree that it does not provide a direct optimization guarantee for Equation 15. The practical objective differs through local endpoint supervision, a finite candidate budget, the source-relative hinge, a changing predictor, soft coverage regularization, and the use of detached OOF harm in the OR outer score. The regression transformations introduce a further difference from the classification relation. We will state this scope limitation alongside Equation 14 in the main text rather than presenting the relation as a guarantee for the practical algorithm.

The source-relative loss compares each selected endpoint with a detached score of its own source. It penalizes only the amount by which the endpoint score exceeds that reference, whereas the absolute variants penalize every nonzero endpoint score. The formulations therefore differ in both their activation sets and the weighting of active updates.

The source score provides an adaptive reference for each source molecule. Detaching this reference prevents the model from reducing the penalty by manipulating the reference itself, while the squared hinge weights active updates according to the magnitude of the violation.

Table bdSH-4. Outer endpoint loss form, full training.

DatasetRelativeAbsolute-squareAbsolute-linear
BACE0.1637 ± 0.01240.1652 ± 0.01490.1668 ± 0.0099
BBBP0.0887 ± 0.01940.0896 ± 0.01600.0908 ± 0.0155
Tox210.0737 ± 0.00240.0739 ± 0.00080.0750 ± 0.0026
ESOL0.6247 ± 0.02390.6288 ± 0.02590.6485 ± 0.0144

The relative formulation attains the lowest mean selective risk on all four tested datasets. Its improvement over absolute-square ranges from 0.0002 to 0.0041 in the respective metric units, and its largest improvement over absolute-linear is 0.0238 on ESOL. These results provide empirical support for the relative formulation on the tested benchmarks, but do not establish universal superiority.

We also compared the gradients at a fixed Base initialization. The relative hinge activates on 67.8–90.6% of sources. Its gradient direction is close to that of absolute-square, with cosine similarities of 0.976–1.000, while the relative-to-absolute-square gradient norm ratios range from 0.376 to 0.870. Thus, at this initialization, the two objectives differ primarily in their activation range and update weighting.

These fixed-Base diagnostics characterize the optimization difference but do not identify which specific mechanism causes the final performance differences. We therefore present the source-relative form as an empirically supported design choice on the tested benchmarks, not as a theoretically guaranteed source of optimization stability.

Request 5 (Clarification 2) — Interaction between UA and OR objectives

We agree that the combined objective should not be expected to dominate either individual objective on every dataset. Although UA and OR target complementary decision failures, they interact through the shared predictor, risk head, coverage constraint, and dynamically reselected endpoints. Their relative benefits may therefore depend on dataset-specific prediction errors, acceptance patterns, and OOF supervision quality.

The matched results illustrate this behavior: UA+OR improves over Base_op on eight datasets, but UA-only obtains a lower mean than UA+OR on BBBP, Tox21, and ESOL. We interpret this as dataset-dependent complementarity, rather than a contradiction between the two objectives.

Gradient conflict and optimization trade-offs are possible explanations, but our current experiments do not isolate either as the specific cause. Equal-norm PCGrad can mitigate destructive gradient interference, but it does not guarantee that combining UA and OR will produce a better validation or test operating point than either branch individually.

We will therefore avoid attributing the observed behavior to gradient conflict without direct evidence. We will also revise the manuscript to state explicitly that the combined UA+OR objective is not uniformly optimal and that the benefit of each branch varies across datasets.

Reviewer_4vnLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes Reject-to-Protect (R2P), a decision-aware adversarial training framework for selective molecular property prediction. R2P constructs a frozen, structure-only candidate graph and mines two complementary local failure modes: unsafe acceptance, where harmful predictions are likely to be accepted, and over-rejection, where reliable predictions are likely to be deferred. The mined endpoints guide updates to a prediction-aware risk head and, optionally, the predictor under a target-coverage constraint. The framework separates retrieval, out-of-fold harm supervision, and threshold calibration to reduce leakage. Across nine MoleculeNet benchmarks at 80% coverage, R2P achieves the lowest mean selective risk among the evaluated methods on five of six classification tasks and two of three regression tasks. Ablations and risk–coverage curves provide further evidence of the complementary effects of the two mining objectives.

Critical strengths
  1. Decision-aware formulation. R2P jointly targets unsafe acceptance and over-rejection through local adversarial mining, addressing a practically important problem in selective molecular prediction.

  2. Careful design and encouraging results. Structure-only candidate graphs and separated supervision/calibration reduce leakage concerns. Experiments across nine datasets, supported by ablations and risk–coverage curves, demonstrate benefits over the included baselines.

Critical weaknesses
  1. Incomplete empirical comparisons. Missing strong ensemble-based and learned-selector baselines limit claims of competitiveness. Fixed-count test acceptance also leaves performance under calibration-derived deployment thresholds unclear.

  2. Insufficient robustness and cost analysis. The regression objective remains heuristic, while sensitivity to OOF harm quality and candidate-graph settings is not established. Computational overhead is also unreported, limiting assessment of practicality.

Questions for authors
  1. Does R2P outperform strong ensemble-based selectors and learned loss-prediction heads under the same evaluation protocol?
  2. Do the main conclusions hold when using calibration-derived thresholds instead of fixed-count test acceptance?
  3. How sensitive are the regression results to the chosen log/exp transformations?
  4. How sensitive is R2P to OOF harm quality, neighborhood size, and query budget, and what is its training overhead?
5 comments
AuthorsOpenReview ↗

Rebuttal (1/4)

We thank the reviewer for the careful reading of the empirical comparisons and of the robustness and cost analysis.

Q1 — Strong ensemble and learned-selector baselines

We added two ensemble baselines with 2 and 5 independently trained CMPNNs (20 epochs each), a learned risk head on a frozen Base predictor, and a single CMPNN trained for the same extended budget of 20+100 epochs (Table 4vnL-1). All methods use the same backbone, data splits, seeds, Validation-based checkpoint selection, and 80% fixed-count evaluation.

R2P achieves lower mean selective risk than Ensemble-5, the learned head, and the long-trained single model on eight of nine datasets, and than Ensemble-2 on seven of nine. The main exceptions are Tox21, where the ensemble baselines perform slightly better, and ToxCast, where Ensemble-2 gives 0.0541 compared with 0.0549 for R2P. Except for ToxCast, the stronger ensemble consistently improves over the 2-member ensemble, which in turn generally outperforms the learned head and the long-trained single model. Despite this, R2P remains better on most datasets while using only a single predictor.

The additional controls also separate the effect of training budget from that of the endpoint objectives. Extending the same backbone from 20 to 120 epochs improves over Base on seven of nine datasets by 0.1–3.9%. The matched Base_op control retains the clean prediction loss, OOF risk supervision and coverage penalty, and matches the optimizer updates, OOF refresh budget, checkpoint selection and per-seed initialization of the UA/OR variants, removing only the endpoint terms; it improves on eight datasets by 0.2–2.9% (Table bdSH-1). Adding the UA/OR endpoint losses provides a further 0.4–6.9% reduction on eight datasets. These results indicate that the gains are not explained by ensembling, a learned risk head, or additional optimization alone. The training overhead of R2P is reported separately in Table 4vnL-6.

Table 4vnL-1. Selective risk at 80% acceptance. Ensemble-2/5: 2 or 5 independent CMPNNs, 20 epochs each; learned head: frozen Base predictor, risk head further optimised; single-budget: one model, 20 + 100 epochs.

DatasetR2PEnsemble-2Ensemble-5Learned headSingle-budget
BACE0.1637 ± 0.01240.1681 ± 0.01130.1668 ± 0.01010.1710 ± 0.00880.1742 ± 0.0151
BBBP0.0887 ± 0.01940.0890 ± 0.00920.0889 ± 0.00840.0891 ± 0.01710.0892 ± 0.0179
ClinTox0.0450 ± 0.00180.0469 ± 0.00490.0462 ± 0.00560.0477 ± 0.00330.0486 ± 0.0023
SIDER0.1632 ± 0.00300.1643 ± 0.00270.1638 ± 0.00220.1648 ± 0.00350.1651 ± 0.0026
Tox210.0737 ± 0.00240.0726 ± 0.00290.0719 ± 0.00170.0731 ± 0.00130.0735 ± 0.0022
ToxCast0.0549 ± 0.00430.0541 ± 0.00090.0557 ± 0.00060.0566 ± 0.00110.0572 ± 0.0038
ESOL0.6247 ± 0.02390.6372 ± 0.02690.6338 ± 0.05480.6401 ± 0.07250.6499 ± 0.0301
FreeSolv1.2003 ± 0.16271.2193 ± 0.17901.2146 ± 0.20911.2247 ± 0.09491.2284 ± 0.1418
Lipophilicity0.5992 ± 0.02110.6044 ± 0.03590.6029 ± 0.01950.6062 ± 0.03950.6080 ± 0.0169
AuthorsOpenReview ↗

Rebuttal (2/4)

Q2 — Calibration-derived deployment thresholds

To evaluate deployment-style thresholding, we fix each method's acceptance threshold at the 0.8 quantile of its own score on Calibration and apply it unchanged to Test, without re-ranking or truncation. The score is the predicted risk for R2P, the calibrated prediction-set size for split conformal in classification and the CQR interval width in regression, and the distance to the Train representation centroid for embedding distance.

Table 4vnL-2. Selective risk at fixed 80% acceptance and under a frozen Calibration threshold, with the realized coverage of each threshold rule. Values are mean ± sd over three seeds; coverage is the mean over the three seeds (seed-to-seed sd at most 0.08).

DatasetR2P, 80%R2P, thresholdConformal, thresholdEmbedding, threshold
BACE0.1637 ± 0.01240.1620 ± 0.0113 (0.78)0.1727 ± 0.0166 (0.94)0.1865 ± 0.0054 (0.80)
BBBP0.0887 ± 0.01940.0850 ± 0.0112 (0.77)0.1111 ± 0.0150 (0.96)0.1400 ± 0.0324 (0.81)
ClinTox0.0450 ± 0.00180.0250 ± 0.0071 (0.69)0.0678 ± 0.0039 (1.00)0.0486 ± 0.0017 (0.77)
SIDER0.1632 ± 0.00300.1631 ± 0.0009 (0.77)0.1651 ± 0.0021 (0.90)0.1683 ± 0.0037 (0.84)
Tox210.0737 ± 0.00240.0733 ± 0.0022 (0.79)0.0901 ± 0.0019 (1.00)0.0849 ± 0.0160 (0.81)
ToxCast0.0549 ± 0.00430.0557 ± 0.0014 (0.81)0.0730 ± 0.0009 (0.77)0.0631 ± 0.0078 (0.78)
ESOL0.6247 ± 0.02390.6358 ± 0.0226 (0.91)0.6512 ± 0.0296 (0.79)0.6267 ± 0.0095 (0.83)
FreeSolv1.2003 ± 0.16271.2178 ± 0.1584 (0.92)0.9226 ± 0.1432 (0.66)1.3055 ± 0.1560 (0.71)
Lipophilicity0.5992 ± 0.02110.5931 ± 0.0073 (0.79)0.5945 ± 0.0222 (0.73)0.6036 ± 0.0218 (0.77)

R2P keeps the lower mean risk on eight of the nine datasets against each baseline, and on seven of the nine against both at once. The exceptions are FreeSolv against conformal and ESOL against embedding distance, and in both the gap comes with a difference in coverage: on FreeSolv, conformal accepts 65.6% of Test and reaches 0.9226, while R2P accepts 91.8% and reaches 1.2178; on ESOL, embedding distance accepts 83.2% and reaches 0.6267, while R2P accepts 91.3% and reaches 0.6358.

R2P's own operating point moves with coverage as well. It is at or below the fixed-count value on six datasets, and higher on the three where the threshold accepts more than the nominal 80%: ToxCast (81.4%), ESOL (91.3%) and FreeSolv (91.8%). On ClinTox it accepts 69.4% and the risk falls from 0.0450 to 0.0250. A calibrated threshold fixes a level of the score rather than a fraction of the data, so the accepted set grows or shrinks with the Test distribution, and a larger set brings in the molecules that the ranking places above the cut. On ClinTox and Tox21 the conformal threshold admits the whole Test set, because its set size is discrete and the Calibrated quantile covers every size present; those two entries are full-coverage risks.

The fixed-count protocol remains the like-for-like comparison because it holds coverage constant: for conformal and embedding distance those results are in Table 1 of the submission, and for the learned baselines in Tables 4vnL-1 and bdSH-3. Table 4vnL-2 instead reports the operating point a practitioner obtains when the acceptance rate is set by a Calibration threshold. Realized coverage differs across methods: 0.66–1.00 for conformal, 0.71–0.84 for embedding distance and 0.69–0.92 for R2P, which is exactly the quantity this table exposes.

AuthorsOpenReview ↗

Rebuttal (3/4)

Q3 — Sensitivity to the log/exp transformations

To assess sensitivity to the regression transformations, we ablate the log⁡(1+e)\log(1+e) term used for UA and the exp⁡(−e)\exp(-e) term used for OR. These transformations are specific to regression: in Eqs. (11)–(12), classification uses the bounded Brier harm directly, ajeja_j e_j for UA and (1−aj)(1−ej)(1-a_j)(1-e_j) for OR, and the outer scores in Eq. (24) follow the same form because ej∈[0,1]e_j\in[0,1]. We therefore conduct this ablation on the three regression benchmarks. Each variant changes only one transformation and is retrained from the same initialization with matched minibatch order, while keeping the other transformation, Base initialization, candidate graph, update count, and checkpoint-selection rule fixed.

The variants follow a naming convention. ua- rows change the UA response applied to the normalized error e≥0e ≥ 0 and or- rows change the OR response, with the other branch, the Base, the graph, the update count and the selection rule held fixed. Three rows replace the functional form: ua-linear uses a⋅ea·e instead of a⋅log(1+e)a·log(1+e); ua-bounded uses the saturating map a⋅e/(1+e)a·e/(1+e), which lies in [0,1) and has a decreasing derivative; and or-reciprocal uses (1−a)/(1+e)(1−a)/(1+e) instead of (1−a)⋅exp(−e)(1−a)·exp(−e). The four -scale-β rows instead rescale the normalized error before the transform, giving log(1+βe)log(1+βe) for UA and exp(−βe)exp(−βe) for OR with ββ = 0.5 or 2. Scaling therefore changes how errors affect molecular scores, which in turn determines the selected endpoint and the set of active hinge constraints, rather than simply multiplying an endpoint score or loss by ββ.

Table 4vnL-3. Transform ablation on the three regression benchmarks, selective RMSE at 80% (mean ± sd over three seeds). Bold marks the lowest mean in each dataset column.

VariantESOLFreeSolvLipophilicity
default0.6247 ± 0.02391.2003 ± 0.16270.5992 ± 0.0211
ua-linear0.6106 ± 0.03291.1970 ± 0.14780.6014 ± 0.0039
ua-bounded0.6204 ± 0.02731.2138 ± 0.15500.5982 ± 0.0165
or-reciprocal0.6139 ± 0.03011.0989 ± 0.11130.5926 ± 0.0040
ua-scale-0.50.6273 ± 0.01391.1804 ± 0.16070.5836 ± 0.0163
ua-scale-20.6272 ± 0.02281.2598 ± 0.12520.5933 ± 0.0173
or-scale-0.50.6188 ± 0.02981.0979 ± 0.13850.6012 ± 0.0127
or-scale-20.6125 ± 0.02771.2177 ± 0.12930.5808 ± 0.0079

On ESOL the seven variants span −2.3% to +0.4% of the default and on Lipophilicity −3.1% to +0.4%, so the reported means on these two tasks are not sensitive to the transform choice at this scale. FreeSolv is the informative case: there the variants span −8.5% to +5.0%, and the two lowest means both come from the OR branch — or-scale-0.5 (1.0979) and or-reciprocal (1.0989) — whereas the UA variants stay at 1.1804–1.2598 and reach +5.0% at ββ = 2.

log(1+e)/exp(−e)log(1+e)/exp(−e) is therefore a reasonable default for two of the three regression benchmarks, with a measurable response on FreeSolv that comes from both factors and is larger for the OR shape; Table 4vnL-3 reports the per-variant means, including the two OR alternatives as optional settings.

Reviewer_wyJuOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors tackle the problem of accept-defer decisions on molecular property predictions, where properties are predicted using trained models, but at inference time a decision is made to trust the model output (accept) or defer to expert review. Reject-to-protect (RTP) proposes a multi-objective training approach that trains models to perform predictions but also improve coverage while lowering unsafe acceptances and over-rejection. The method relies on retrieving candidate graphs from existing data to calibrate the risk and acceptance mechanism.

Critical strengths
  • The authors tackle an important problem of selective molecular prediction where fast in-sillico model-based filtering is cheap but first-principles calculations is computationally costly and experimental evaluation is prohibitively expensive. They present a sound approach in exploring overly confident predictions and too cautious rejections. The joint training objective attempts to calibrate risk and optimize coverage

  • The paper is easy to read and follow and provides a strong motivation for the method and a sound step-by-step description of the authors proposed method. While the method relies on some significant assumptions, the technique is sound.

Critical weaknesses
  • The authors use EXFP4 Tanimoto similarity to rank candidate molecules. In some domains, such a fingerprint tend to not be very representative of molecular similarity. I would urge the authors to consider or at least discuss the possibilities of using other similarity measures.
  • In general, the notion that the errors most likely to occur in local neighborhoods of the molecules is a very strong prior. Ideally, we would have some empirical claims to back up such a conjecture. One can imagine significant errors in other circumstances as well. Maybe showing correlation (or lack there of) between similarity and labels and risk could help.
Questions for authors
  • What As\mathcal{A}_s in line 198?
3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for the positive assessment and constructive feedback.

Weakness 1 — ECFP4 Tanimoto similarity

To test whether R2P depends on ECFP4-based retrieval, we rebuilt and re-froze the candidate graph using ECFP6 (radius 3) and the RDKit path fingerprint, both with 2048 bits and Tanimoto similarity, while keeping the initialization, losses, training budget, and selection rule unchanged.

As shown in Table wyJu-1, the alternative fingerprints produce substantially different candidate sets, with mean Jaccard overlap with the ECFP4 graph ranging from 0.275 to 0.841. Despite these changes, mean selective risk varies by only −3.6%-3.6\% to +1.4%+1.4\% across the tested datasets. These results suggest that R2P is not strongly dependent on the specific fingerprint used for candidate retrieval. We have added this analysis and clarified that other structure-based similarity measures, such as 3D pharmacophore or shape similarity and learned molecular representations, could also be considered.

Table wyJu-1. Selective risk at 80% acceptance under three retrieval representations.

DatasetECFP4 (default)ECFP6RDKit pathQuery Jaccard (ECFP6 / path)Δ% (ECFP6 / path)
BACE0.1637 ± 0.01240.1578 ± 0.01650.1651 ± 0.00710.841 / 0.551−3.6% / +0.9%
Tox210.0737 ± 0.00240.0746 ± 0.00190.0739 ± 0.00250.673 / 0.275+1.2% / +0.3%
ESOL0.6247 ± 0.02390.6177 ± 0.01610.6216 ± 0.01790.680 / 0.331−1.1% / −0.5%
FreeSolv1.2003 ± 0.16271.2134 ± 0.15271.2175 ± 0.16640.684 / 0.319+1.1% / +1.4%
AuthorsOpenReview ↗

Rebuttal (2/2)

Weakness 2 — Is a local neighbourhood a justified search space?

To characterize the role of structural similarity in the local candidate space, we added a Train-set analysis of its relationship with label differences and prediction risk (Table wyJu-2). For each graph edge, we compute the mean absolute label difference over shared tasks and compare it with a label-compatible random pair drawn for the same source, with one random draw per edge and resampling across seeds. We also compute Spearman correlations between structural similarity and the candidate's OOF harm, and between structural similarity and the predicted-risk gap within each local neighbourhood.

All quantities in Table wyJu-2 are computed on Train from held-out predictions of the three-fold shadow predictors and from the risk scores of the shared Base model. Local pairs are fixed by the frozen graph, so the local label difference is a single value; random pairs are resampled per seed, and the random-pair columns together with both correlation columns are therefore reported as mean ± standard deviation over the three seeds. The risk gap in the last column is the absolute difference between the risk-head outputs of the source and of the candidate, ∣r^i−r^j∣|\hat r_i-\hat r_j|, taken over the graph edges of the dataset, while the candidate's own OOF harm is the error of its shadow prediction; the two ρ\rho columns therefore test different questions, namely whether the risk score separates structural neighbours, and whether similarity itself predicts the error.

Table wyJu-2. Structural similarity, label difference and predictive risk on Train.

DatasetLabel diff., local pairsLabel diff., random pairsρ (similarity, candidate OOF harm)ρ (similarity, risk gap)
BACE0.2550.499 ± 0.0040.061 ± 0.029−0.289 ± 0.010
BBBP0.1320.353 ± 0.0040.057 ± 0.049−0.178 ± 0.016
ClinTox0.0650.094 ± 0.0040.094 ± 0.015−0.209 ± 0.020
SIDER0.2720.348 ± 0.0050.052 ± 0.016−0.371 ± 0.024
Tox210.0840.125 ± 0.0010.029 ± 0.006−0.188 ± 0.005
ToxCast0.0690.090 ± 0.0010.113 ± 0.002−0.130 ± 0.015
ESOL1.3522.339 ± 0.0130.063 ± 0.008−0.068 ± 0.058
FreeSolv2.2664.116 ± 0.0620.080 ± 0.002−0.149 ± 0.012
Lipophilicity0.9351.318 ± 0.008−0.010 ± 0.010−0.381 ± 0.020

Across all nine benchmarks, local pairs have smaller label differences than random pairs, by approximately 1.3-2.7×. Within local neighbourhoods, higher structural similarity is consistently associated with a smaller predicted-risk gap (ρ=−0.068\rho=-0.068 to −0.381-0.381), whereas its correlation with the candidate's own OOF harm is positive in eight of the nine datasets (ρ=0.029\rho=0.029 to 0.1130.113). Within the neighbourhood, the risk score is therefore smoothest exactly where the error is not. These results support the intended separation of roles in R2P: structural similarity determines where to search, while the UA/OR scores determine which candidates constitute decision failures. We therefore revised the manuscript to avoid implying that prediction errors are necessarily concentrated in local neighbourhoods. Consistently, Table bdSH-2 shows that local endpoint mining yields lower selective risk than both random variants on all nine datasets and lower risk than global-hard-Q mining on six.

Question — Notation in line 198

We apologize for the unclear notation. Line 198 falls in the definition of the frozen candidate graph, Eq. (7):

Gs=(Vs,As)G_s = (V_s, A_s), Cs(i)={j:(i,j)∈As}C_s(i) = \lbrace j : (i,j) \in A_s\rbrace.

In Eq. (7), VsV_s is the set of measured molecules in split ss and AsA_s is the candidate-edge set linking each source to its eligible retrieved molecules, Cs(i)C_s(i) is then the candidate set of source ii. We have clarified this notation in the revised manuscript by adding the following definition:

"where VsV_s denotes the set of measured molecules in split ss, and AsA_s contains the candidate edges connecting each source molecule to its eligible retrieved molecules."

I thank the authors for their clarifications and response, and keep my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.