- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Reject-to-Protect (R2P), a framework for selective molecular prediction that jointly optimizes predictive performance and accept/defer decisions. R2P constructs a frozen candidate graph based on molecular structural similarity and identifies Unsafe Acceptance (UA) and Over-Rejection (OR) endpoints within local neighborhoods. These endpoints are used for decision-aware adversarial training to jointly update the molecular predictor and risk head. The framework also employs scaffold-based out-of-fold (OOF) prediction harm for risk supervision. Experiments on nine MoleculeNet datasets show that R2P achieves the lowest mean selective risk on seven datasets at 80% coverage.
Critical strengths
-
Clear problem formulation: The paper addresses the objective mismatch between prediction-accuracy-oriented training and selective decision-making, explicitly defining two types of decision failure: UA and OR.
-
Decision-aware adversarial training: The framework identifies challenging accept/defer decisions among structurally similar, observed molecules and jointly updates the predictor and risk head.
-
Empirical performance: R2P achieves the lowest mean selective risk on seven of nine datasets and provides analyses across different coverage levels as well as individual molecular case studies.
Critical weaknesses
-
Insufficient isolation of the contribution of adversarial training: R2P(Base) does not undergo additional training after initialization, whereas the UA/OR variants receive further optimization using clean prediction, risk-supervision, and coverage losses. Therefore, it is unclear whether the improvements result from the adversarial endpoint loss itself or from additional training. This distinction is particularly important because R2P(Base) already learns prediction harm through OOF supervision and a ranking loss; the incremental benefit of the UA/OR objectives requires direct validation.
-
Insufficient evidence for the necessity of structural locality: A central component of R2P is mining UA/OR endpoints within structural neighborhoods, but the paper does not compare this strategy against global hard-example mining or random endpoint selection. It is therefore unclear whether the improvements are due to structural locality or hard-example mining in general.
-
Limited strong baselines: The paper does not compare against existing joint selective learning methods such as SelectiveNet or Deep Gamblers. Thus, although improvements over the evaluated post-hoc methods are reported, R2P's relative advantages in joint selective optimization remain insufficiently established.
-
Gap between the theoretical analysis and practical optimization objective: Equation (14) relates global UA/OR scores to selective risk under fixed prediction harm and coverage, whereas Equation (15) uses a local source-relative squared hinge loss. The theoretical relationship therefore does not directly guarantee that optimizing the practical objective reduces selective risk. In particular, the motivation for choosing a source-relative formulation over an absolute endpoint score, and evidence of its benefits, remain limited.
Questions for authors
Clarification Request 1: Rationale for the Relative Endpoint Loss
Equation (14) establishes a relationship between global UA/OR scores and selective risk, whereas the actual training objective in Equation (15) uses a source-relative squared hinge loss. Why did the authors choose a source-relative formulation instead of an absolute endpoint loss? Could the authors explain its advantages, particularly regarding optimization stability or improvements in selective risk?
Clarification Request 2: Interaction Between UA and OR Objectives
The ablation results show that UA+OR underperforms UA-only or OR-only on some datasets. How do the authors interpret this behavior, given that the two objectives are designed to correct different decision failures? In particular, could gradient conflicts or optimization trade-offs between the objectives explain these results?
Additional Comparison Request 1: Independent Contribution of Adversarial Training
R2P(Base) is trained using OOF harm supervision and a ranking loss, whereas the UA/OR variants undergo additional optimization using clean prediction, risk-supervision, and coverage losses. Do the authors have results for a baseline that receives the same additional training budget but excludes the UA/OR endpoint loss? Such a comparison is important for determining whether the gains come from the proposed adversarial objective rather than additional training alone.
Additional Comparison Request 2: Contribution of Structural Locality
The paper's central methodological choice is to mine UA/OR endpoints within local neighborhoods of structurally similar molecules. Do the authors have comparisons of local endpoint mining against global hard-example mining or random endpoint selection under otherwise identical training conditions? These comparisons would distinguish the benefit of structural locality from that of hard-example mining more generally.
Additional Comparison Request 3: Joint Selective Learning Baselines
The paper compares R2P with post-hoc selective prediction approaches, including confidence-based and conformal methods, but does not include methods such as SelectiveNet or Deep Gamblers that jointly optimize prediction and selection. Do the authors have results comparing these joint selective learning methods under the same backbone and evaluation conditions? Such evidence would help establish whether R2P's improvements go beyond the benefits of joint optimization itself and stem from the proposed decision-aware adversarial training.
5 comments
Rebuttal (2/3)
Request 2 (Additional Comparison 2) — Structural locality versus global hard-example mining
We agree that the original experiments did not distinguish the effect of structural locality from that of hard-example mining. We therefore added four matched-budget variants:
- Local-hard is the default R2P configuration.
- Local-random samples endpoints uniformly from the same local query set.
- Global-hard-Q performs hard endpoint selection over size-matched candidate lists drawn from label-compatible Train molecules without structural constraints.
- Global-random-Q samples endpoints uniformly from the corresponding global candidate lists.
All variants share the same Base initialization, clean-batch order, loss formulation, update count, OOF refresh budget, and query budget.
Table bdSH-2. Endpoint mining: local versus global, hard versus random, at equal query budget.
| Dataset | Local-hard | Local-random | Global-hard-Q | Global-random-Q |
|---|---|---|---|---|
| BACE | 0.1637 ± 0.0124 | 0.1684 ± 0.0113 | 0.1665 ± 0.0074 | 0.1678 ± 0.0153 |
| BBBP | 0.0887 ± 0.0194 | 0.0906 ± 0.0086 | 0.0898 ± 0.0140 | 0.0909 ± 0.0102 |
| ClinTox | 0.0450 ± 0.0018 | 0.0464 ± 0.0066 | 0.0438 ± 0.0092 | 0.0457 ± 0.0056 |
| SIDER | 0.1632 ± 0.0030 | 0.1640 ± 0.0006 | 0.1638 ± 0.0009 | 0.1642 ± 0.0003 |
| Tox21 | 0.0737 ± 0.0024 | 0.0746 ± 0.0045 | 0.0741 ± 0.0011 | 0.0748 ± 0.0021 |
| ToxCast | 0.0549 ± 0.0043 | 0.0558 ± 0.0009 | 0.0553 ± 0.0008 | 0.0559 ± 0.0006 |
| ESOL | 0.6247 ± 0.0239 | 0.6320 ± 0.0122 | 0.6192 ± 0.0319 | 0.6284 ± 0.0096 |
| FreeSolv | 1.2003 ± 0.1627 | 1.2145 ± 0.2160 | 1.1937 ± 0.1689 | 1.2062 ± 0.1820 |
| Lipophilicity | 0.5992 ± 0.0211 | 0.6044 ± 0.0043 | 0.5999 ± 0.0113 | 0.6008 ± 0.0069 |
The results support two distinct conclusions.
First, hard-example mining is the most consistent source of improvement. Both local-hard and global-hard-Q achieve lower mean risk than both random controls on all nine datasets. Thus, the benefit of endpoint mining is not explained by merely introducing additional training examples.
Second, structural locality provides an additional but dataset-dependent benefit. Local-hard achieves lower mean risk than global-hard-Q on six of the nine datasets and is the lowest-mean arm among the four variants on those six datasets. Global-hard-Q performs better on ClinTox, ESOL, and FreeSolv, by 0.5–2.7%.
We therefore do not interpret these results as showing that structural locality is uniformly necessary. Rather, local retrieval improves over unconstrained global hard mining on most, but not all, evaluated benchmarks. The local graph additionally provides a finite, structure-based, and auditable search space: retrieval and ranking use no label values, predictions, harms, or risk scores, while observed-label availability is used only to determine endpoint eligibility.
We will revise the manuscript to distinguish clearly between the general benefit of hard-example mining and the incremental, dataset-dependent benefit of structural locality.
Request 3 (Additional Comparison 3) — Joint selective learning baselines
We agree that comparisons with joint selective-learning methods were missing. We therefore added SelectiveNet and Deep Gamblers as joint selective-learning baselines. Both methods use the same CMPNN backbone, data splits, seeds, and fixed-acceptance evaluation protocol as R2P, while retaining their original selection mechanisms. Deep Gamblers is evaluated on the six classification benchmarks.
Table bdSH-3. SelectiveNet and Deep Gamblers under the same backbone and evaluation protocol.
| Dataset | R2P | SelectiveNet | Deep Gamblers |
|---|---|---|---|
| BACE | 0.1637 ± 0.0124 | 0.1689 ± 0.0172 | 0.1956 ± 0.0205 |
| BBBP | 0.0887 ± 0.0194 | 0.0914 ± 0.0036 | 0.1084 ± 0.0177 |
| ClinTox | 0.0450 ± 0.0018 | 0.0478 ± 0.0107 | 0.0512 ± 0.0098 |
| SIDER | 0.1632 ± 0.0030 | 0.1643 ± 0.0006 | 0.1655 ± 0.0008 |
| Tox21 | 0.0737 ± 0.0024 | 0.0731 ± 0.0031 | 0.0787 ± 0.0017 |
| ToxCast | 0.0549 ± 0.0043 | 0.0566 ± 0.0009 | 0.0592 ± 0.0016 |
| ESOL | 0.6247 ± 0.0239 | 0.7135 ± 0.0207 | n/a |
| FreeSolv | 1.2003 ± 0.1627 | 1.5445 ± 0.0827 | n/a |
| Lipophilicity | 0.5992 ± 0.0211 | 0.6140 ± 0.0520 | n/a |
R2P achieves lower mean selective risk than SelectiveNet on eight of the nine datasets and lower mean risk than Deep Gamblers on all six classification benchmarks. The exception is Tox21, where SelectiveNet obtains 0.0731 compared with 0.0737 for R2P. We explicitly report this as a dataset-specific advantage for SelectiveNet rather than claiming uniform superiority for R2P.
These comparisons strengthen the empirical evidence that the benefits of R2P are not explained solely by jointly optimizing prediction and selection. Nevertheless, we limit our conclusion to the evaluated methods and benchmarks rather than making a broader state-of-the-art claim.
Rebuttal (3/3)
Request 4 (Clarification 1) — Rationale for the relative endpoint loss
For classification, the per-molecule summed UA/OR score is
and Eq. (14) states that its global sum equals
Consequently, under fixed prediction harms, fixed acceptance mass , and a global sum over molecules, the summed score and selective risk have the same minimizers. This relation establishes the intended direction of the two decision scores.
We agree that it does not provide a direct optimization guarantee for Equation 15. The practical objective differs through local endpoint supervision, a finite candidate budget, the source-relative hinge, a changing predictor, soft coverage regularization, and the use of detached OOF harm in the OR outer score. The regression transformations introduce a further difference from the classification relation. We will state this scope limitation alongside Equation 14 in the main text rather than presenting the relation as a guarantee for the practical algorithm.
The source-relative loss compares each selected endpoint with a detached score of its own source. It penalizes only the amount by which the endpoint score exceeds that reference, whereas the absolute variants penalize every nonzero endpoint score. The formulations therefore differ in both their activation sets and the weighting of active updates.
The source score provides an adaptive reference for each source molecule. Detaching this reference prevents the model from reducing the penalty by manipulating the reference itself, while the squared hinge weights active updates according to the magnitude of the violation.
Table bdSH-4. Outer endpoint loss form, full training.
| Dataset | Relative | Absolute-square | Absolute-linear |
|---|---|---|---|
| BACE | 0.1637 ± 0.0124 | 0.1652 ± 0.0149 | 0.1668 ± 0.0099 |
| BBBP | 0.0887 ± 0.0194 | 0.0896 ± 0.0160 | 0.0908 ± 0.0155 |
| Tox21 | 0.0737 ± 0.0024 | 0.0739 ± 0.0008 | 0.0750 ± 0.0026 |
| ESOL | 0.6247 ± 0.0239 | 0.6288 ± 0.0259 | 0.6485 ± 0.0144 |
The relative formulation attains the lowest mean selective risk on all four tested datasets. Its improvement over absolute-square ranges from 0.0002 to 0.0041 in the respective metric units, and its largest improvement over absolute-linear is 0.0238 on ESOL. These results provide empirical support for the relative formulation on the tested benchmarks, but do not establish universal superiority.
We also compared the gradients at a fixed Base initialization. The relative hinge activates on 67.8–90.6% of sources. Its gradient direction is close to that of absolute-square, with cosine similarities of 0.976–1.000, while the relative-to-absolute-square gradient norm ratios range from 0.376 to 0.870. Thus, at this initialization, the two objectives differ primarily in their activation range and update weighting.
These fixed-Base diagnostics characterize the optimization difference but do not identify which specific mechanism causes the final performance differences. We therefore present the source-relative form as an empirically supported design choice on the tested benchmarks, not as a theoretically guaranteed source of optimization stability.
Request 5 (Clarification 2) — Interaction between UA and OR objectives
We agree that the combined objective should not be expected to dominate either individual objective on every dataset. Although UA and OR target complementary decision failures, they interact through the shared predictor, risk head, coverage constraint, and dynamically reselected endpoints. Their relative benefits may therefore depend on dataset-specific prediction errors, acceptance patterns, and OOF supervision quality.
The matched results illustrate this behavior: UA+OR improves over Base_op on eight datasets, but UA-only obtains a lower mean than UA+OR on BBBP, Tox21, and ESOL. We interpret this as dataset-dependent complementarity, rather than a contradiction between the two objectives.
Gradient conflict and optimization trade-offs are possible explanations, but our current experiments do not isolate either as the specific cause. Equal-norm PCGrad can mitigate destructive gradient interference, but it does not guarantee that combining UA and OR will produce a better validation or test operating point than either branch individually.
We will therefore avoid attributing the observed behavior to gradient conflict without direct evidence. We will also revise the manuscript to state explicitly that the combined UA+OR objective is not uniformly optimal and that the benefit of each branch varies across datasets.
Rebuttal (1/3)
We thank the reviewer for the detailed and constructive requests. These comments helped us clarify the independent contribution of the endpoint objectives, the respective roles of hard-example mining and structural locality, and the scope of the theoretical motivation. We added matched training controls, local/global and hard/random mining comparisons, joint selective-learning baselines, and absolute-loss variants. The new results support an incremental contribution from the proposed endpoint objectives, while also showing that additional training and hard-example mining account for part of the observed gains. We have revised our claims accordingly.
Request 1 (Additional Comparison 1) — Independent contribution of adversarial training
We agree that the original comparison did not fully isolate the effect of the endpoint objectives, because R2P(Base) was frozen while the dynamic variants received additional optimization. To address this confound, we added a matched Base_op control. Base_op retains the clean prediction loss, OOF risk supervision, and coverage penalty, while matching the optimizer updates, OOF refresh budget, checkpoint-selection procedure, and per-seed Base initialization of the UA/OR variants. Only the UA/OR endpoint losses are removed.
Table bdSH-1. Matched-budget attribution. The last column is the difference between the two means, Δ = R2P(UA+OR) − Base_op, in the metric's own units; per-seed differences carry their own spread, which is not shown in this column.
Base_op improves over Base on eight of the nine datasets, by 0.2–2.9%; on ClinTox, its mean risk is 0.4% higher (0.0452 against 0.0450). Thus, additional optimization accounts for a real part of the improvement over the original frozen Base. At the same training budget, the joint UA+OR model further lowers mean selective risk relative to Base_op on eight of the nine datasets, by 0.4–6.9%. Tox21 is the exception, where the joint model increases risk relative to both Base_op and Base. Some incremental differences, particularly on BBBP and ClinTox, are small relative to the seed variability.
These results therefore support an incremental and dataset-dependent contribution from the endpoint objectives, rather than attributing the entire Base-to-R2P improvement to adversarial training. They also do not imply that the joint objective uniformly dominates the individual branches: UA-only has a lower mean on BBBP, Tox21, and ESOL. We will revise the component analysis to report this attribution explicitly.