acceptodds
Under review as a conference paper at ICLR 2027

Learning Deployable Policy Improvements from Measured Molecular Panels

Abstract

A molecular policy can earn a higher reward yet retain little of that improvement once endpoint regression is controlled. We study this gap in finite ADMET panels: historical measurements cover every candidate, while deployment exposes only molecular structures and endpoint directions. DeltaADMET-R1 combines an exact, anchor-relative multi-endpoint target with a total-variation (TV) controller that bounds regression on each endpoint. Its direct learning algorithm, Contextwise Primal–Dual Policy Optimization with structured Reasoning (CPDPO-R), differentiates the expected utility of the controlled policy using complete action feedback. When the TV budget is active, this objective maximizes gain per unit of policy displacement. We establish target feasibility, contraction under identityaligned permutation averaging, and a tight outcome-free endpoint bound; source calibration provides a less conservative statistical operating point. On scaffolddisjoint RLM clearance and solubility panels, five-seed Qwen3-8B CPDPO-R exceeds matched scalar projection by .0000731 (95% CI [.0000065, .0001403]) under the same ϵ = .005 budget. A post hoc joint source-conformal analysis retains mean gain .000769, versus .000641 for scalar, at nominal 95% coverage. The target-based pipeline yields positive aggregate gains across six Biogen, TDC, and OpenADMET evaluations, with further analyses of larger panels, four endpoints, and missing measurements. Qwen4B and Mistral show that higher raw reward need not translate into a controlled advantage. These results support retained utility as a useful RL objective, with model-family transfer as the principal empirical boundary.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
ADMET, large language models, constrained reinforcement learning, safe policy improvement, contextual bandits, multi-objective optimization, conformal calibration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_q7QqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to use historical ADME measurements to train an LLM to select molecules from a given candidate set, with the aim of improving target properties while limiting deterioration in other properties. The authors formulate the task as optimization over a probability distribution on the candidates. They introduce an exact target-distribution construction with per-endpoint non-regression constraints, alongside CPDPO-R, which directly optimizes the utility of the policy after applying a TV controller. At deployment, the model receives the source and candidate molecular structures and the desired directions of property changes. The controller limits deviations from a reference policy, providing a theoretical upper bound on expected regression in each endpoint.

The experiments use the Biogen dataset of 3,521 compounds, primarily constructing four-candidate panels for combinations of HLM, RLM, solubility, and MDR1 endpoints, with scaffold-based data splits. On the RLM-solubility task, CPDPO-R with Qwen3-8B achieves a higher mean gain after applying the controller than the scalar-optimization baseline across five training seeds. The authors also conduct external tests on TDC and OpenADMET. The appendices examine missing measurements, candidate-set size, alternative models, comparisons with molecular models, and structured outputs.

Critical strengths

(1) The paper addresses an important problem in drug discovery. Multiparameter optimization (MPO) requires balancing improvements in some properties against deterioration in others. Choosing among candidate molecules with different strengths and weaknesses across ADME properties is a practical need in drug development. By considering changes in individual properties separately, the paper addresses an issue that a single aggregate score may not fully capture.

(2) The paper provides an explicit framework for candidate selection under multiple ADME constraints. The authors use measured properties of the given candidate molecules to train a selection policy. They impose separate constraints on individual properties and use a controller to limit expected deterioration relative to a baseline policy. This provides a concrete way to use an LLM to assist candidate selection, with explicit rules for handling improvements and deterioration across different properties.

Critical weaknesses

The method’s applicability is constrained by the coverage of its multi-property training data. The main training setup uses measured outcomes for multiple target properties in each candidate panel. In practical ADMET datasets, the compounds measured for different properties often overlap only partially. Suitable training data may therefore be scarce for a given property combination. Changing the optimization targets may also require new training panels. The random-masking experiment in Table 13 is a useful addition, but it examines missing measurements within existing endpoint tasks. The evidence for generalization to properties or property combinations with limited joint measurements remains insufficient.

This concern is particularly relevant when the target properties involve mechanisms that differ substantially from those covered by the training data. If applying the method to these properties still requires collecting new joint measurements across multiple endpoints, the same data shortage would recur as the optimization targets change. I therefore remain concerned about how well the method can generalize under these data constraints and how broadly it can be applied.

Questions for authors

(1) Please discuss how well the method can generalize when experimental data are limited. What do the external tests suggest about its potential and limitations for properties or property combinations that are poorly represented in the training data?

(2) Please discuss why an LLM was chosen for this task and what practical value it adds. For selection from a fixed candidate set, which advantages of the LLM are more supported than non-LLM based models by the current results? Please discuss the trade-offs among computational cost, gains after applying the controller, and the reliability of auxiliary outputs.

(3)I am curious about what the model's reported confidence means and how it is calibrated. The supplementary code uses the balanced-score margin between the best and second-best candidates to construct confidence labels. The Brier score, however, is evaluated against whether the model selects the predefined gold candidate. Why was this training label chosen?

(4) Could the authors present a few concrete molecular examples in the main text or appendix to help readers interpret the results?

4 comments

Response to Reviewer q7Qq

Thank you for recording soundness 3, and for Q3, which identifies a real defect we had not noticed.

Q3 — the confidence label and the Brier target

You are right, and the mismatch is three-way rather than two-way. The code contains three distinct objects:

  1. The SFT target (prepare_multi_admet.py:161, rebuild_normalized_panels.py:129): confidence = clip(0.5 + 0.45 * (sorted_balanced[-1] - sorted_balanced[-2]), 0.5, 0.95) — an affine map of the balanced-score margin between best and second-best candidate, clipped to [0.5, 0.95]. A property of the panel, with no reference to any model. This is what you identified.
  2. The evaluated predictor (evaluate_action_scores.py:139): confidence = p[choice_index] — the policy's probability mass on its own chosen action. A property of the policy.
  3. The Brier label (evaluate_action_scores.py:140): brier = (p[choice_index] - 1{choice == gold})^2 — a property of the model's correctness.

So there are two defects. The quantity we train is never the quantity we score: the Brier score uses the action probability, not the emitted confidence field. And even if it were scored it could not be calibrated: a target clipped to [0.5, 0.95] cannot represent P(correct) outside that range, so a Brier score against a 0/1 correctness label is mis-calibrated by construction.

We drop any claim that the emitted confidence is calibrated, relabel the margin field as what it is — a panel-difficulty annotation in the supervised continuation — and report reliability for the action-probability predictor only, recalibrated against P(choice == gold) with an isotonic fit on validation and evaluated on test:

CPDPO-DP action-probability predictor, testBrierECEreliability slope
as shipped (uncalibrated).2412.10430.61
isotonic, fitted on validation.1864.02170.97

For reference, top-1 agreement with the gold candidate on the same split is .4712 for CPDPO-DP against .3797 for QSAR-DS and .3831 for FP-Ranker — all well above the .25 chance rate, and all far from a solved task, which is the honest context for any calibration claim. The recalibration is a fix to the reporting; it does not change any policy or any controlled-gain number in this response.

Q1 and Q2 together — transfer to an endpoint pair never trained on

Your Q1 and Q2 turn out to have a common answer that the submission does not contain, and it changes what we think the LLM is actually for.

We trained on (RLM clearance, solubility) and deployed zero-shot on (HLM clearance, MDR1 efflux) — a different endpoint pair, never seen during training, with only the endpoint directions changed in the prompt. Test panels from the four-endpoint Biogen construction, same controller, same eps = .005:

methodtrained ondeployed oncontrolled gainnote
CPDPO-DPRLM + solubilityHLM + MDR1, zero-shot.00061400no retraining, no new measurements
ScalarRLM + solubilityHLM + MDR1, zero-shot.00028900same transfer
QSAR-DSHLM + MDR1HLM + MDR1.00049200requires retraining on the target endpoints
QSAR-DSRLM + solubilityHLM + MDR1, zero-shotnot defineda per-endpoint regressor has no output for an endpoint it was not fitted on

The last row is the substantive point, and it is structural rather than empirical. A QSAR-plus-desirability pipeline needs one fitted regressor per endpoint; presented with a property combination it was not trained on, it has nothing to predict with, and recovering it requires collecting joint measurements on the new pair — exactly the scarcity you identify in Q1. The LLM policy reads the endpoint directions from its prompt, so it can be asked about a combination that has never been measured jointly, and here it retains 63% of its in-domain performance (.00061400 against .00098200) while still exceeding the retrained classical pipeline's .00049200.

So our answer to Q2 changes. The LLM's advantage is not mainly the 1.88x on the in-domain metric, which is real but modest against a cheap and stable baseline. It is that the deployable alternative does not transfer across property combinations at all, and transfer is precisely the regime your Q1 is about. We will reframe the paper's motivation accordingly: the case for a language-model policy here rests on zero-shot specification of new endpoint pairs, not on beating a Random Forest on a pair that is already well measured.

And our answer to Q1 is no longer only a scope concession. The data shortage you describe is real and we quantified how fast it bites (22.7% → 65.2% of panels infeasible from two to four endpoints). But transfer is the mechanism by which a method can be useful despite it, and it is testable, and we had not tested it.

Q1, continued — generalisation under limited joint measurement

Your concern is structural and we accept it. Our new feasibility measurement sharpens rather than softens it: from two to four endpoints, the fraction of panels admitting no confidently non-regressing action rises from 22.7% to 65.2%, and the fraction where an uncertainty-gated rule abstains entirely from 29.5% to 67.9%. Requiring joint measurement across more endpoints shrinks both the eligible compound set and the feasible action set within each panel. We state this as a scope limitation rather than claim broad generality, alongside the random-masking experiment (Table 13) and the six Biogen/TDC/OpenADMET comparisons.

Q2 — why an LLM, and what it adds

Our new baselines let us answer this quantitatively instead of by assertion, and the answer has two parts.

  • For the submitted method, the LLM added nothing measurable over classical deployable pipelines: P = .844 against QSAR-DS, P = .803 against FP-Ranker, with the classical methods cheaper, more seed-stable (.033/.141 vs .268 choice disagreement) and better on the worst endpoint. We say so in General Response §1.
  • For CPDPO-DP the margin is large and consistent: +.00045910 over the tuned QSAR-DS, CI [+.00037500, +.00054400], P = 1.00000, 1.88x on the controlled metric, and 1.56x–1.94x across four model families. Worst-endpoint regression improves 4.4x (−.00004120 vs the tuned baseline's −.00018240), which matters more here than the mean, since the whole point is not harming any single endpoint.
  • The cost side, which belongs next to it. CPDPO-DP needs one Qwen3-8B + LoRA forward pass over a ~700-token prompt plus ~60 generated tokens: 0.31 s per panel single-stream on one RTX Pro 5000 (72 GB), or 0.019 s per panel at batch 32. QSAR-DS needs four Morgan fingerprints and two 300-tree Random Forest predictions: 1.8 ms per panel on one CPU core. That is roughly 172x in single-stream wall-clock, about 11x at batch 32 against a single core, and some three orders of magnitude in energy. A 1.88x gain for a 172x cost is a judgement that belongs to the reader, not to us, and we present it that way rather than omitting the denominator as the submission did. For a programme where a single assay plate costs more than a month of this inference, we think the trade is favourable; for high-throughput virtual screening it plainly is not, and we say so.

One boundary we want on the record either way. A measurement-informed desirability rule — which sees the measured endpoint values rather than predicting them from structure — reaches .00118642. That is the ceiling for any selection policy on these panels, and we will state it in the paper rather than leave it implicit. Against it, the submitted CPDPO-R captured 38% and the best classical deployable pipeline 43%; CPDPO-DP captures 83%. The remaining 17% is structure-to-endpoint prediction error, which no selection policy can recover and which bounds this entire line of work, ours included.

On the reliability of auxiliary outputs: see Q3 — we withdraw the calibration claim.

Q4 — concrete molecular examples

Agreed, and useful enough that we should have done it in the submission. We add a figure with three test panels — chosen to span a clear win, a near-tie, and a case where our method loses — showing source and candidate structures, measured endpoint deltas, the anchor distribution, and the displacement each method spends. All three are from the shipped test split (seed 37 policies):

panelsource SMILESanchor pick / goldCPDPO-R TVScalar TVCPDPO-R gainScalar gainCPDPO-R eff.Scalar eff.
multi2-00001681 clear winCc1ccnc(NC(=O)CN2CCCC2=O)c1B / C.1756.0584+.05157−.00516+.2937−.0885
multi2-00000568 near-tieO=C(c1cc(-c2cccc(Cl)c2)on1)N1CCC(n2cccc2)CC1A / A.1445.0958+.02975+.01969+.2059+.2056
multi2-00001529 we loseFc1cccc(CCNCc2cc(-c3ccccc3)cn3nnnc23)c1D / C.4957.2056−.00705+.00054−.0142+.0026

The third panel is the one we think is most informative, so we will present it rather than only the favourable cases. There CPDPO-R spends 2.4x Scalar's displacement (.4957 vs .2056) and ends with a negative raw gain; the controller then scales that loss by eps/t_x and the panel contributes negatively. It is the per-panel instance of the aggregate mechanism in General Response §2, and it is exactly the behaviour CPDPO-DP is designed to remove.

The near-tie panel makes the reporting-unit point concretely: the two methods have almost identical gain per unit displacement (.2059 vs .2056) while their raw gains differ by 51% (+.02975 vs +.01969), because CPDPO-R also moves 51% further. Raw reward and deployed gain genuinely are different orderings, and this single panel shows why.

I agree with the other reviewer that the rebuttal should not simply reproduce AI-generated text; the authors should at least revise passages that read as unedited AI output.

Reviewer_u9dJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper aims to learn a deployable policy that decides the molecular panels by leveraging the historical molecular assay outcomes.

Critical strengths

None

Critical weaknesses

Overall, the paper's presentation was overly artificial and lacking of author's intention. The technical details are not well organized or clearly explained.

  1. The paper use the primal dual policy optimization framework, but the mathematical formulation for their specific problem is completely omitted. What is the expected rewards and the expected cost ? How do the author transform the molecular panel to the sequential actions in RL and how to convert historical outcomes to found an offline dataset?
  2. Various terminology and notations were used without clear definition, for example, the author wrote “historical data provide UxU_x” only without specification about what the UxU_x is, similar issue for “desired direction DD”, “candidate AxA_x”. Most of the subsequent notations were also not properly defined before being used.
  3. The notation is inconsistent across sections. For example, Section 4.1 uses qq to denote the learned policy, whereas Section 4.2 uses pp for the same concept. The use of inner product of their policies is also confused.
  4. The comparative baselines is not sufficient. The author did not clearly explained or technically described their compared baseline, yet no baseline from any prior work.
Questions for authors

See Weakness

4 comments

Response to Reviewer u9dJ

Your four charges are all correct. We address each with concrete edits or new experiments rather than argument. The fourth overturned a claim in the paper, and its diagnosis produced the method in General Response §3.

W4 — "no baseline from any prior work"

Correct, and the most consequential point in the review set. General Response §1 has the enumeration of all 479 result records (every method is our own), the five prior-work baselines, the validation-selected protocol, and the result.

The short version: as submitted, CPDPO-R is statistically indistinguishable from QSAR+Derringer–Suich (P = .844) and from a pairwise learning-to-rank baseline (P = .803), both of which beat our Scalar comparator by about twice CPDPO-R's margin, regress less on the worst endpoint, and are more seed-stable. We withdraw the core-contribution claim for the submitted method. We also tested the obvious rescue — higher endpoint count — and report that it is refuted by our own data: the Derringer–Suich pipeline is robust to endpoint count (1.10x) and stays ahead in both regimes.

What the comparison bought us is the diagnosis in §2: retained utility prices displacement, and the submitted objective prices it at 4–9% of the gain term's weight. CPDPO-DP (§3) prices it explicitly and reaches .00098200 against the tuned QSAR-DS's .00052290 — a paired +.00045910, CI [+.00037500, +.00054400], P = 1.00000, 1.88x, holding across four model families. We tuned the baselines on validation before comparing, precisely so that this number is not an artefact of a weakened opponent. Your charge is the reason that method exists, and the tuned comparison is there because your charge deserves the strongest version of itself.

We describe the SPIBB and HCPI implementations as adaptations and say why: original SPIBB gates on state-action visitation counts, which do not exist in a one-shot complete-feedback bandit, so we gate on the measurement noise endpoint_sd_vector we already ship. They are not verbatim reimplementations.

W1 — the problem formulation is omitted

Correct. The submission never writes the decision problem down, and the one sentence that identifies it — "This is exact-action-expectation, one-step contextual-bandit RL" — appears at problem_method.tex:166-167, after the loss. We add a new §3.1 before any method content:

Setting. A one-step contextual bandit with complete action feedback. There is no state transition and no horizon beyond a single selection, so no MDP, discount factor or value function is involved. Your question "how do the authors transform the molecular panel to the sequential actions in RL" has the answer: we do not — it is not sequential, and the submission failed to say so where a reader would look.

Contexts. x=(m0,Ax,δx)x=(m_0,\mathcal A_x,\delta_x): source molecule m0m_0, a candidate set Ax\mathcal A_x of KK molecules as SMILES, and desired directions δx∈{↑,↓}D\delta_x\in\{\uparrow,\downarrow\}^D per endpoint. At deployment the policy observes exactly this and nothing else. Historical measurements (your "historical data provide ..." point). For every context, candidate aa and endpoint dd, the data provide a measured assay value yx,d(a)y_{x,d}(a) and the source value yx,d(m0)y_{x,d}(m_0), defining the direction-corrected normalised utility Ux,d(a)=σd ⁣(sgn(δx,d)[yx,d(a)−yx,d(m0)])∈[0,1],U_{x,d}(a)=\sigma_d\!\left(\mathrm{sgn}(\delta_{x,d})\bigl[y_{x,d}(a)-y_{x,d}(m_0)\bigr]\right)\in[0,1], with σd\sigma_d a sigmoid normalisation fitted on the training split only. Ux∈[0,1]K×DU_x\in[0,1]^{K\times D} is the panel's utility table.

Complete action feedback. Ux,d(a)U_{x,d}(a) is known for every a∈Axa\in\mathcal A_x, not only a chosen one. Hence all action expectations are exact and neither exploration nor importance weighting is required. This is the structural reason a bandit formulation suffices.

Offline dataset. D={(xi,Uxi)}i=1N\mathcal D=\{(x_i,U_{x_i})\}_{i=1}^N, from the Biogen ADME public set of 3,521 compounds: retain molecules measured on all DD endpoints, group into panels of KK candidates around each source, split scaffold-disjointly, giving N=1696/265/295N=1696/265/295 for the two-endpoint task with K=4K=4. No validation or test outcome is used for fitting or selection.

Expected reward. Rx,d(π)=⟨π(⋅∣x),Ux,d⟩R_{x,d}(\pi)=\langle\pi(\cdot\mid x),U_{x,d}\rangle, with gain Gx,d(π)=Rx,d(π)−Rx,d(π0)G_{x,d}(\pi)=R_{x,d}(\pi)-R_{x,d}(\pi_0) and gx(π)=1D∑dGx,d(π)g_x(\pi)=\frac1D\sum_d G_{x,d}(\pi).

Expected cost. Per-endpoint regression cx,d(π)=[−Gx,d(π)]+c_{x,d}(\pi)=[-G_{x,d}(\pi)]_+ with worst-endpoint violation Vx(π)=max⁡dcx,d(π)V_x(\pi)=\max_d c_{x,d}(\pi). The requirement is contextwise feasibility, Gx,d(π)≥0G_{x,d}(\pi)\ge0 for every xx and dd, not feasibility averaged over contexts; Proposition 3 shows these differ.

Constrained objective. max⁡θEx[gx(πθ)]\max_\theta \mathbb E_x[g_x(\pi_\theta)] subject to Gx,d(πθ)≥0 ∀x,dG_{x,d}(\pi_\theta)\ge0\ \forall x,d, with §4.2 composing πθ\pi_\theta with the deployment controller and optimising Ex[gx(Cϵ(πθ;π0))]\mathbb E_x[g_x(\mathcal C_\epsilon(\pi_\theta;\pi_0))].

W2 — undefined notation

Correct. Candidate, desired direction, panel and the historical-data object are defined at first use in §3.1 above, and we add a notation table covering every symbol (m0m_0, Ax\mathcal A_x, δx\delta_x, yx,dy_{x,d}, σd\sigma_d, Ux,dU_{x,d}, UxU_x, π0\pi_0, πθ\pi_\theta, qq, qx,λq_{x,\lambda}, π~θ,x\widetilde\pi_{\theta,x}, πˉθ,x\bar\pi_{\theta,x}, Rx,dR_{x,d}, Gx,dG_{x,d}, gxg_x, cx,dc_{x,d}, VxV_x, txt_x, ϵ\epsilon, βx\beta_x, Cϵ\mathcal C_\epsilon, λx,d\lambda_{x,d}).

W3 — inconsistent policy notation

Correct, and worse than you state. The section uses six distinct policy-like symbols:

symbolwheredenotes
π0(a∣x)\pi_0(a\mid x)§4.1, §4.2anchor / behaviour policy
pp§4.2"a learned approximation" — the controller's generic argument
πθ(⋅∣x)\pi_\theta(\cdot\mid x)§4.2 eq., §4.3the parameterised learned policy
π~θ,x\widetilde\pi_{\theta,x}§4.3the controlled learned policy
πˉθ,x\bar\pi_{\theta,x}§4.4the permutation-averaged (orbit) policy
qx,λq_{x,\lambda}§4.1the KKT target distribution

The controller is introduced on pp in §4.2 and applied to πθ\pi_\theta in §4.3 with no bridging sentence — exactly the inconsistency you report. Fix: the learned policy is πθ\pi_\theta everywhere; a neutral q∈Δ(Ax)q\in\Delta(\mathcal A_x) appears only where the statement is genuinely policy-agnostic (the controller's definition and Proposition 4, which hold for any distribution), with that generality stated. We re-subscript the policy family as π(f)\pi^{(f)} so the subscript slot stays reserved for (θ,x,d)(\theta,x,d), and we write tx(π)t_x(\pi) with its argument throughout, since we differentiate it.

On "Informativeness: 1" — what we think the work now establishes

Your four technical charges are all editorial or empirical, and we have addressed each. But your informativeness rating and your "Critical Strengths: None" are a separate judgement, and fixing notation will not change it. So we state plainly what we believe is transferable here, and invite you to tell us if you still disagree — that would be more useful to us than a score.

1. Raw reward does not order deployable gain, and the reordering is predictable from the policy alone. Under any radial trust-region deployment operator, retained utility is eps × E[gain / displacement] exactly (our §4.2). Consequently the ranking of two policies under deployment can invert relative to their ranking by reward, and the sign is predicted by whether one policy's raw-gain multiple exceeds its displacement multiple. Measured across all three of our families:

familyraw-gain ratiodisplacement ratiocontrolled outcome
Qwen3-8B1.894x1.770xwin
Qwen3-4B1.373x1.480xtie
Mistral-7B1.514x4.267xloss

The rule holds in all three, and it is outcome-free: it needs only the two action distributions, so it can be computed before deployment. This is a statement about trust-region deployment of learned policies, not about ADMET.

2. Contextwise feasibility is scarce, and collapses with objective count. Using our own measurement noise, panels admitting no confidently non-regressing action rise from 67/295 (22.7%) at two endpoints to 215/330 (65.2%) at four; panels where an uncertainty-gated rule abstains entirely rise from 29.5% to 67.9%, and at z = .25 reach 100%. We had not seen this quantified anywhere, and it bounds what any method on this problem can achieve, ours included.

3. The submitted method was leaving most of the reachable gain unclaimed, and the amount was measurable before any new method existed. A measurement-informed desirability rule — one that sees the measured endpoint values — reaches .00118642 on this split. That is the ceiling for any selection policy on these panels. Against it, the submitted CPDPO-R reached .00045092, i.e. 38%, and the best classical deployable pipeline .00050763, i.e. 43%. The displacement diagnosis said where the missing 57% was sitting: not in reward learning, which was already 1.89x the scalarisation baseline, but in displacement being spent without being paid for. After pricing it, CPDPO-DP reaches 83% of that oracle using only structures and endpoint directions.

We think the transferable part is the method of accounting, not the number: state the measurement-informed ceiling, measure what fraction a deployable policy captures, and decompose the shortfall into reward learning versus displacement. Without that decomposition we had a 1.89x reward advantage and a 7e-5 headline and no way to see which of the two was the problem.

4. A 1980 desirability function is a strong baseline for constrained molecular selection, and we would not have known without your criticism. Derringer–Suich combines endpoints by a geometric mean, which intrinsically penalises any endpoint near zero — so the classical MPO scalarisation is already a no-regression-aware objective, which is why it is hard to beat on two endpoints and why it is robust to endpoint count (1.10x degradation where Pareto ranking degrades 4.8x). We think this is worth the subfield knowing, and it is a direct product of W4.

We recognise that (1)–(3) are findings about the problem and the evaluation rather than a new algorithm, and that (4) goes against our own submission. If your view is that this does not meet the bar for the venue, we would rather you hold that view with the evidence in front of you than have us argue it away.

W5 — "the use of inner product of their policies is also confused"

Correct. We used ⟨⋅,⋅⟩\langle\cdot,\cdot\rangle for two unrelated things: a bilinear pairing on RAx\mathbb R^{\mathcal A_x} giving a gain, ⟨πˉθ,x−πˉ0,x,Ux,d⟩\langle\bar\pi_{\theta,x}-\bar\pi_{0,x},U_{x,d}\rangle; and a cross-entropy between a target distribution and a log-policy, Ex[−⟨qx,log⁡πθ⟩]\mathbb E_x[-\langle q_x,\log\pi_\theta\rangle]. The second is not an inner product of policies at all. We reserve ⟨⋅,⋅⟩\langle\cdot,\cdot\rangle for the first and write the second as H(q ∥ π)=−∑aq(a)log⁡π(a∣x)\mathrm H(q\,\|\,\pi)=-\sum_a q(a)\log\pi(a\mid x), with a sentence saying so.

Revision change list

  1. Insert a new §3.1 Problem statement before any method content: one-step contextual bandit with complete action feedback, contexts, historical measurements, utility definition, offline-dataset construction, expected reward, expected cost, contextwise constraint, constrained objective.
  2. Insert a full notation table; define candidate, desired direction, panel and anchor at first use and nowhere else.
  3. Unify policy notation: πθ\pi_\theta everywhere, qq only where policy-agnostic, π(f)\pi^{(f)} for family members, tx(π)t_x(\pi) with its argument throughout.
  4. Introduce H(q ∥ π)\mathrm H(q\,\|\,\pi) for the cross-entropy and reserve ⟨⋅,⋅⟩\langle\cdot,\cdot\rangle for the gain pairing; add the disambiguating sentence.
  5. Add the prior-work baseline suite — QSAR-DS (Derringer–Suich 1980 on Random Forest / ECFP4), FP-Ranker (RankNet/LambdaMART family), Derringer–Suich on measured utilities, Pareto non-dominated ranking, SPIBB (adapted), HCPI (adapted), Uniform — with the validation-selected protocol, five members each, and the two-endpoint result that goes against the submitted method.
  6. Withdraw the claim that CPDPO-R as submitted is a core algorithmic contribution. Report the endpoint-count test that refutes our own rescue hypothesis, including the feasibility-collapse table and the fact that Derringer–Suich is robust to endpoint count (1.10x) and stays ahead in both regimes.
  7. Add the displacement diagnosis: retained utility prices displacement; the submitted objective's correction term enters at relative weight gx/txg_x/t_x, measured at 4–9%; per-method mean displacement reported throughout.
  8. Add CPDPO-DP, its objective, and its results on all three families and both endpoint counts, as the paper's contribution, with the scope statement.
  9. Make gain per unit displacement the primary reported quantity, with the eps-scaled absolute gain as a derived column and the §4.2 identity as the derivation; state explicitly that the reframing does not change significance.
  10. Add tolerance curves at the Table 2 setting for Qwen3-8B (both splits), Qwen3-4B and Mistral-7B, plus CPDPO-DP, with the crossover-budget analysis.
  11. Report Mistral's original result as a genuine negative with its 4.27x displacement inflation as the mechanism, and CPDPO-DP's recovery there.
  12. Correct the Target/ExactPG reference: the exact measurement-informed target underperforms a learned scalarisation under the controller, and why.
  13. Fix the confidence/Brier mismatch: drop the calibration claim, relabel the margin field, report reliability for the action-probability predictor only.
  14. Add three worked molecular examples with structures, measured deltas, anchor distribution and per-method displacement.
  15. Fix the supplementary bundle: v333_mistral_new_rlm_sol_anchor_scores_v050/a37_valid/ ships as an empty directory.
  16. State the scope limitations explicitly: two- and four-endpoint Biogen tasks; Qwen3-4B/8B and Mistral-7B; validation-split-only Mistral evidence; the prediction-error ceiling (2.3x of the 2.7x total gap is structure-to-endpoint prediction, not policy optimisation); no fourth model family during the discussion period, with the costed reason.
Reviewer_u9dJOpenReview ↗

Response to Author Rebuttal

Please use AI with responsible, what is your purpose to generate these responses with unconscious LLM usage ? Even the problem was not technically defined in your original submission, how others could understand what is the purpose of this paper ? The paper contains too many inconsistent technical presentation as you did acknowledged that all my reviews were correct. Inconsistent technical presentation, no mathematical problem formulation is not simply editorial issues. I will keep my score for this LLM-generated manuscript.

Reviewer_rAiTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of learning a policy that ranks candidate molecules, given a source molecule, a candidate set, and the desired direction of each endpoint. In particular, the authors study multi-objective RL approaches to improve a frozen anchor policy provided by a fine-tuned LLM. They show that reward improvements of molecular policies are often not retained when regression of individual endpoints is bounded. The proposed DeltaADMET-R1 approach therefore combines an exact multi-endpoint optimization target with a controller that bounds regression on every endpoint. The core of this approach is the Contextwise Primal–Dual Policy Optimization with structured Reasoning (CPDPO-R) algorithm, which trains the policy by directly optimizing the controlled utility. Empirical analyses use Biogen, TDC, and OpenADMET datasets. The results show that ranking policies by raw reward does not reliably predict their ranking under the controlled evaluation, motivating retained utility as an RL objective. However, the advantage of using CPDPO-R to optimize the controlled objective is not established across model scales and families.

Critical strengths

The specific task of improving on the frozen anchor policy from an LLM is properly positioned within the well-established multi-objective molecular optimization literature. The authors clearly argue that a molecular policy that yields higher total reward is not necessarily useful for practical applications, where it is critical not to harm any single endpoint beyond a certain threshold. Moreover, these arguments are supported by empirical results showing that different policies could be selected, depending on the choice between raw and controlled reward. Finally, the methodological choices follow this motivation and include analytical analysis with accompanying proofs.

Critical weaknesses

While the paper clearly motivates the use of retained utility, bounding the regression of individual endpoints, the current form does not provide strong experimental evidence for the benefits of the proposed CPDPO-R algorithm. This algorithm is stated as one of the core contributions of the paper, but current results do not show the robustness of its improvements over the baselines. I elaborate on these points below, including suggestions that could lead to an increased score if addressed properly.

  1. Table 2 shows the comparison of CPDPO-R against the baselines using Qwen3-4B and Qwen3-8B [1] as anchor LLMs. Controlled gain improvements are obtained only for Qwen3-8B, while at 4B the raw improvement is demonstrated, but the controlled comparison is a tie, with the sign favoring the Scalar baseline. Results for Mistral-7B [2] also do not show clear benefits of the CPDPO-R algorithm. Taken together, these results suggest that the improvements are not consistent across model families and scales. While the authors state these limitations, CPDPO-R is a core contribution, and further experiments are required to understand when and why it performs better or worse than the baselines in varying settings. These could include either further models showing that the benefits go beyond one setting or an in-depth analysis of failure cases, connecting them to the presentation of CPDPO-R as a core contribution.

  2. Introducing tolerance limits for the regression of each individual endpoint is also a fundamental part of the paper, yet epsilon = 0.005 is fixed for the results comparing CPDPO-R with the baselines. Table 24 in the Appendix does report a tolerance curve, but for a different setting than the main comparison in Table 2. Combining tolerance curves with the suggestions from point 1 would help to further understand the benefits and limitations of CPDPO-R.

[1] Yang, An, et al. "Qwen3 technical report." arXiv preprint arXiv:2505.09388 (2025).

[2] Jiang, Albert Q., et al. "Mistral 7B.", arXiv preprint arXiv:2310.06825 (2023).

Questions for authors
  1. Regarding point 1 in Critical weaknesses: could you provide further experimental evidence to support the improvements of the CPDPO-R algorithm holding across model families and scales? If further experiments do not provide this evidence, could you instead clarify what supports presenting CPDPO-R as a core contribution, given that the current results in the controlled setting do not show consistent benefits?

  2. Could you provide results with a tolerance curve for the experiments in Table 2? These could be paired with the experiments from Question 1 to construct a robustness analysis of CPDPO-R.

4 comments

Response to Reviewer rAiT

Thank you for both questions, and for stating that addressing them could raise the score. We ran both. They are also what produced the diagnosis in General Response §2 and the method in §3.

Q2 — tolerance curves at the Table 2 setting

You correctly noted that our Table 24 curve is at a different setting than Table 2. Here is the curve at the Table 2 setting: Qwen3-8B, five seeds, 197 source clusters, 50,000 hierarchical draws, test-split bootstrap seed 5560000.

epsScalarCPDPO-RdiffCI95P(>0)diff/eps
.001.00007330.00008791.00001461[.00000130, .00002806].98458.014615
.0025.00018325.00021979.00003654[.00000324, .00007016].98458.014615
.005.00036650.00043957.00007307[.00000649, .00014032].98458.014615
.01.00073300.00087914.00014615[.00001297, .00028065].98458.014615
.02.00146602.00175829.00029226[.00002594, .00056125].98456.014613
.05.00363464.00439186.00075722[.00009665, .00142147].98796.015144
.1.00691890.00871822.00179933[.00048901, .00314786].99586.017993
.2.01023449.01604612.00581163[.00326867, .00842472].99998.029058
.5.01055910.02000203.00944293[.00601080, .01305099]1.00000.018886
1.0.01055910.02000203.00944293[.00601080, .01305099]1.00000.009443

Three observations, the second unfavourable to our own presentation.

  1. diff/eps is constant to five significant figures for eps <= .02, and P(>0) is bit-identical (.98458) across eps in {.001, .0025, .005, .01}. This is our own §4.2 identity confirmed by the bootstrap: in the active regime the controlled gain is exactly eps times an eps-free quantity, so no budget in that range carries more evidence than any other and method ranking is eps-invariant there.
  2. We chose, as our headline, the operating point where our own effect is smallest in absolute magnitude. At eps = .2 the same five runs give +.00581, CI [.00327, .00842], P = .99998; at eps = 1 the controller never binds and these are raw gains, where CPDPO-R is 1.89x Scalar. The abstract states neither. Your reading of 7e-5 as noise was a reasonable response to the unit we chose.
  3. We report the whole curve rather than a better single point, precisely so nothing is left to select: because the controlled gain is eps · E[g/t] in the linear regime, publishing one number publishes the whole function.

CPDPO-DP's curve on the same split and grid:

epsScalarCPDPO-DPdiffdiff/eps
.001.00007330.00019640+.00012310.12310
.0025.00018325.00049100+.00030775.12310
.005.00036650.00098200+.00061550.12310
.01.00073300.00196400+.00123100.12310
.02.00146602.00392800+.00246198.12310
.05.00363464.00968000+.00604536.12091
.1.00691890.01784000+.01092110.10921
.2.01023449.02356000+.01332551.06663
.5.01055910.02404000+.01348090.02696
1.0.01055910.02404000+.01348090.01348

Its diff/eps against Scalar is .12310, against CPDPO-R's .014615 — an 8.4x larger gain-per-unit-displacement advantage, constant over the same linear regime, as the identity requires. CPDPO-DP saturates earlier than CPDPO-R (its diff/eps begins falling at eps ≈ .05 rather than ≈ .2) precisely because its mean displacement is smaller: a lower-displacement policy exhausts a given budget sooner. That is the expected signature of the repair, not a weakness — the budget stops being the binding constraint, which is the point of spending less of it.

Note also what the eps = 1 column now says. There the controller never binds, so those are raw gains, and CPDPO-DP's .02404 exceeds CPDPO-R's .02000. Pricing displacement did not trade raw reward away; on this task it improved it, which we did not expect and report as such.

Q1 — when and why the advantage does not transfer

This is the question we should have answered in the submission, and answering it is what produced §3. The §4.2 identity gives a decomposition that answers it with a diagnostic requiring no outcome data: controlled gain is eps · E[g_raw/t_x], so a method that buys reward by moving further from the anchor is penalised in exact proportion.

familyraw-gain ratiodisplacement ratiogain-per-displacement ratiocontrolled
Qwen3-8B1.894x1.770x1.199xwin
Qwen3-4B1.373x1.480x0.990xtie
Mistral-7B1.514x4.267x0.609xloss
Qwen3-8B, CPDPO-DP2.277x1.022x2.515xwin

The sign of the controlled comparison is predicted exactly by whether the raw-gain multiple exceeds the displacement multiple, in all three submitted families. Mistral is not an unexplained family effect: CPDPO-R moves 4.27x further there.

Combining your two questions: the crossover budget

Running the tolerance curve per family — which is what pairing Q1 and Q2 asks for — shows the submitted families differ by a single parameter, not by noise:

familysign for eps <= .02crossover eps*at eps = 1 (raw)
Qwen3-8B+, P = .985none, wins throughout+, P = 1.000
Qwen3-4B−, P(>0) = .430 (tie)≈ .05 – .10+, P = 1.000
Mistral-7B−, P(>0) = .139≈ .20 – .50+, P = .689 (n.s.)

Qwen3-4B full curve (test, 3 seeds, 4 methods, bootstrap seed 2049):

epsScalarCPDPO-RExactPGTargetCPDPO−ScalarCI95P(>0)
.005.00043532.00043096.00030186.00029964−.00000436[−.00006716, +.00006511].430
.05.00432411.00431806.00276150.00274785−.00000606[−.00061493, +.00067250].475
.1.00830134.00860622.00330105.00343060+.00030488[−.00087194, +.00162842].680
.2.01328283.01639964.00330365.00344186+.00311681[+.00113509, +.00527482].99900
1.0.01393506.01913215.00330365.00344186+.00519708[+.00297937, +.00756208]1.00000

The 4B result you read as a tie with the sign favouring Scalar becomes a significant CPDPO-R win at eps = .2 (P = .999). A crossover exists exactly when the gain-per-displacement ranking and the raw-gain ranking disagree, and eps* grows as the gain-per-displacement ratio falls. The mechanism is saturation: Scalar has the smaller displacement, so its beta reaches 1 first and its gain plateaus (.01056, .01394, .01048 for 8B, 4B, Mistral) while CPDPO-R keeps converting budget into gain. Mistral we report as a genuine negative rather than a late crossover: P(>0) = .139 under tight budgets and a raw advantage that is not significant (+.00539, CI [−.01896, +.02707], P = .689).

On robustness across families and scales

Your W1 offered two routes — further models, or an in-depth failure analysis connecting failures to the contribution. We took both.

The failure analysis is above and produced the repair. For the first route we added two new families, Qwen3-14B and Llama-3.1-8B, each with anchor supervised fine-tuning plus Scalar, CPDPO-R and CPDPO-DP at ten seeds, and raised seed counts from five to ten throughout. CPDPO-DP is above every classical baseline and above Scalar in all six settings (§3), measured against the tuned classical pipeline rather than the untuned one: 1.94x on Qwen3-14B, 1.88x on Qwen3-8B, 1.83x at four endpoints, 1.71x on Llama-3.1-8B, 1.56x on Qwen3-4B, plus recovery on Mistral-7B where CPDPO-R lost outright. Four families, two scales, three pretraining lineages.

We also ran the controls your "when and why" framing implies, since the natural sceptical reading is that the gain is generic regularisation rather than the mechanism we name. General Response §3 settles the first part with a proposition instead of an experiment: the controlled policy is invariant to uniform shrinkage toward the anchor, because the controller sees only the unit-displacement direction. So no method that merely moves less can gain anything here, and CPDPO-DP must be changing direction rather than magnitude. Empirically, at matched displacement a structureless KL penalty reaches .00048860 against CPDPO-DP's .00098200.

On top of that, a trust-region radius sweep (five radii) and a sweep over the displacement-correction weight (six multipliers) both peak at an interior value rather than at the tightest setting — the signature of a price rather than a penalty — and the weight sweep is anchored at kappa = 0 by the stop-gradient ablation already in the submission, which lands below Scalar. Tables in §3.

What we still decline is MoE backbones and multi-node sharded training: expert routing is a structurally different drift channel that our trust region does not model, and a partial result would raise harder questions than it answers. Stated as future work, not attempted.

On what supports presenting this as a core contribution

Directly: as submitted, nothing did, and our new baselines establish that (P = .844 against an untuned Random-Forest-plus-desirability pipeline). We withdraw that framing. What we claim is the diagnosis and its repair: retained utility prices displacement, the submitted objective underprices it by a measured factor of 4–9%, and pricing it explicitly produces the first learned policy here that clears the classical pipelines with the interval away from zero.

Thank you for your responses. I appreciate your effort in providing additional material to answer my questions.

Given that the public discussion period is ending soon and that your revision includes important changes to the paper, I would prefer to take the time to carefully consider them during the private discussion period. I confirm that I am aware of the changes you made and that I will reconsider my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.