acceptodds
Under review as a conference paper at ICLR 2027

Multi-Teacher On-Policy Distillation for Multi-Property Molecular Optimization

Abstract

Lead optimization requires a generative policy to improve every property named in an instruction while keeping the molecule valid and structurally close to the lead. Property specialists trained with reinforcement learning are cheap to obtain, but composing them is not: a molecular instruction activates several specialists at once, they differ in how dependable they are at a given molecular state, and specialists that propose comparably reasonable molecules may still prefer different continuations of the same partial structure. We propose reliability-aware multi-teacher on-policy distillation (RA-MOPD), which folds a bank of frozen property experts into a single policy that queries neither experts nor property oracles at inference. At every state the student visits, RA-MOPD weights the requested experts by held-out reliability together with verified outcomes of their own proposals at that state. The weighted experts are combined into a full-vocabulary product of experts on the student's realized prefixes, and we show that the log-normalizer of this product is exactly the minimum weighted reverse-KL disagreement among the active experts, which we use as a token-local gate on the distillation loss. A group-relative verifier reward supplies the complementary trajectory-level signal, scoring complete recurrent edits, while the weighted teachers provide token-level distributional supervision on the molecular answer. Across ten three- and four-property tasks, RA-MOPD improves constrained success over supervised fine-tuning, verifier-only reinforcement learning, offline expert imitation, parameter merging and uniformly weighted multi-teacher distillation, and its advantage widens as every property is required to improve by a larger margin.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Large Language Models, Multi-Property Optimization, Molecular Editing, Expert Coordination, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SK5YOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper propose RA-MOPD, a multi-teacher distillation strategy to fuse multiple-teachers knowledge on lead design to a single unified model. Compared to conventional multi-teacher OPD, the author put the problem in a more dedicated lead optimization setting, where multiple objectives need to be optimized. In this case, multiple teacher's log-prob need to be fused together to get the student signal. However, the authors notice the insufficiency of uniform fusion, where some ineffective teachers' log-prob is given the same weight as the effective one. Hence the authors propose to compute the teacher weight from held-out realiability and weight each token via a product of experts. Under this training, the distillation process can get high success rates.

Critical strengths

The paper identified a critical setting, lead optimization, that is well-tuned for the application OPD. It propose a dedicated solution for handle the multi-target scenario. Compared to off-the-shelve solutions, this one measures a teacher's usefulness at a specific state, which I found is new. More specifically, the usefulness is from a current lead molecule to the student, and gating and weighting technique is a natural subsequent step to utilize such usefulness score.

Critical weaknesses

I have two complaints one on the conceptual novely and one on the empirical results:

  • The novelty is real, but I think this is not really exciting (maybe other reviewers can say more on if this is important). Essentially, one can think of this as a version of soft best-of-N, but we don't know why this is the method and potentially the preferrable one. There is no theoretical justification of why a single greedy sample of teacher satisfy the weighting job, and how that relates to a student distillation target. And it seems not straight forward to adopt the methodology to other domains.

  • For the results, compare table 3 and table 2, Mixed RL + Vanilla MOPD is outperforming every ablated cases. From table 12, this also seems to be the case. This means that any single componenet is infact harmful, and the composition of the three somehow revert the harmfulness, and outperform vanila, this phenomenon is not explained. Since the key hypothesis is that routing and gating are helpful, the ablation actually shows a contradiction.

Questions for authors
  • What is the dataset used? There is no mention of the dataset in the paper, just different cases.
4 comments
AuthorsOpenReview ↗

Response to Q1

Thank you for this suggestion! We use the publicly available MuMOInstruct dataset, which covers six molecular properties, including blood–brain barrier permeability, dopamine D2 receptor activity, human intestinal absorption, mutagenicity, penalized logP, and quantitative estimate of drug-likeness. Starting from its official training split, we construct data for four training stages through sample filtering, rationale synthesis, and multi-step trajectory construction.

StageData construction
Stage I — Continual Pre-trainingSelect local-edit and single-property optimization examples, supplemented with automatically checked rationales and high-similarity replay examples.
Stage II — Shared Multi-property Supervised Fine-TuningConstruct multi-property optimization examples with rationales, and organize reference molecular pairs into two-step supervised examples containing editing history and feedback.
Stage III — Single-property Teacher Reinforcement LearningPartition prompts by property. Each teacher generates candidate molecules online, and the verifier provides training rewards.
Stage IV — Multi-Teacher On-Policy Distillation and Reinforcement LearningThe student generates editing trajectories online from multi-property prompts. Frozen teachers provide distillation supervision, while the verifier supplies reinforcement-learning rewards.

The main-text evaluation uses a balanced set of 500 inputs, with 50 per combination. The full evaluation set contains 3,905 deduplicated initial-molecule–property-combination pairs.

We will explicitly identify and cite the dataset in the main experimental setup and consolidate the descriptions of the four-stage data construction pipeline and evaluation splits.

AuthorsOpenReview ↗

Response to W1

Thank you for this suggestion. Our motivation is to integrate the capabilities of multiple property specialists into a single student, improving candidate quality under a limited generation budget. Best-of-N draws on the base model’s existing output distribution by generating and scoring more candidates, increasing the opportunity to find a high-quality answer. Its effectiveness depends on the model’s coverage of desirable solutions and the reliability of the scoring function.

For molecular optimization, candidate quality matters because downstream synthesis and experimental evaluation are costly. As a simplified illustration, assuming independent candidates and equal validation costs, increasing the success probability per candidate from 20% to 50% reduces the expected number of evaluations needed to obtain one successful molecule from five to two. Computational screening can further reduce the number of candidates entering wet-lab experiments. This motivates our focus on joint multi-property success under a fixed, single-candidate generation budget.

We therefore train six single-property teachers through specialized reinforcement learning, allowing each teacher to acquire editing strategies for its corresponding property. We then combine their supervision at states visited by the student. Teacher weights depend on the property improvement, collateral harm, and reliability of their current proposals, while token-level conflict gating adjusts supervision strength at positions where teachers disagree. This makes the basis for combining expert knowledge directly inspectable.

A single greedy proposal from each teacher is used to estimate its routing utility at the current state. Each teacher subsequently provides its full next-token distribution conditioned on the student’s actual generated prefix. Although reinforcement learning has been observed to reduce output diversity in some settings, the adequacy of one proposal still requires empirical examination.

We therefore directly tested whether increasing the teacher proposal budget improves performance. The following supplementary matched-control results use compact500 and greedy student evaluation.

Teacher proposal configurationProcedureTask success rate on compact500
One greedy proposal per teacherScore each proposal to compute the current teacher routing weights.28.00%
Four sampled proposals per teacherSelect each teacher’s highest-utility proposal, then compute the routing weights across teachers.26.20%

Existing multi-teacher on-policy distillation research has explored integrating the capabilities of reinforcement-learning specialists from different domains into a single student. Our work focuses on how to combine teacher knowledge according to the current state and moderate inconsistent supervision when the same molecule must satisfy several potentially conflicting objectives. We see multi-objective coordination as a promising direction for multi-teacher on-policy distillation and welcome the reviewer’s perspective on this research direction and additional ways to evaluate it.

AuthorsOpenReview ↗

Response to W2

Thank you for this observation. We agree that the ablation results require further explanation.

The three components are interdependent rather than additive. The reliability prior and collateral-harm term jointly determine state-level teacher routing weights (Eq. 6), which subsequently affect both the product-of-experts target and its conflict gate (Eqs. 7–8). Removing either routing component changes the teacher weights and consequently the token-level supervision. In contrast, removing conflict gating preserves the original routing weights but forces the student to follow the fused teacher target even at positions of strong disagreement.

This explains why partial configurations need not outperform uniform weighting. On compact500, Mixed RL + Vanilla MOPD achieves 26.20%, compared with 26.20%, 25.40%, and 25.80% for the three component ablations (without reliability prior, collateral-harm term, and token-level gating, respectively). The complete RA-MOPD achieves 28.00%. These results suggest that the benefit arises from coordinating reliability-aware routing, collateral-harm prevention, and conflict-sensitive supervision, rather than from independently adding their contributions.

We will clarify that our ablations measure each component's contribution within the complete framework, rather than establishing that every partial configuration individually improves upon uniform multi-teacher distillation.

Dear Authors,

Thank you for your response. For W1, I think "single sample" works, but the question was on why. The experiment explain a bit, but how the weighting and distillization target connects is not explained. For W2, I think its just a restate of the phenomon, and no clear explanation. Again, the idea is interesting to me, but the mechanism needs more study. I will keep my current score.

Reviewer_VYfmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper targets instruction-conditioned lead optimization, where a policy must improve every property named in a natural-language instruction while keeping the molecule valid and structurally close to the lead. The authors propose RA-MOPD, which distills six frozen single-property RL experts into one student that queries neither experts nor property oracles at inference. Composition acts at two resolutions. At each molecular state, every requested expert decodes a candidate edit which the frozen property functions score for own-property progress, collateral harm to co-requested properties and similarity violation; these verified outcomes combine with a held-out reliability prior into softmax routing weights. At each token, the routed experts are fused into a full-vocabulary weighted product of experts over the student's realized prefix, and the authors prove that the log-normalizer of this product equals the minimum weighted reverse-KL divergence from any single distribution to the active experts, which they convert into a multiplicative attenuation on the distillation loss. A group-relative verifier reward supplies trajectory-level supervision, and distillation is restricted to the tagged SMILES span. Validation is a strict-conjunction constrained success rate over ten property combinations, five seen during multi-objective training and five held out, under a two-round single-sample greedy protocol, against SFT, task arithmetic, verifier-only RL, RePO, off-policy imitation and uniformly weighted multi-teacher distillation, with component ablations, a divergence study, margin and recurrence analyses, and a full-set confirmation in the appendix.

Critical strengths

The theoretical analysis is correct, and I worked through it independently rather than reading the proofs. For any distribution on the common support, the weighted sum of reverse KLs rewrites as the KL to the product distribution minus the log-normalizer, so the product is the unique minimizer and the minimum equals the negative normalizer; Proposition 2 follows from feasibility of each expert, and the forward-KL gradient in Section 3.4 is right. The arithmetic is also sound throughout, including the table aggregates, the reliability posterior means and the recurrence decomposition. I mention this because it means my objections are about what the evidence supports rather than about errors, and because an exactly stated identity gives the token-level design a firmer footing than a tuned heuristic would have. It is not sufficient on its own, since the identity is the standard analysis of the logarithmic opinion pool rather than a new result, and the paper would be stronger for saying so and citing it.

Critical weaknesses
  • I do not think the comparison against uniformly weighted multi-teacher distillation is established. There are no seeds, no error bars and no significance test anywhere in the paper, and on the compact benchmark the margin over that baseline falls within the sampling error of the metric at this number of prompts per combination. Since this is the comparison the paper's contribution rests on, I would want some indication of run-to-run spread before reading it as a result.

  • Removing any single component drops performance below Vanilla MOPD, which contains none of the three, and on the larger evaluation set all three partial configurations sit below it. I would expect them to fall somewhere between that baseline and the full method. As the table stands, the attribution of a specific role to each component in Section 5.2 does not follow from it, and I cannot tell whether the components only function as a package or whether these differences are noise.

  • The cost table makes me doubt the ablations were run under matched budgets, since the distillation-only variant is reported as considerably more expensive than the full method even though it drops the policy-gradient path. If the variants differ in update count or compute, their ordering says little about the components themselves.

  • I cannot tell from the paper whether the token-level attenuation is doing much work. The expert disagreement that motivates it is never measured directly, and neither is the distribution of the conflict term the method computes at every position. At the temperature chosen here the attenuation falls off quickly, so it may be suppressing the distillation loss at most positions with the large distillation weight compensating, though that is a guess on my part and a simple diagnostic would settle it either way.

  • The reading of the margin analysis in Section 5.4 goes further than the evidence. Holding up under stricter margins is consistent with the student having learned real structure–property relationships, but the student was trained against the same frozen property functions that define the evaluation while the general-purpose comparators never saw them, and that alone would produce the same pattern. The method also loses at the loosest margin and leads by very little at the intermediate one, so I would be cautious about this analysis on its own terms before reaching for the stronger interpretation.

Questions for authors
  1. Were the ablation variants trained with the same number of updates, the same prompt schedule and the same compute as the full method? The cost table suggests they were not, and if that is so, could you say what differed?
  2. What run-to-run variation did you observe during development? Even a single repeated run of RA-MOPD and Vanilla MOPD, or seeds on a subset of the combinations rather than the full set, would help place the reported margins.
  3. Can you report the distribution of the conflict term over student-visited positions, or just the fraction of positions where the attenuation factor is non-negligible? Either would show whether the token-level mechanism is active in practice.
  4. How sensitive are the results to the conflict temperature? If you have any run with it set an order of magnitude higher, or with the attenuation removed while the routed product target is kept, that would be informative.
  5. The setup promises validity for all methods and Section 5.2 draws a conclusion from the validity of one ablation, but no validity numbers appear in any table, could you add them? Tables 1 and 10 also disagree on whether off-policy fine-tuning uses the verifier.

Two things I would find interesting but would not expect for this submission, and which have not affected my assessment: evaluating on a property whose scoring function was never used in training or routing, which would separate chemical improvement from alignment with the training oracles, and a held-out split that varies property identity and request arity one at a time, since the current one differs in both at once and also contains the only direction-reversed property.

6 comments
AuthorsOpenReview ↗

Response to Q1

Thank you for this suggestion. The supplementary controlled ablations use the same number of updates and prompt scheduling strategy. Computational costs vary across methods, and Table 10 reports the computational costs of the original experiments.

AuthorsOpenReview ↗

Response to Q2

Thank you for this suggestion. Evaluation uses deterministic greedy decoding with H=2, K=1, a single parser, and a fixed oracle stack. Repeated evaluation of the same checkpoint produced identical outputs. Training uses B=4 sampled rollouts per prompt for the group-relative objective, allowing different training seeds to visit different molecular states and produce different parameter updates.

We therefore repeated fusion-stage training with three random seeds under matched configurations. All methods shared the same initialization, frozen teacher pool, update budget, and prompt scheduling strategy, with identical prompt order across methods within each seed. On the original compact500 benchmark, the success rates were as follows. Values are percentages, and SD denotes the sample standard deviation across training seeds. The original reported result (28.00%) comes from a separate training run and is not included in the three-seed statistics (27.20% ± 0.72%).

MethodSeed 17Seed 18Seed 19Mean ± SD
Reliability-Aware Multi-Teacher On-Policy Distillation27.4027.8026.4027.20 ± 0.72
Vanilla Multi-Teacher On-Policy Distillation with reinforcement-learning feedback26.2026.4024.6025.73 ± 0.99
Vanilla Multi-Teacher On-Policy Distillation, distillation only25.4025.6025.4025.47 ± 0.12

The full method outperforms Vanilla Multi-Teacher On-Policy Distillation with reinforcement-learning feedback in all three training seeds.

AuthorsOpenReview ↗

Response to Q3

Thank you for this suggestion. We measured the attenuation factor at student-visited token positions selected for distillation in the supplementary training diagnostic. Across 25,923 answer tokens, at the default conflict temperature τ=0.05\tau=0.05, the proportions are as follows.

Attenuation factorProportion of positions
Non-negligible weight (γ>10−6\gamma>10^{-6})99.65%
Weight reduced by at least 50% (γ≤0.5\gamma\leq0.5)6.23%
Weight reduced by more than 90% (γ<0.1\gamma<0.1)2.95%

Thus, conflict gating preserves supervision at most positions while substantially downweighting a small subset with stronger teacher disagreement.

Reviewer_G5tcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies natural-language-instruction-driven multi-property molecular optimization, aiming to improve all requested properties simultaneously while maintaining molecular validity and sufficient structural similarity to the initial molecule. The authors propose RA-MOPD, which distills six independently trained single-property experts into a single student model. The method dynamically adjusts teacher weights based on their historical reliability and the verified outcomes of their proposed edits at the current molecular state. It then combines the teachers’ token distributions and attenuates distillation where they disagree. Training combines reinforcement learning rewards over complete generation trajectories with distillation restricted to the SMILES span. At inference, the student requires neither experts nor property oracles.

Critical strengths
  1. The method addresses practical challenges in multi-property optimization and is reasonably motivated. Single-property experts may harm other properties while improving their own objectives. This paper assigns weights by examining the specific edits proposed by experts and weakens distillation when teachers’ opinions conflict.
  2. The authors prove that the negative log-normalizer of the fused distribution corresponds to the minimum weighted reverse-KL disagreement, which can be directly used to control distillation strength.
Critical weaknesses
  1. The improvement over the closest baseline is small, and evidence of its stability is insufficient. Across the ten test groups covering both IID and OOD settings, the full RA-MOPD method achieves the best performance in only three groups.
  2. The paper has not sufficiently demonstrated why conflict gating is effective. Teachers’ disagreement over the next token does not necessarily reflect genuine conflicts between molecular properties.
  3. The scope of validation remains relatively limited. The OOD tasks mainly involve unseen property combinations and do not establish generalization to new scaffolds or new targets.
Questions for authors
  1. The paper provides a representative case of resolving property conflicts through multiple rounds of editing, illustrating the role of dynamic teacher weighting. However, this case lacks baseline comparisons on the same input and does not analyze the specific behavior of token-level conflict gating. Therefore, the current case study is insufficient to determine which types of molecules or conflict conditions benefit most from the full method. In addition, please clarify whether the dynamic routing in this case occurs during training or is a post-hoc analysis of a test trajectory.
  2. Distillation is applied only to the SMILES portion, while the preceding reasoning portion is trained only through RL. Could this cause the final generated molecule to be inconsistent with the preceding reasoning, thereby reducing interpretability?
4 comments
AuthorsOpenReview ↗

Response to Q1 (Part II)

Together with the paired trajectory, these results motivate examining a specific state-dependent conflict. An edit improves some objectives but creates a shortfall in another, and the next edit must repair that shortfall without losing the existing gains. A proposal that is useful for one specialist’s objective may consequently be unsuitable for the full conjunction. The collateral-harm term explicitly distinguishes these cases, while state-dependent teacher weights allow the relative emphasis of teacher supervision to change as the molecule evolves.

Molecular-feature analyses identify two groups with positive mean gains against both baselines. We grouped inputs by their initial molecular descriptors and compared methods within each group. The favorable patterns occur among molecules with fewer rotatable bonds and those with a lower fraction of aromatic heavy atoms.

Our working interpretation of these exploratory patterns is that they align with the method’s emphasis on preserving the main structure while coordinating localized edits. Teacher cooperation helps the student select modifications that accommodate several requirements within the structural-similarity constraint.

Dynamic routing is performed online during training. At each student-visited molecular state, the frozen teachers propose candidates, the current teacher weights are computed, and distillation is performed on the student’s actual generated prefixes. At test time, molecule generation is performed by the distilled student alone.

AuthorsOpenReview ↗

Response to Q2

Thank you for this suggestion. We don’t want to use OPD everywhere; we only want to implement it where it matters most. Our design concentrates multi-teacher on-policy distillation on the molecular answer that directly determines the generated structure, while reinforcement learning connects the preceding rationale to the editing outcome. We want to assign only the most critical tasks to the OPD.

To test this design, we added full-response on-policy distillation. The frozen teachers provide next-token distributions on the student’s own generated prefixes. The same reliability- and outcome-aware routing and token-level conflict gating then supervise the entire response, including the preceding rationale, while retaining the original reinforcement-learning objective.

The following supplementary comparison uses the same initialization, teacher pool, update budget, and prompt schedule, with both models evaluated on compact500 using greedy decoding.

MethodConstrained success rate
Reliability-Aware Multi-Teacher On-Policy Distillation with molecular-answer-only distillation28.00%
Reliability-Aware Multi-Teacher On-Policy Distillation with full-response distillation25.00%

Full-response distillation did not improve the task success rate, supporting retention of the original division of supervision.

Our reinforcement-learning reward already includes consistency-related feedback. Beyond property improvement, it checks whether the rationale’s edit claims are supported by actual molecular graph changes and whether the structural entities mentioned are present. Reinforcement-learning updates cover the entire response, so the rationale also receives feedback from these structural checks. On-policy distillation supplies expert guidance for molecular editing, while reinforcement learning connects the editing rationale, structural changes, and optimization outcome.

AuthorsOpenReview ↗

Response to W1

Thank you for this suggestion. We report the original checkpoints’ aggregate performance together with the breakdown between seen and unseen property combinations.

MethodOverall success rateSeen property combinationsUnseen property combinations
Reliability-Aware Multi-Teacher On-Policy Distillation28.00%30.00%26.00%
Vanilla Multi-Teacher On-Policy Distillation with reinforcement-learning feedback26.20%28.80%23.60%
Vanilla Multi-Teacher On-Policy Distillation, distillation only26.00%28.40%23.60%

The full method improves overall success over the two baselines. On unseen property combinations, the improvement is 2.40 percentage points over both. These results show the aggregate benefit of the complete method in both evaluation regimes under the original protocol.

AuthorsOpenReview ↗

Response to W2

The conflict term directly measures incompatibility among teacher probability distributions. In our method, property gains and collateral harm are evaluated by the verifier and used to determine teacher routing weights. Token-level gating then controls the strength of distillation when these teachers disagree on the student's current prefix. These two components address teacher weighting at the property level and supervision strength at the distribution level, respectively.

We examined the effects of attenuation strength and token assignment. The original full-method checkpoint provides the reference, and each added configuration uses one trained checkpoint.

Gating configurationImplementationTask success rate on compact500
Original conflict gatingDetermine attenuation from teacher disagreement at each token.28.00%
Shuffled gatingPreserve the attenuation values within each response and randomly shuffle their token assignments.25.40%
Constant gatingApply a fixed attenuation factor of approximately 0.9265, calibrated from the reference training trace, to every token.25.80%

The original gating achieves 28.00%, compared with 25.40% for shuffled gating and 25.80% for constant gating. These results suggest that the benefit comes not merely from reducing the overall distillation strength, but from applying attenuation at specific positions where routed teachers disagree.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.