- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper propose RA-MOPD, a multi-teacher distillation strategy to fuse multiple-teachers knowledge on lead design to a single unified model. Compared to conventional multi-teacher OPD, the author put the problem in a more dedicated lead optimization setting, where multiple objectives need to be optimized. In this case, multiple teacher's log-prob need to be fused together to get the student signal. However, the authors notice the insufficiency of uniform fusion, where some ineffective teachers' log-prob is given the same weight as the effective one. Hence the authors propose to compute the teacher weight from held-out realiability and weight each token via a product of experts. Under this training, the distillation process can get high success rates.
Critical strengths
The paper identified a critical setting, lead optimization, that is well-tuned for the application OPD. It propose a dedicated solution for handle the multi-target scenario. Compared to off-the-shelve solutions, this one measures a teacher's usefulness at a specific state, which I found is new. More specifically, the usefulness is from a current lead molecule to the student, and gating and weighting technique is a natural subsequent step to utilize such usefulness score.
Critical weaknesses
I have two complaints one on the conceptual novely and one on the empirical results:
-
The novelty is real, but I think this is not really exciting (maybe other reviewers can say more on if this is important). Essentially, one can think of this as a version of soft best-of-N, but we don't know why this is the method and potentially the preferrable one. There is no theoretical justification of why a single greedy sample of teacher satisfy the weighting job, and how that relates to a student distillation target. And it seems not straight forward to adopt the methodology to other domains.
-
For the results, compare table 3 and table 2, Mixed RL + Vanilla MOPD is outperforming every ablated cases. From table 12, this also seems to be the case. This means that any single componenet is infact harmful, and the composition of the three somehow revert the harmfulness, and outperform vanila, this phenomenon is not explained. Since the key hypothesis is that routing and gating are helpful, the ablation actually shows a contradiction.
Questions for authors
- What is the dataset used? There is no mention of the dataset in the paper, just different cases.
4 comments
Response to W1
Thank you for this suggestion. Our motivation is to integrate the capabilities of multiple property specialists into a single student, improving candidate quality under a limited generation budget. Best-of-N draws on the base model’s existing output distribution by generating and scoring more candidates, increasing the opportunity to find a high-quality answer. Its effectiveness depends on the model’s coverage of desirable solutions and the reliability of the scoring function.
For molecular optimization, candidate quality matters because downstream synthesis and experimental evaluation are costly. As a simplified illustration, assuming independent candidates and equal validation costs, increasing the success probability per candidate from 20% to 50% reduces the expected number of evaluations needed to obtain one successful molecule from five to two. Computational screening can further reduce the number of candidates entering wet-lab experiments. This motivates our focus on joint multi-property success under a fixed, single-candidate generation budget.
We therefore train six single-property teachers through specialized reinforcement learning, allowing each teacher to acquire editing strategies for its corresponding property. We then combine their supervision at states visited by the student. Teacher weights depend on the property improvement, collateral harm, and reliability of their current proposals, while token-level conflict gating adjusts supervision strength at positions where teachers disagree. This makes the basis for combining expert knowledge directly inspectable.
A single greedy proposal from each teacher is used to estimate its routing utility at the current state. Each teacher subsequently provides its full next-token distribution conditioned on the student’s actual generated prefix. Although reinforcement learning has been observed to reduce output diversity in some settings, the adequacy of one proposal still requires empirical examination.
We therefore directly tested whether increasing the teacher proposal budget improves performance. The following supplementary matched-control results use compact500 and greedy student evaluation.
| Teacher proposal configuration | Procedure | Task success rate on compact500 |
|---|---|---|
| One greedy proposal per teacher | Score each proposal to compute the current teacher routing weights. | 28.00% |
| Four sampled proposals per teacher | Select each teacher’s highest-utility proposal, then compute the routing weights across teachers. | 26.20% |
Existing multi-teacher on-policy distillation research has explored integrating the capabilities of reinforcement-learning specialists from different domains into a single student. Our work focuses on how to combine teacher knowledge according to the current state and moderate inconsistent supervision when the same molecule must satisfy several potentially conflicting objectives. We see multi-objective coordination as a promising direction for multi-teacher on-policy distillation and welcome the reviewer’s perspective on this research direction and additional ways to evaluate it.
Response to W2
Thank you for this observation. We agree that the ablation results require further explanation.
The three components are interdependent rather than additive. The reliability prior and collateral-harm term jointly determine state-level teacher routing weights (Eq. 6), which subsequently affect both the product-of-experts target and its conflict gate (Eqs. 7–8). Removing either routing component changes the teacher weights and consequently the token-level supervision. In contrast, removing conflict gating preserves the original routing weights but forces the student to follow the fused teacher target even at positions of strong disagreement.
This explains why partial configurations need not outperform uniform weighting. On compact500, Mixed RL + Vanilla MOPD achieves 26.20%, compared with 26.20%, 25.40%, and 25.80% for the three component ablations (without reliability prior, collateral-harm term, and token-level gating, respectively). The complete RA-MOPD achieves 28.00%. These results suggest that the benefit arises from coordinating reliability-aware routing, collateral-harm prevention, and conflict-sensitive supervision, rather than from independently adding their contributions.
We will clarify that our ablations measure each component's contribution within the complete framework, rather than establishing that every partial configuration individually improves upon uniform multi-teacher distillation.
Dear Authors,
Thank you for your response. For W1, I think "single sample" works, but the question was on why. The experiment explain a bit, but how the weighting and distillization target connects is not explained. For W2, I think its just a restate of the phenomon, and no clear explanation. Again, the idea is interesting to me, but the mechanism needs more study. I will keep my current score.
Response to Q1
Thank you for this suggestion! We use the publicly available MuMOInstruct dataset, which covers six molecular properties, including blood–brain barrier permeability, dopamine D2 receptor activity, human intestinal absorption, mutagenicity, penalized logP, and quantitative estimate of drug-likeness. Starting from its official training split, we construct data for four training stages through sample filtering, rationale synthesis, and multi-step trajectory construction.
The main-text evaluation uses a balanced set of 500 inputs, with 50 per combination. The full evaluation set contains 3,905 deduplicated initial-molecule–property-combination pairs.
We will explicitly identify and cite the dataset in the main experimental setup and consolidate the descriptions of the four-stage data construction pipeline and evaluation splits.