- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Learn to Evolve (L2E), and RL algorithm for LLMs doing molecular optimization via editing an initial molecules in multiple turns with evaluator feedback. Their primary contribution is the addition of a weighted local advantage turn that, when combined with a global advantage, defines a turn advantage. They motivate this by examining failure modes in pure outcome-based RL baselines like GRPO and RLOO. Benchmarking is performed against the aforementioned outcome-based RL baselines as well as training-free and evolutionary algorithms. An extensive ablation study is also conducted along with a brief docking oracle optimization experiment at the end.
Critical strengths
The paper is well-organized and throughly presents its motivations, clearly explains the method, and extensively benchmarks against the related literature with sensible ablations. For the most part the results are presented transparently and honestly, and most of the choices the authors make in their evaluation and presentation of the method are well-justified.
Critical weaknesses
The primary weaknesses of this paper are in the results themselves. While the experimental setups all make sense, the performance of the method is not particularly impressive, especially in the multi-objective benchmark setting. I was particularly concerned by the fact that the performance relative to the RL baselines became worse after scaling up the model size, and the extremely poor performance of REINVENT which is usually one of the strongest baselines in these molecular optimization settings. Furthermore, the performance on the docking benchmark at the end is not convincing, and its extremely brief discussion does not really add more than it detracts from the overall picture of the paper.
Questions for authors
Clarification:
- Can you explain further why the "selection score" and "molecular score" are decoupled? I'm not sure I fully understood that in my reading.
- Why use an older Qwen model like 2.5-Instruct?
- Could this method be used to train on trajectories of different lengths? Why or why not?
- I'm a bit unclear on what the green color gradient means in the tables. What are the different shades supposed to indicate?
- Can you discuss further why the weight sweep ends up being non-monotonic?
Suggestions:
- Can you reexamine the REINVENT benchmark that you ran to ensure it's properly calibrated and a fair comparison?
- Can you run multiple training seeds in your benchmarking to not just account for sampling noise? The method being presented is a post-training algorithm, so using multiple training seeds would be a more appropriate significance test
- Can you examine the scaling results on multi-property benchmarks as well?
- Can you provide statistical significance information on the docking benchmark?
10 comments
Response to Reviewer fQve (part 2/5)
W1.1. Performance on MuMOInstruct
W1.1: […] the performance of the method is not particularly impressive, especially in the multi-objective benchmark setting.
Reply: L2E achieves the highest SR and RI on BDP under both seen and unseen instructions, while GraphGA leads on BPQ and RLOO achieves the highest RI on BDQ (Tab. 2). GraphGA’s additional BPQ successes come from molecules with fewer heavy atoms. On jointly successful BDQ searches, RLOO achieves larger relative DRD2 gains, while L2E achieves larger QED gains. These results identify the objective combinations where each method performs best.
L2E improves every requested property together more often than GRPO across all three property combinations and both instruction splits. BDP requires joint improvement in BBBP, DRD2, and penalized LogP.
(1) On BPQ, GraphGA’s higher SR comes entirely from additional successes that reduce heavy-atom count. Across the two instruction splits, L2E and GraphGA each return a successful molecule with unchanged or higher heavy-atom count in 8.7% of all searches. GraphGA achieves higher SR and RI on both splits. Its crossover and mutation operators can replace bulky polar fragments with smaller, less polar ones. Such edits can improve penalized LogP while moving molecular size and polarity toward values favored by QED.
(2) On BDQ, RLOO’s larger relative DRD2 gains give it higher RI when both methods succeed. We pair searches by starting molecule and decoding repeat. Among jointly successful pairs, L2E achieves larger QED gains and higher molecular scores, while RLOO achieves larger relative DRD2 gains. Both methods retain trajectory-level credit; L2E additionally distinguishes edits by their molecular scores. One possible explanation is that L2E gives more local credit to a higher-scoring QED edit than a lower-scoring DRD2 edit within the same trajectory. RLOO assigns both the same trajectory advantage, so a positive advantage can reinforce a large relative DRD2 gain despite a modest molecular score.
W1.2 & S3. Model scaling
W1.2: the performance relative to the RL baselines became worse after scaling up the model size
S3: Can you examine the scaling results on multi-property benchmarks as well?
Reply: L2E’s RI lead over GRPO narrows at 7B because GRPO’s RI increases proportionally more with model size. L2E retains the highest RI on all three single-property tasks, although GRPO and RLOO succeed more often on LogP and MR. We conduct additional 7B MuMOInstruct experiments and find 16.8% higher overall SR and 1.7% higher overall RI for L2E than GRPO.
(1) GRPO’s larger relative RI increase with model size narrows L2E’s lead. From 3B to 7B, GRPO’s RI increases by a larger proportion on every single-property task (Tabs. 1 and 6). L2E starts from higher LogP and MR RI at 3B and improves further at 7B, achieving the highest RI on all three tasks.
| Target | Metric | L2E | GRPO | RLOO |
|---|---|---|---|---|
| QED | SR (%) | 61.3 | 57.7 | 57.3 |
| QED | RI | 0.422 | 0.408 | 0.406 |
| LogP | SR (%) | 84.3 | 89.3 | 93.0 |
| LogP | RI | 19.899 | 8.924 | 10.058 |
| MR | SR (%) | 45.7 | 60.3 | 56.0 |
| MR | RI | 6.055 | 2.431 | 2.603 |
L2E’s relative RI advantage over RLOO increases on LogP and MR with scale, while its SR decreases and falls below both RL baselines. On QED, both SR and RI increase, and L2E leads both baselines.
(2) We conduct additional 7B MuMOInstruct experiments and find higher overall SR and RI for L2E than GRPO. Both methods use Qwen2.5-7B-Instruct with rank-16 LoRA and the seen and unseen instructions in Tab. 2. L2E’s overall SR advantage over GRPO narrows from 69.0% at 3B to 16.8% at 7B; its overall RI is 1.7% higher at 7B.
At 3B, L2E has higher RI than GRPO on BDP and BDQ and lower RI on BPQ under both instruction splits. At 7B, L2E has higher SR in 83.3% of the six settings. Its RI is higher on both BDP splits and BPQ with seen instructions, and lower on both BDQ splits and BPQ with unseen instructions.
| Targets | Instructions | Metric | L2E (7B) | GRPO (7B) |
|---|---|---|---|---|
| BDP | seen | SR (%) | 36.1 | 29.9 |
| BDP | seen | RI | 1.060 | 0.866 |
| BDP | unseen | SR (%) | 26.8 | 21.6 |
| BDP | unseen | RI | 0.509 | 0.476 |
| BDQ | seen | SR (%) | 33.7 | 25.2 |
| BDQ | seen | RI | 0.621 | 0.783 |
| BDQ | unseen | SR (%) | 31.0 | 28.8 |
| BDQ | unseen | RI | 0.316 | 0.323 |
| BPQ | seen | SR (%) | 12.2 | 13.5 |
| BPQ | seen | RI | 0.087 | 0.069 |
| BPQ | unseen | SR (%) | 11.6 | 10.6 |
| BPQ | unseen | RI | 0.053 | 0.076 |
L2E returns valid molecules meeting the similarity threshold in 41.0% of searches, compared with 31.4% for GRPO. Among these molecules, 61.4% from L2E and 68.6% from GRPO improve every scored property. This yields overall SR of 25.2% and 21.6%, respectively. GRPO achieves 14.8% higher RI among successful searches (1.991 vs. 1.735), leaving L2E with a smaller advantage in RI across all searches.
Response to Reviewer fQve (part 3/5)
W1.3 & S1. REINVENT calibration and baseline performance
W1.3: the extremely poor performance of REINVENT which is usually one of the strongest baselines in these molecular optimization settings.
S1: Can you reexamine the REINVENT benchmark that you ran to ensure it's properly calibrated and a fair comparison?
Reply: We conduct additional calibration experiments using L2E’s training reward and find nonzero REINVENT success on all three test targets. REINVENT receives 4,096 adaptation proposals per starting molecule before evaluation. Experience replay further improves SR and RI on every target, while L2E retains higher SR and RI. In the diagnostic, none of REINVENT’s proposals reaches the required similarity threshold, although 91.5% are valid and connected.
(1) We conduct an additional five-proposal diagnostic and find that none of REINVENT’s proposals meets the similarity requirement. Although 91.5% are valid and connected, 0% meet the similarity threshold of 0.4. REINVENT generates molecules without taking the starting molecule as input, whereas L2E receives it in the prompt. REINVENT relies on reward feedback to adapt to each starting molecule.
Appendix A.3 describes the original baseline as using PMO’s default configuration; Tab. 1 reports zero SR and RI on all three targets. For the diagnostic, we retain PMO’s learning rate of 0.0005 and , with batch size 1. We reset the model and optimizer for each starting molecule and verify parameter updates and rewards favoring the requested direction.
(2) We conduct additional calibration experiments using L2E’s training reward and find that REINVENT succeeds on all three test targets. Experience replay improves SR and RI further. We give REINVENT 4,096 adaptation proposals per starting molecule, use PMO’s default batch size of 64, and evaluate five fresh proposals per search. The reward is .
PMO samples stored molecules in proportion to their scores. Our reward can be negative when the similarity penalty exceeds the property reward. With mixed positive and negative scores, PMO’s sampler can produce negative probabilities. The unmodified sampler fails in 100% of searches. We defer replay whenever memory contains negative scores, preserving the scores, retention rule, and replay loss. We use PMO’s memory of 100 examples and 24 replay examples per update. Without replay, updates use only newly generated molecules. Both variants use the same reward and adaptation budget.
We first test the no-replay variant on six starting molecules with three seeds. On LogP and MR, all selected molecules improve the target property, but their highest similarities are only 0.149 and 0.255. Property gains can offset the similarity penalty during adaptation, while evaluation still rejects these molecules below the 0.4 threshold.
| Target | REINVENT SR without replay (%) | REINVENT RI without replay |
|---|---|---|
| QED | 33.3 | 0.2966 |
| LogP | 0.0 | 0.0000 |
| MR | 0.0 | 0.0000 |
| Overall | 11.1 | 0.0989 |
We then compare both variants on the full test set, keeping the development-selected settings fixed.
| Target | REINVENT SR without replay (%) | REINVENT SR with replay (%) | L2E SR (Tab. 1, %) | REINVENT RI without replay | REINVENT RI with replay | L2E RI |
|---|---|---|---|---|---|---|
| QED | 3.67 | 13.33 | 41.0 | 0.01430 | 0.08443 | 0.386 |
| LogP | 0.33 | 6.33 | 100.0 | 0.00873 | 0.42426 | 19.357 |
| MR | 0.67 | 3.00 | 69.7 | 0.00328 | 0.04240 | 4.138 |
Without replay, REINVENT achieves 1.56% overall SR; its QED SR across the three search seeds is 5%, 3%, and 3%. Replay’s largest relative SR increase occurs on LogP: 1,800% over the no-replay variant. L2E retains higher SR and RI on all three targets after calibration and replay.
Q1. Definitions and roles of the molecular score and selection score
Q1: Can you explain further why the "selection score" and "molecular score" are decoupled? […]
Reply: The selection score chooses the output molecule; the molecular score assigns training credit to every proposal. The final best score alone does not distinguish the other proposals’ contributions. L2E therefore uses molecular scores to compare turns within trajectories and cumulative scores across trajectories. Evaluation selects one output from the starting molecule and feasible proposals.
(1) During evaluation, we need one output per starting molecule. Section 2 defines the output as the molecule with the highest selection score among the starting molecule and feasible proposals. A chemically valid proposal with similarity falls below the threshold and is excluded from final selection.
(2) During training, the final best score does not assess the other proposals. We therefore use each turn’s molecular score to assign credit. For the same proposal, we subtract the similarity shortfall from its property score (Sec. 4.1). Two trajectories returning the same molecule can consequently provide different training signals.
Response to Reviewer fQve (part 1/5)
Thank you for the detailed and thoughtful feedback. We have incorporated all additional results, analyses, and clarifications presented below into the revised paper. We summarize our response as follows: