acceptodds
Under review as a conference paper at ICLR 2027

Learning to Evolve Molecules with Local-Global Credit Assignment

Abstract

Large language models (LLMs) optimize molecules through trajectories of successive proposals and evaluator feedback, seeking property improvements under chemical validity and similarity constraints. Outcome-level reinforcement learning trains on these trajectories but assigns the same advantage to every turn, despite differences in molecular quality. We introduce Learning to Evolve (L2E), which trains LLMs through a local-global principle. For trajectories sharing an initial molecule and objectives, global advantage compares cumulative molecular scores across trajectories, while local advantage compares molecular scores within each trajectory. Adding weighted, centered local advantage distinguishes turns by molecular score while preserving global advantage as their mean turn advantage. Our analysis characterizes the local gradient contribution and gives sufficient conditions for improving the best feasible score over a global-only update. On S2-Bench, L2E improves molar refractivity in 69.7% of cases versus 45.3% for the strongest outcome-level RL baseline under the same proposal budget, while mean relative improvement reaches 1.66×RLOO’s. On MuMOInstruct, gains in joint BBBP, DRD2, and penalized LogP optimization extend to unseen instructions. L2E also benefits from additional refinement turns.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Large Language Model, Molecular Optimization, LLM Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fQveOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Learn to Evolve (L2E), and RL algorithm for LLMs doing molecular optimization via editing an initial molecules in multiple turns with evaluator feedback. Their primary contribution is the addition of a weighted local advantage turn that, when combined with a global advantage, defines a turn advantage. They motivate this by examining failure modes in pure outcome-based RL baselines like GRPO and RLOO. Benchmarking is performed against the aforementioned outcome-based RL baselines as well as training-free and evolutionary algorithms. An extensive ablation study is also conducted along with a brief docking oracle optimization experiment at the end.

Critical strengths

The paper is well-organized and throughly presents its motivations, clearly explains the method, and extensively benchmarks against the related literature with sensible ablations. For the most part the results are presented transparently and honestly, and most of the choices the authors make in their evaluation and presentation of the method are well-justified.

Critical weaknesses

The primary weaknesses of this paper are in the results themselves. While the experimental setups all make sense, the performance of the method is not particularly impressive, especially in the multi-objective benchmark setting. I was particularly concerned by the fact that the performance relative to the RL baselines became worse after scaling up the model size, and the extremely poor performance of REINVENT which is usually one of the strongest baselines in these molecular optimization settings. Furthermore, the performance on the docking benchmark at the end is not convincing, and its extremely brief discussion does not really add more than it detracts from the overall picture of the paper.

Questions for authors

Clarification:

  • Can you explain further why the "selection score" and "molecular score" are decoupled? I'm not sure I fully understood that in my reading.
  • Why use an older Qwen model like 2.5-Instruct?
  • Could this method be used to train on trajectories of different lengths? Why or why not?
  • I'm a bit unclear on what the green color gradient means in the tables. What are the different shades supposed to indicate?
  • Can you discuss further why the weight sweep ends up being non-monotonic?

Suggestions:

  • Can you reexamine the REINVENT benchmark that you ran to ensure it's properly calibrated and a fair comparison?
  • Can you run multiple training seeds in your benchmarking to not just account for sampling noise? The method being presented is a post-training algorithm, so using multiple training seeds would be a more appropriate significance test
  • Can you examine the scaling results on multi-property benchmarks as well?
  • Can you provide statistical significance information on the docking benchmark?
10 comments
AuthorsOpenReview ↗

Response to Reviewer fQve (part 1/5)

Thank you for the detailed and thoughtful feedback. We have incorporated all additional results, analyses, and clarifications presented below into the revised paper. We summarize our response as follows:

  • W1.1: L2E exceeds GRPO in joint SR across all MuMOInstruct tasks and instruction splits. We examine GraphGA’s advantage on BPQ and RLOO’s larger RI on BDQ.
  • W1.2 and S3: We conduct additional 7B MuMOInstruct experiments and find higher overall SR and RI for L2E than GRPO. We also explain why L2E’s single-property RI lead over GRPO narrows with scale.
  • W1.3 and S1: We conduct additional REINVENT calibration experiments and find that replay improves SR and RI on every target. L2E retains higher SR and RI.
  • W1.4 and S4: Our ligand-paired analysis favors L2E over GRPO for all three starting ligands in Fig. 8. GRPO has the lower mean docking score over the first five training updates.
  • Q1: We distinguish training credit from evaluation selection over the starting molecule and feasible proposals.
  • Q2: We explain the backbone choice. We conduct additional Qwen3.5-4B experiments and find higher SR and RI with L2E than Base on all three properties.
  • Q3: We conduct additional mixed-length training experiments and find that mean scoring improves RI over cumulative scoring on all three properties. Fixed-length models retain higher SR and RI on MR and LogP than cumulative-score mixed-length training.
  • Q4: We clarify the color scales and the separate boldface rankings for seen and unseen instructions.
  • Q5: Our MR analysis shows how a large local weight can override positive global advantage, providing a possible explanation for the non-monotonic SR.
  • S2: We conduct additional GRPO and L2E training experiments and find higher RI for L2E on all three properties. Mean SR is higher on LogP, similar on MR, and lower on QED.
AuthorsOpenReview ↗

Response to Reviewer fQve (part 2/5)

W1.1. Performance on MuMOInstruct

W1.1: […] the performance of the method is not particularly impressive, especially in the multi-objective benchmark setting.

Reply: L2E achieves the highest SR and RI on BDP under both seen and unseen instructions, while GraphGA leads on BPQ and RLOO achieves the highest RI on BDQ (Tab. 2). GraphGA’s additional BPQ successes come from molecules with fewer heavy atoms. On jointly successful BDQ searches, RLOO achieves larger relative DRD2 gains, while L2E achieves larger QED gains. These results identify the objective combinations where each method performs best.

L2E improves every requested property together more often than GRPO across all three property combinations and both instruction splits. BDP requires joint improvement in BBBP, DRD2, and penalized LogP.

(1) On BPQ, GraphGA’s higher SR comes entirely from additional successes that reduce heavy-atom count. Across the two instruction splits, L2E and GraphGA each return a successful molecule with unchanged or higher heavy-atom count in 8.7% of all searches. GraphGA achieves higher SR and RI on both splits. Its crossover and mutation operators can replace bulky polar fragments with smaller, less polar ones. Such edits can improve penalized LogP while moving molecular size and polarity toward values favored by QED.

(2) On BDQ, RLOO’s larger relative DRD2 gains give it higher RI when both methods succeed. We pair searches by starting molecule and decoding repeat. Among jointly successful pairs, L2E achieves larger QED gains and higher molecular scores, while RLOO achieves larger relative DRD2 gains. Both methods retain trajectory-level credit; L2E additionally distinguishes edits by their molecular scores. One possible explanation is that L2E gives more local credit to a higher-scoring QED edit than a lower-scoring DRD2 edit within the same trajectory. RLOO assigns both the same trajectory advantage, so a positive advantage can reinforce a large relative DRD2 gain despite a modest molecular score.

W1.2 & S3. Model scaling

W1.2: the performance relative to the RL baselines became worse after scaling up the model size

S3: Can you examine the scaling results on multi-property benchmarks as well?

Reply: L2E’s RI lead over GRPO narrows at 7B because GRPO’s RI increases proportionally more with model size. L2E retains the highest RI on all three single-property tasks, although GRPO and RLOO succeed more often on LogP and MR. We conduct additional 7B MuMOInstruct experiments and find 16.8% higher overall SR and 1.7% higher overall RI for L2E than GRPO.

(1) GRPO’s larger relative RI increase with model size narrows L2E’s lead. From 3B to 7B, GRPO’s RI increases by a larger proportion on every single-property task (Tabs. 1 and 6). L2E starts from higher LogP and MR RI at 3B and improves further at 7B, achieving the highest RI on all three tasks.

TargetMetricL2EGRPORLOO
QEDSR (%)61.357.757.3
QEDRI0.4220.4080.406
LogPSR (%)84.389.393.0
LogPRI19.8998.92410.058
MRSR (%)45.760.356.0
MRRI6.0552.4312.603

L2E’s relative RI advantage over RLOO increases on LogP and MR with scale, while its SR decreases and falls below both RL baselines. On QED, both SR and RI increase, and L2E leads both baselines.

(2) We conduct additional 7B MuMOInstruct experiments and find higher overall SR and RI for L2E than GRPO. Both methods use Qwen2.5-7B-Instruct with rank-16 LoRA and the seen and unseen instructions in Tab. 2. L2E’s overall SR advantage over GRPO narrows from 69.0% at 3B to 16.8% at 7B; its overall RI is 1.7% higher at 7B.

At 3B, L2E has higher RI than GRPO on BDP and BDQ and lower RI on BPQ under both instruction splits. At 7B, L2E has higher SR in 83.3% of the six settings. Its RI is higher on both BDP splits and BPQ with seen instructions, and lower on both BDQ splits and BPQ with unseen instructions.

TargetsInstructionsMetricL2E (7B)GRPO (7B)
BDPseenSR (%)36.129.9
BDPseenRI1.0600.866
BDPunseenSR (%)26.821.6
BDPunseenRI0.5090.476
BDQseenSR (%)33.725.2
BDQseenRI0.6210.783
BDQunseenSR (%)31.028.8
BDQunseenRI0.3160.323
BPQseenSR (%)12.213.5
BPQseenRI0.0870.069
BPQunseenSR (%)11.610.6
BPQunseenRI0.0530.076

L2E returns valid molecules meeting the similarity threshold in 41.0% of searches, compared with 31.4% for GRPO. Among these molecules, 61.4% from L2E and 68.6% from GRPO improve every scored property. This yields overall SR of 25.2% and 21.6%, respectively. GRPO achieves 14.8% higher RI among successful searches (1.991 vs. 1.735), leaving L2E with a smaller advantage in RI across all searches.

AuthorsOpenReview ↗

Response to Reviewer fQve (part 3/5)

W1.3 & S1. REINVENT calibration and baseline performance

W1.3: the extremely poor performance of REINVENT which is usually one of the strongest baselines in these molecular optimization settings.

S1: Can you reexamine the REINVENT benchmark that you ran to ensure it's properly calibrated and a fair comparison?

Reply: We conduct additional calibration experiments using L2E’s training reward and find nonzero REINVENT success on all three test targets. REINVENT receives 4,096 adaptation proposals per starting molecule before evaluation. Experience replay further improves SR and RI on every target, while L2E retains higher SR and RI. In the diagnostic, none of REINVENT’s proposals reaches the required similarity threshold, although 91.5% are valid and connected.

(1) We conduct an additional five-proposal diagnostic and find that none of REINVENT’s proposals meets the similarity requirement. Although 91.5% are valid and connected, 0% meet the similarity threshold of 0.4. REINVENT generates molecules without taking the starting molecule as input, whereas L2E receives it in the prompt. REINVENT relies on reward feedback to adapt to each starting molecule.

Appendix A.3 describes the original baseline as using PMO’s default configuration; Tab. 1 reports zero SR and RI on all three targets. For the diagnostic, we retain PMO’s learning rate of 0.0005 and σ=500\sigma=500, with batch size 1. We reset the model and optimizer for each starting molecule and verify parameter updates and rewards favoring the requested direction.

(2) We conduct additional calibration experiments using L2E’s training reward and find that REINVENT succeeds on all three test targets. Experience replay improves SR and RI further. We give REINVENT 4,096 adaptation proposals per starting molecule, use PMO’s default batch size of 64, and evaluate five fresh proposals per search. The reward is r=max⁡(0,RI)−max⁡(0,0.4−Sim)r=\max(0,\mathrm{RI})-\max(0,0.4-\mathrm{Sim}).

PMO samples stored molecules in proportion to their scores. Our reward can be negative when the similarity penalty exceeds the property reward. With mixed positive and negative scores, PMO’s sampler can produce negative probabilities. The unmodified sampler fails in 100% of searches. We defer replay whenever memory contains negative scores, preserving the scores, retention rule, and replay loss. We use PMO’s memory of 100 examples and 24 replay examples per update. Without replay, updates use only newly generated molecules. Both variants use the same reward and adaptation budget.

We first test the no-replay variant on six starting molecules with three seeds. On LogP and MR, all selected molecules improve the target property, but their highest similarities are only 0.149 and 0.255. Property gains can offset the similarity penalty during adaptation, while evaluation still rejects these molecules below the 0.4 threshold.

TargetREINVENT SR without replay (%)REINVENT RI without replay
QED33.30.2966
LogP0.00.0000
MR0.00.0000
Overall11.10.0989

We then compare both variants on the full test set, keeping the development-selected settings fixed.

TargetREINVENT SR without replay (%)REINVENT SR with replay (%)L2E SR (Tab. 1, %)REINVENT RI without replayREINVENT RI with replayL2E RI
QED3.6713.3341.00.014300.084430.386
LogP0.336.33100.00.008730.4242619.357
MR0.673.0069.70.003280.042404.138

Without replay, REINVENT achieves 1.56% overall SR; its QED SR across the three search seeds is 5%, 3%, and 3%. Replay’s largest relative SR increase occurs on LogP: 1,800% over the no-replay variant. L2E retains higher SR and RI on all three targets after calibration and replay.

Q1. Definitions and roles of the molecular score and selection score

Q1: Can you explain further why the "selection score" and "molecular score" are decoupled? […]

Reply: The selection score chooses the output molecule; the molecular score assigns training credit to every proposal. The final best score alone does not distinguish the other proposals’ contributions. L2E therefore uses molecular scores to compare turns within trajectories and cumulative scores across trajectories. Evaluation selects one output from the starting molecule and feasible proposals.

(1) During evaluation, we need one output per starting molecule. Section 2 defines the output as the molecule with the highest selection score among the starting molecule and feasible proposals. A chemically valid proposal with similarity 0.350.35 falls below the 0.40.4 threshold and is excluded from final selection.

(2) During training, the final best score does not assess the other proposals. We therefore use each turn’s molecular score to assign credit. For the same proposal, we subtract the 0.050.05 similarity shortfall from its property score (Sec. 4.1). Two trajectories returning the same molecule can consequently provide different training signals.

Reviewer_unQjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper studies multi-turn molecular optimization with an LLM, where a seed/lead molecule was iteratively refined to have more desirable chemical property while remaining similar to the orignial molecule. Like agentic setting, step-wise credit assignement is critical. Especially in this application, the author found that molecules proposed in the same trajectory has vastly different behavior. The paper observe that naive RL techniques like GRPO and RLOO where entire trajectory share the advantage would fail, and propose a trajectory-level credit normalization technique to incorporate this information. Combined with global score, the method achieve promising perfomance on molecule optimizations tasks.

Critical strengths

The central novelty of the paper is simplistic and clear. It also leads to several advantage like no preference pair, no critic, etc. The author also has theoretical results to justify the intuition that a local credit nudge leads to actual gain.

The non-monotone increase, and validation vs property conflict observation are helpful for the related and future research.

Critical weaknesses
  • The author propose a turn-level credit assignment technique, but it is not clear why this is the best approach, clear advantage over others is not shown. Especially when turn level crediet assignment is not new, without study of the advantage this works appears an application of turn-level credit assignment to molecule optimization domain. To be more specific, HCAPO already does within trajectory normalization [1], others [2,3] have turn-level credit assignment using different tools. And at the current shape, the work appear as a application specific to chemistry case where each turn outputs similar things with verifiable credit.

[1] Tan, Hui-Ze, et al. "Hindsight credit assignment for long-horizon llm agents." arXiv preprint arXiv:2603.08754 (2026). [2] Feng, Lang, et al. "Group-in-group policy optimization for llm agent training." Advances in Neural Information Processing Systems 38 (2026): 46375-46408. [3] Zhang, Naifan, et al. "MICA: Multi-granularity Intertemporal Credit Assignment for Long-Horizon Emotional Support Dialogue." arXiv preprint arXiv:2603.06194 (2026).

Questions for authors
  • I am curious what would happen if you just run per-turn RL, break at a certain turn, and do GRPO? This is a simple alternative if we are to solve the problems observed at the beginning of the paper.

  • Looks like the local-only train is still effective, just causing high dissimilarity, how to understand this? Essentially, if you make the search space around the lead molecule big enough, you can of course increase the target value, does that say this work just somehow put on some regularization? What happen if you simply do localized credit with more regularization?

7 comments
AuthorsOpenReview ↗

Response to Reviewer unQj (part 1/5)

Thank you for the detailed and thoughtful feedback. We have incorporated all additional results, analyses, and clarifications presented below into the revised paper. We summarize our response as follows:

  • To respond to W1: We compare how L2E, HCAPO, GiGPO, and MICA assign credit to individual responses. L2E computes turn advantages directly from molecular scores, without matching identical states across trajectories or using an additional language model to score responses. L2E achieved higher relative improvement (RI) than GRPO and RLOO on QED, LogP, and molar refractivity (MR) in single-property optimization (Tab. 1).
  • To respond to Q1: We conducted an additional per-turn GRPO training experiment to test whether comparing alternative responses from the same history is sufficient. We sampled the alternatives at the same turn from the same history of molecules and evaluator feedback. Under a five-proposal budget, L2E achieved higher RI than per-turn GRPO on all three properties, although per-turn GRPO succeeded more often on QED. L2E’s local advantage distinguishes molecular scores within each trajectory, and its global advantage compares cumulative molecular scores across trajectories.
  • To respond to Q2: We conducted two additional training experiments with local-only variants to test a stronger KL penalty and a stricter similarity penalty. With the stricter similarity penalty, the local-only variant achieved a higher success rate (SR) on MR and lower MR RI than L2E. Successful outputs whose heavy-atom count increased by more than 100% relative to the initial molecule accounted for 96.0% of this variant’s total MR RI.

W1. Contribution of L2E’s local-global credit assignment to molecular optimization

W1: The author propose a turn-level credit assignment technique, but it is not clear why this is the best approach, clear advantage over others is not shown. […] To be more specific, HCAPO already does within trajectory normalization [1], others [2,3] have turn-level credit assignment using different tools. And at the current shape, the work appear as a application specific to chemistry case where each turn outputs similar things with verifiable credit.

Reply: We designed L2E for molecular optimization, where the evaluator returns feedback after each proposal. The local-global principle could also apply to other iterative optimization tasks that provide verifiable scores for intermediate candidates. L2E uses molecular scores to distinguish individual turns while retaining an assessment of the complete trajectory. Below, we explain how L2E assigns turn advantages, how this differs from prior methods, and what the molecular optimization results show.

(1) L2E assigns turn advantages by comparing molecular scores within and across trajectories. The evaluator checks chemical validity and structural similarity and evaluates the target properties. L2E uses these molecular scores directly, without an additional language model to score responses or a requirement that the same state recur across trajectories.

  • Local advantage. Property gains can reverse across turns, and even a successful trajectory can include proposals rejected by the evaluator. Outcome-level RL assigns the same advantage to every turn. L2E standardizes molecular scores within each trajectory, giving higher-scoring proposals a larger local advantage.
  • Global advantage. A proposal can score higher than the other proposals in a trajectory whose cumulative molecular score is low. L2E standardizes cumulative molecular scores across trajectories with the same initial molecule and instruction. The local advantage has zero mean within each trajectory. Adding weighted local advantage therefore preserves global advantage as the mean turn advantage for any local weight.
  • Molecular optimization results. L2E exceeds GRPO and RLOO in SR and RI on QED, LogP, and MR (Tab. 1). In multi-property optimization, L2E also exceeds both baselines on both metrics for joint optimization of BBBP, DRD2, and penalized LogP (BDP), under both seen and unseen instructions (Tab. 2).
AuthorsOpenReview ↗

Response to Reviewer unQj (part 2/5)

(2) The methods differ in their local learning signals and requirements for credit estimation. All four methods support multi-turn learning and fine-grained credit assignment. The table compares their signal sources and scoring requirements.

Method propertyHCAPO [1]GiGPO [2]MICA [3]L2E
Multi-turn learning✓✓✓✓
Fine-grained credit assignment✓✓✓✓
State-matching independence✓×✓✓
Model-free credit estimation×✓×✓
Local learning signalHindsight Q-valuesSame-state returnsRewards and returnsMolecular scores

✓/× indicate whether the original method has the listed property. Here, model-free refers to credit estimation without LLM-based response scoring.

HCAPO and L2E both use within-trajectory normalization. HCAPO uses action probabilities conditioned on the final state to estimate hindsight credit. L2E uses the evaluator’s molecular score for each proposal to compute local advantage.

(3) L2E improves single- and multi-property molecular optimization under a five-proposal budget.

  • For single-property optimization (Tab. 1), L2E achieves 93.9% higher RI on LogP than RLOO. On MR, L2E achieves 69.9% higher SR and 65.8% higher RI than RLOO. L2E also exceeds GRPO on both metrics across all three properties. On QED, L2E’s RI is 2.3% higher than RLOO’s. Under the same proposal budget, L2E therefore improves the target property more often and achieves larger average relative improvements, particularly on LogP and MR.
  • For multi-property optimization (Tab. 2), L2E achieves 30.6% higher SR and 257.7% higher RI than GRPO on BDP with unseen instructions. The corresponding gains over RLOO are 36.2% and 330.6%. L2E also exceeds both baselines on both metrics for BDP with seen instructions. The ranking changes for the other objective combinations. Among L2E, GRPO, and RLOO, RLOO has the highest SR and RI on BDQ. On BPQ, GRPO has the highest RI of the three, and RLOO has the highest SR. These rankings hold for both seen and unseen instructions. Across all methods in Tab. 2, GraphGA has the highest BPQ SR and RI on both instruction splits. L2E’s gains on BDP thus extend to unseen instructions for that objective combination.

In summary, L2E gives higher-scoring proposals larger turn advantages while preserving global advantage as their trajectory mean. It requires no LLM scoring or state matching. L2E achieved higher RI than GRPO and RLOO on all three single-property tasks, with the largest gains on LogP and MR. Its advantage on BDP extended to unseen instructions.

AuthorsOpenReview ↗

Response to Reviewer unQj (part 3/5)

Q1. Comparison of per-turn GRPO and L2E

Q1: I am curious what would happen if you just run per-turn RL, break at a certain turn, and do GRPO? […]

Reply: We conducted additional experiments and show that per-turn GRPO achieved lower RI than L2E on MR, QED, and LogP. A response can receive a positive advantage under per-turn GRPO when its molecular score exceeds the mean for the sampled responses, even if all its molecular scores are low. With L2E, a trajectory whose cumulative molecular score is below the group mean has a negative global advantage. This reduces the turn advantage even for a proposal with positive local advantage and may help explain L2E’s higher RI.

We chose the turn in advance and sampled four alternative responses using the same initial molecule, instruction, prior responses, and evaluator feedback. We standardized the molecular scores of the resulting proposals across these responses and applied the GRPO update only to the generated tokens in each response. We refer to the GRPO baseline in Tab. 1 as trajectory-level GRPO because it assigns the same advantage to every turn in a trajectory. We evaluated all three methods on the same initial molecules following the setting as in Tab.1.

TaskMetricPer-turn GRPOTrajectory-level GRPOL2E
MRSR ↑\uparrow50.3%45.3%69.7%
MRRI ↑\uparrow0.3740.5484.138
QEDSR ↑\uparrow45.0%35.0%41.0%
QEDRI ↑\uparrow0.1400.1840.386
LogPSR ↑\uparrow66.0%73.7%100.0%
LogPRI ↑\uparrow1.7792.48119.357

Overall, per-turn GRPO succeeded more often than trajectory-level GRPO on QED and MR, and more often than L2E on QED. Under the same five-proposal budget, however, per-turn GRPO achieved lower RI than both trajectory-level GRPO and L2E on every property. L2E achieved the largest average improvements on all three.

Reviewer_3dSqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Learning to Evolve (L2E), a reinforcement learning method for training language models to optimize molecules through successive proposals and evaluator feedback. For each initial molecule and instruction, L2E samples multiple trajectories and combines two signals: a global advantage obtained by standardizing cumulative molecular scores across trajectories, and a local advantage obtained by standardizing individual molecular scores within each trajectory. The resulting advantage weights each response in a clipped policy objective. Centering the local component preserves the global advantage as the mean turn advantage.

The contributions comprise an analysis of optimization trajectories, the proposed advantage construction, a score–gradient covariance identity, and a conditional theorem establishing improvement over a global-only update when the local direction aligns with the final-selection objective. Evaluation covers single-property optimization on S²-Bench and multi-property optimization on MuMOInstruct, with component and reward ablations, two model sizes, additional refinement turns, and supplementary docking analysis. The strongest reported improvements concern LogP and molar refractivity, with additional gains on the BBBP/DRD2/penalized-LogP combination. Results are more mixed on QED and the other multi-property combinations.

Critical strengths
  1. L2E adds turn-specific learning signals using scores already obtained during trajectory generation, without requiring a learned critic or additional evaluator calls for credit estimation. Equation (4) makes the modification easy to understand and implement. Proposition 4.1 and Appendix B.2 precisely characterize its effect through a score–gradient covariance term, while the appendix explains important boundary conditions rather than claiming unconditional improvement. This combination of implementation simplicity and analytical transparency makes the method potentially useful for evaluator-guided optimization.

  2. Under the common five-proposal evaluation budget, L2E improves LogP relative improvement from 9.984 to 19.357 and MR relative improvement from 2.496 to 4.138 compared with RLOO; MR success rate also increases from 0.410 to 0.697 (Table 1). The relative-improvement advantages on LogP and MR persist with the 7B backbone and within all three initial-property tertiles (Table 6 and Figure 9). The tertile analysis strengthens the evidence that the gains are not explained solely by a concentration of inputs with small initial property values. The BDP results additionally demonstrate useful transfer to unseen instructions over the same property combination (Table 2 and Appendix A.3). Although these benefits are task dependent, their magnitude warrants serious consideration.

Critical weaknesses
  1. At (\omega=0), Equations (4)–(5) use a standardized cumulative trajectory-score advantage shared across turns. However, the global-only ablation in Table 4 performs substantially better than the GRPO baseline in Table 1.

There may be legitimate implementation or training differences, but the manuscript does not identify them. This matters because the controlled addition of local credit raises MR success from 0.650 to 0.697, substantially less than the headline comparison from GRPO’s 0.453 to 0.697. On QED, adding local credit reduces both success rate and RI relative to global-only. The ablation therefore supports a smaller and more task-dependent contribution than the main baseline comparison alone suggests.

A related ambiguity affects the closest molecular optimization comparator, POLO. Its original formulation uses property-change rewards and combines trajectory optimization with preferences over intermediate molecules. Appendix C.2 does not specify the reward and preference-learning configuration used in the reproduction. This omission is consequential: replacing L2E’s absolute-score reward with previous-state differences reduces LogP RI from 19.357 to 11.207 (Table 5), close to POLO’s 11.268 (Table 3). These numbers do not establish equivalence between the methods, but they make a reward-matched comparison essential to attributing the reported margin to credit assignment.

  1. The evidence does not yet establish when the combined rule is preferable to its simpler variants under the relevant molecular-quality criteria.

Table 4 presents a substantial tradeoff. Local-only achieves higher MR success and RI than full L2E—0.910/13.135 versus 0.697/4.138—and much higher LogP RI. Full L2E retains greater mean similarity, which is a meaningful benefit. The authors also reasonably interpret the local-only results in light of extreme molecular growth documented in Appendix D.3. However, this means that the argument for combining the two components depends on molecular-quality considerations that are not consistently evaluated across the variants under the main protocol.

Figure 10 is a useful step toward addressing this issue: it adds a single-component requirement and still shows advantages for L2E over GRPO and RLOO. However, it uses 50 proposals instead of the main five-proposal budget, omits the global-only and local-only ablations, and reports three evaluation repeats for L2E versus one for the baselines. It therefore does not resolve whether the combination offers the best tradeoff under a common, explicitly specified quality criterion at the main budget.

This concern directly affects the interpretation of the central ablation. The paper should substantiate its reason for preferring full L2E through a consistent comparison of successful, selected candidates, including the structural characteristics used to diagnose failure. Such evidence could establish a compelling advantage even where the unconstrained benchmark metrics favor local-only.

Questions for authors
  1. Clarification: What accounts for the difference between GRPO and global-only?

Please identify the differences between Table 1’s GRPO baseline and Table 4’s global-only ablation in reward definition, advantage normalization, token/turn loss reduction, optimization settings, and checkpoint selection. Which comparison changes only the proposed local term? Explaining the large performance gap is necessary to determine how much of the headline improvement is attributable to L2E’s credit construction.

  1. Clarification: Were the POLO and L2E reward definitions and training conditions matched?

What reward, preference weight, reference-policy initialization, and pair-selection settings were used for POLO? Were training data and training budgets matched? Given the reward sensitivity in Table 5, does the reported advantage over POLO persist when the reward definition is held constant?

  1. Clarification: What are the precise held-out evaluation and output-selection protocols?

Please provide the training, validation, and test instance counts and the overlap rules for initial molecules and instructions, together with total training updates and checkpoint-selection criteria. Please also specify the concrete property transforms (g_j), selection score, and tie rules for each evaluation. Section 2 states that the evaluation block fixes these rules, but I could not locate their complete specification. These details determine both the scope of generalization and the meaning of the multi-property results.

  1. Additional evidence for the current quality claim: How do the variants compare under the same explicit quality criteria at five proposals?

Can full L2E, global-only, local-only, and the principal RL baselines be compared at the main five-proposal budget using the same single-component requirement as Figure 10? Reporting success rate, RI, and size/similarity distributions for the selected candidates would directly test the explanation offered for preferring the combined rule. The existing 50-proposal results provide encouraging evidence but do not answer this comparison.

  1. Additional evidence for the credit-assignment interpretation: What supports the value of this local rule for sequential decisions?

What evidence distinguishes the benefit of within-trajectory immediate-score weighting from that of a conventional turn-specific return estimator? A matched reward-to-go comparison, such as the TRLOO approach discussed in Appendix E.2, would help establish the contribution. Alternatively, a diagnostic related to the target-alignment quantity in Equation (23) could support the proposed mechanism. This is particularly relevant because Appendix B.4 shows that immediate-score credit can oppose beneficial delayed effects.

9 comments
AuthorsOpenReview ↗

Response to Reviewer 3dSq (part 1/5)

Thank you for the detailed and thoughtful feedback. We have incorporated all additional results, analyses, and clarifications presented below into the revised paper. We summarize our response as follows:

  • To respond to W1 and Q1: We compared the training configurations and examined variation across independently trained policies. We hypothesize that instability in RL training contributes to the GRPO/global-only performance gap. We conduct additional GRPO and L2E training experiments and find higher relative improvement (RI) for L2E on all three properties. In Tab. 4, adding local credit increased the MR success rate (SR) by 7.2% relative to global-only L2E and lowered both QED success and RI.
  • To respond to W2 and Q4: We conducted additional evaluations of global-only and local-only at five and 50 proposals to compare molecular quality with L2E and the RL baselines. We report the results using the single-component requirement in Fig. 10 and examine the sizes of successful molecules. At 50 proposals per start, L2E achieved higher success rates and RI than GRPO and RLOO on LogP and MR. At five proposals, global-only had the highest SR on both tasks, local-only had the highest LogP RI, and L2E had the highest MR RI.
  • To respond to W1 and Q2: We conducted additional training experiments with POLO and L2E after matching the absolute molecular score, data, and training budget. We describe POLO’s preference objective and test whether L2E’s SR advantage persists. Following Tab. 1’s evaluation setting, we found that L2E achieved higher success rates than POLO on QED, MR, and LogP.
  • To respond to Q3: We give the split sizes and trace how a five-proposal search yields one reported outcome. In each benchmark, 0% of test starting molecules appeared in training.
  • To respond to Q5: We conducted an additional alignment analysis comparing L2E local credit with TRLOO on the same trajectories. L2E local credit aligned more closely with final selection on average for MR. TRLOO aligned more closely in 100% of QED batches. On LogP, TRLOO had a slightly higher mean signed inner product and L2E local credit a slightly higher mean cosine similarity.

W1 / Q1. GRPO and global-only L2E: training settings and the local-credit comparison

W1: […] However, the global-only ablation in Table 4 performs substantially better than the GRPO baseline in Table 1. […] This matters because the controlled addition of local credit raises MR success from 0.650 to 0.697, substantially less than the headline comparison from GRPO's 0.453 to 0.697. On QED, adding local credit reduces both success rate and RI relative to global-only. […]

Q1: […] Please identify the differences between Table 1's GRPO baseline and Table 4's global-only ablation in reward definition, advantage normalization, token/turn loss reduction, optimization settings, and checkpoint selection. Which comparison changes only the proposed local term? […]

Reply: GRPO and global-only L2E share the global advantage rules. Their separately trained policies could yield different results. Tab. 4 isolates local credit. (1) Both runs used the same reward, global advantage, and main optimization settings.

  • Both methods use the cumulative molecular score over the given multi-turn budget, summing the molecular scores defined in Eq. (3).
  • Both methods use within-prompt normalization. Eq. (4) centers each trajectory’s cumulative molecular score on its group mean and divides by the population standard deviation plus a stabilizer. GRPO used the sample standard deviation, as specified for global-only. At ω=0\omega=0, every turn receives the trajectory’s global advantage. In the GRPO run, an invalid response format incurred a 0.1 deduction before normalization.
  • The GRPO run averaged the loss over generated tokens within each microbatch. The global-only training configuration has the same default. Eq. (5) averages tokens within each response, then weights turns equally.
  • Both methods use Qwen2.5-3B-Instruct, 128 prompts per batch, 16 five-turn trajectories per prompt, a learning rate of 10−610^{-6} and a KL coefficient of 10−310^{-3}.
  • We evaluated both policies at the end of training.
AuthorsOpenReview ↗

Response to Reviewer 3dSq (part 2/5)

(2) We hypothesize that instability in RL training contributes to this performance gap. Training involves stochastic trajectory sampling, which could change the rewards, advantages, and subsequent policy updates. To examine variation, we conduct additional training with multiple seeds for GRPO and L2E. MR showed substantial variation across independently trained policies for both methods. L2E achieved higher RI on all three tasks.

Property and metricGRPOL2E
MR success (%)64.2±25.764.2 \pm 25.764.5±20.564.5 \pm 20.5
MR RI0.376±0.2430.376 \pm 0.2432.502±2.3142.502 \pm 2.314
LogP success (%)82.8±4.082.8 \pm 4.094.0±8.594.0 \pm 8.5
LogP RI1.768±1.0071.768 \pm 1.00712.622±9.52512.622 \pm 9.525
QED success (%)45.0±11.845.0 \pm 11.840.3±0.940.3 \pm 0.9
QED RI0.154±0.0430.154 \pm 0.0430.340±0.0660.340 \pm 0.066

(3) Tab. 4 isolates the local advantage. Both variants use the same global advantage. Raising the local weight ω\omega from 0 to 1 adds the local advantage in Eq. (4). Relative to the global-only variant, L2E increased the MR success rate by 7.2% but lowered both the success rate and RI on QED.

In summary, we hypothesize that instability in RL training contributes to the GRPO/global-only performance gap. The results show substantial variation on MR, and L2E achieved higher RI than GRPO on all three tasks. Tab. 4 isolates the effect of adding local credit, which increased MR success but lowered QED success and RI.

W2 / Q4. Performance comparison of L2E and its variants based on molecular quality

W2: […] Local-only achieves higher MR success and RI than full L2E-0.910/13.135 versus 0.697/4.138-and much higher LogP RI. Full L2E retains greater mean similarity, which is a meaningful benefit. […] Figure 10 is a useful step toward addressing this issue: it adds a single-component requirement and still shows advantages for L2E over GRPO and RLOO. However, it uses 50 proposals instead of the main five-proposal budget, omits the global-only and local-only ablations, and reports three evaluation repeats for L2E versus one for the baselines. […]

Q4: […] Can full L2E, global-only, local-only, and the principal RL baselines be compared at the main five-proposal budget using the same single-component requirement as Figure 10? Reporting success rate, RI, and size/similarity distributions for the selected candidates would directly test the explanation offered for preferring the combined rule. […]

Reply: At five proposals, global-only had the highest success rate on both tasks, local-only had the highest LogP RI, and L2E had the highest MR RI. To respond to your request for a five-proposal quality comparison, we conducted additional evaluations of global-only and local-only. We rescored the Fig. 10 trajectories for L2E, GRPO, and RLOO. We used the first five proposals of the first trajectory for each of the 100 starting molecules per property. Following Fig. 10, we selected the highest-reward proposal (earliest in a tie).

AuthorsOpenReview ↗

Response to Reviewer 3dSq (part 3/5)

Global-only also had the highest median similarity to the start among successful selections on both tasks. The table reports the median heavy-atom count and median similarity among successful selections. We pooled these structural measurements across three decoding repeats for L2E and each component variant. GRPO and RLOO each had one decoding repeat. We used the trajectories from Fig. 10 for L2E, GRPO, and RLOO. We evaluated separately trained component variants on the same starts, with a training KL coefficient of 0.003 for local-only and 0.001 for global-only.

PropertyMethodSR (%)RIMedian heavy atomsMedian similarity
LogPL2E25.75.204220.583
LogPGlobal-only53.00.420240.700
LogPLocal-only44.710.50882.50.587
LogPGRPO35.01.327280.603
LogPRLOO5.00.107370.524
MRL2E19.04.111530.556
MRGlobal-only61.00.114230.674
MRLocal-only35.71.021360.561
MRGRPO22.00.700550.585
MRRLOO0.00.000——

The median heavy-atom counts of L2E and local-only reversed order between LogP and MR. On LogP, local-only achieved higher RI than L2E, with a 275.0% higher median heavy-atom count (82.5 versus 22). On MR, L2E achieved higher RI than local-only, with a 47.2% higher median heavy-atom count (53 versus 36). Successful L2E selections on MR also included molecules with thousands of heavy atoms. RLOO had no successful MR selections at five proposals.

At 50 proposals, L2E achieved higher SR and RI than GRPO and RLOO on both tasks. This comparison uses all five ten-turn trajectories per start, as in Fig. 10.

MethodLogP SR (%)LogP RIMR SR (%)MR RI
L2E36.312.51934.35.985
Global-only42.00.67253.70.282
Local-only35.316.52848.03.081
GRPO28.03.29924.00.895
RLOO3.00.2231.00.0005

At 50 proposals, global-only also succeeded most often on both properties. Local-only achieved 32.0% higher RI than L2E on LogP. On MR, L2E achieved 94.3% higher RI than local-only.

Relative to GRPO at 50 proposals, L2E achieved 29.8% higher SR on LogP and 43.1% higher SR on MR. Its LogP RI was 279.5% higher, corresponding to the 3.8-fold ratio reported in Appendix D.2. Its MR RI was 568.4% higher than GRPO’s. Among L2E’s successful LogP increases, the selected molecules had a median of 332 more heavy atoms than their starting molecules.

Overall, under the five-proposal budget and single-component requirement, L2E achieved the highest RI on MR. Local-only achieved the highest RI on LogP. Global-only achieved the highest success rate and median similarity on both tasks.

W1/Q2. Reward definitions and training conditions for POLO and L2E

W1 / Q2: […] replacing L2E's absolute-score reward with previous-state differences reduces LogP RI from 19.357 to 11.207 (Table 5), close to POLO's 11.268 (Table 3). […] What reward, preference weight, reference-policy initialization, and pair-selection settings were used for POLO? Were training data and training budgets matched? Given the reward sensitivity in Table 5, does the reported advantage over POLO persist when the reward definition is held constant?

Reply: We conducted additional training experiments with both methods using the same absolute molecular score, data, and training budget. L2E retained its SR advantage over POLO on QED, MR, and LogP. L2E and POLO use similar reward functions with different molecular scores. When we matched the score, data, and training budget, L2E retained its success-rate advantage on all three tasks.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A General Summary by the Authors

Dear Area Chairs and Reviewers,

Thank you for your time and constructive feedback. All three reviewers confirmed that our replies helped address their concerns. Reviewer fQve and unQj raised their ratings from 2 to 3 (weak acceptance). Reviewer 3dSq confirmed that our replies addressed their concerns. We provide a summary below.

(1) Overview of our work

We propose Learning to Evolve (L2E), a reinforcement learning method that trains language models to optimize molecules through successive proposals and evaluator feedback. L2E learns which edits to reinforce using scores collected during the search. Our work makes three contributions.

  • Trajectory analysis. Later edits can undo earlier property gains. Improving the target property can also reduce similarity to the starting molecule.
  • Local-global credit assignment. L2E compares cumulative molecular scores across trajectories and individual molecular scores within each trajectory. The centered local term distinguishes turns and preserves the global advantage on average.
  • Empirical findings. On S2S^2-Bench, L2E achieved a 53.7% higher molar refractivity (MR) success rate than trajectory-level GRPO (69.7% vs. 45.3%). On MuMOInstruct, L2E’s gains in joint BBBP, DRD2, and penalized LogP optimization extended to unseen instructions (Tables 1 and 2).

(2) Key strengths recognized by the reviewers

  • Practical turn-specific credit. L2E assigns credit to individual proposals using evaluator scores already collected during search. Its analysis identifies conditions under which a local update can improve final selection. (3dSq, unQj)
  • Motivation grounded in search behavior. The trajectory analysis shows that property gains can reverse across turns and conflict with feasibility, motivating credit that distinguishes proposals within a trajectory. (unQj, fQve)
  • Broad evaluation. The evaluation includes outcome-level RL and evolutionary baselines, along with ablations of the credit rule. L2E’s gains in relative improvement (RI) on MR and LogP persist at 7B and across initial-property tertiles. (fQve, 3dSq)

(3) Rebuttal responses

  • Assessing local credit. We examined the Table 4 ablation, trained a per-turn GRPO baseline, and compared L2E’s local credit with TRLOO’s reward-to-go on the same trajectories. Adding local credit to global-only L2E improved MR success and lowered QED success and RI. L2E achieved higher RI than per-turn GRPO on all three properties, and per-turn GRPO succeeded more often on QED. Alignment favored L2E’s local credit on MR and TRLOO on QED. (3dSq W1, Q1, Q5; unQj W1, Q1)
  • Reward-matched POLO comparison and REINVENT recalibration. We trained POLO and L2E with the same absolute molecular score, data, and budget. L2E retained higher success rates on all three properties. We recalibrated REINVENT. Experience replay improved its success rates and RI, and L2E still led on both metrics. (3dSq W1, Q2; fQve W1.3, S1)
  • Comparing molecular quality under shared criteria. We compared L2E and its variants with five proposals per search and the same single-component requirement. On MR and LogP, global-only had the highest success rates and the highest median similarity among successful outputs. L2E led on MR RI, and local-only led on LogP RI. L2E achieved higher RI per output heavy atom than GRPO on MR and LogP. (3dSq W2, Q4; unQj Q2; fQve S2 and follow-up)
  • Scaling the multi-property evaluation to 7B. On MuMOInstruct, L2E met all objectives more often than GRPO and achieved slightly higher mean RI across all searches. We paired their searches to examine the RI difference. Among jointly successful searches, L2E achieved higher RI per output heavy atom than GRPO. (fQve W1.1, W1.2, S3 and follow-up)
  • Additional experiments on more models. We trained GRPO on additional models and rescored both methods’ outputs under a shared molecular growth limit. With the limit, L2E succeeded more often on all three properties and led on RI for QED and MR. (fQve Q2 and follow-up)
  • Clarifying evaluation and checking docking. We specified the data splits, property transforms, and output-selection rules, and verified that starting molecules were disjoint across training and test sets. We paired results by starting ligand. L2E achieved a lower mean EGFR docking score than GRPO on the Figure 8 candidates. (3dSq Q3; fQve Q1, W1.4, S4)

(4) Reviewer feedback

  • Reviewer fQve (2 → 3) noted that the additional experiments addressed most of their concerns.
  • Reviewer unQj (2 → 3) noted that our replies addressed most of their concerns.
  • Reviewer 3dSq confirmed that their concerns had been addressed.

We thank the Reviewers and Area Chairs for their time and effort. Your feedback helped us clarify the method, strengthen the evaluation, and improve the manuscript.

Sincerely,
Authors of #35944

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.