acceptodds
Under review as a conference paper at ICLR 2027

Training Large Language Models for Small-Molecule Design with Synthetic Task Scaling

Abstract

Designing viable drug candidates requires searching a combinatorially large and rugged chemical space for molecules that satisfy multiple, often competing, objectives. Large language models (LLMs) provide a useful generative prior for this problem because of their representational capacity, reasoning ability, and flexibility when incorporating information from the external environment. While reinforcement learning from verifiable rewards (RLVR) can be used to improve the capabilities of LLMs, many chemically relevant scoring functions require hours or even days per evaluation, making them prohibitively expensive to use directly during online training. Here, we investigate whether LLMs can learn molecular design strategies from cheaper synthetic tasks that generalize to expensive molecular lead optimization settings. We find that curriculum-based training recipes that gradually incorporate more challenging synthetic design tasks enable strong performance that surpasses that of much larger frontier models on structure-based lead optimization. Our results suggest that scaling post-training using synthetic tasks is an effective strategy for adapting LLMs to high-cost experimental scenarios that are too expensive to directly train on.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
large language models, drug design, small molecules, post training, reinforcement learning, RLVR, synthetic data, scaling, chemical reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_APSyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether an LLM can be post-trained for molecular lead optimization against an expensive scoring function that is too slow to train on directly. The authors build two tiers of cheap synthetic design tasks: Tier-1, 200,000 randomly generated tasks over 116 RDKit/OpenEye descriptors with a BRICS scaffold constraint, and Tier-2, 999 structure-based tasks on PLINDER pairs scored by POSIT docking plus Chemgauss4. They then post-train Qwen3.6-35B-A3B with asynchronous RLVR (prime-rl, IPO loss) under the 12 recipes of Table 1,each for a total of 500 steps split across two stages, and evaluate on 40 held-out PLINDER systems scored by SQM2.20, which costs tens of minutes per call. The reported findings are: Tier-1-only training already moves the base model a long way on the held-out SQM task; adding a Tier-2 stage helps further; training on Tier-2 alone from the base model collapses to a single-turn policy; and the best curricula exceed five frontier models evaluated zero-shot in the same harness. The appendix adds failure-mode, rollout and chemical analyses.

Critical strengths
  1. The paper provides useful evidence of transfer from cheap synthetic tasks to an expensive structure-based design task. This is practically relevant because direct RL training with the evaluation oracle is too expensive. The use of the official PLINDER training and test splits, recipes matched at 500 steps on identical hardware, and a common five-call SQM budget for trained and zero-shot models helps make the comparisons interpretable. In particular, a model trained only on descriptor-based tasks, without protein structures, improves on the base model by 2.89 kcal/mol in the SQM-based relative binding-affinity metric and raises the constraint reward from 36.2% to 94.0% (Table 2). This supports using cheap synthetic tasks to prepare a model for tasks with expensive feedback, within the scoring setup studied here.

  2. The appendix provides useful diagnostics for evaluating molecular-design agents. Table A3 separates formatting failures from scoring failures, which helps readers assess how much reliability contributes to the comparison. Fig. A6 examines whether the model submits the best molecule it scored, making final candidate selection an explicit part of the evaluation. This matters because a model can find a good molecule but submit a worse one. Fig. A9 separates results for anionic and neutral seed ligands, which helps identify where the gains are concentrated. These analyses help explain the aggregate scores and identify limitations that an average score alone would hide.

Critical weaknesses

W1. The aggregate advantage over frontier models is concentrated in anionic seeds, and the chemical meaning of these gains needs independent validation. This limits how broadly the result can support improved molecular design.

Fig. A9 separates 14 anionic and 26 neutral seed ligands. On neutral seeds, my reading of the figure is that GPT-5.6 Sol and the best curriculum model, T1(400)-T2(100), both achieve about +3.1 kcal/mol relative to T1(500). For the recipe with the largest aggregate SQM improvement, T1(300)-T2(200), the gap over GPT-5.6 Sol is +3.88 - 1.44 = +2.44 kcal/mol. Digitising the subgroup means gives +2.59 from the anionic systems and -0.15 from the neutral ones. Thus, this aggregate advantage comes entirely from 35% of the test set. The overall average remains a valid benchmark result, but the subgroup dependence should be clear when discussing its scope.

The authors report that "removing an anionic group improved the relative binding affinity by 26.1 kcal/mol" (p. 32) across anionic modifications. This is an association across edits, so it does not establish that anion removal causes the model advantage. However, a large score change associated with charge modification makes an independent check important. SQM2.20 supplies both the optimization feedback during evaluation and the final affinity assessment, with no second scoring function or free-energy spot-check reported for the generated designs. It therefore remains unclear how much of the gain reflects improved binding versus behavior specific to SQM. Subgroup results in the main text and an independent check on a small set of charged designs would directly address this concern.

W2. The size and stability of the reported advantage are difficult to assess because the metric combines task completion with score improvement, while setting selection and training variability are not fully evaluated.

Fig. 2 states that "invalid rollouts are assigned an improvement of 0". The failure rates differ substantially: 0-2 of 200 rollouts fail for the trained models, compared with 19 for GPT-5.6 Sol. The cause also matters: GPT-5.6 Sol has 2 formatting failures and 17 scoring failures. Reliability is a relevant capability, but results over valid rollouts are needed to assess improvement among successfully scored designs. Table 2's headline metric is normalized by subtracting the average of the T1(500) model, and no per-model mean improvement over the seed is reported for the 40 systems. Reporting that mean alongside the valid-rollout results would clarify the effect of assigning zero to failed rollouts.

The reported +/-1 SE error bars describe evaluation uncertainty, while no independent repetitions of the training runs are reported. They therefore do not establish whether differences among the best schedules are stable across training runs. This matters for interpreting which curriculum choices are useful, rather than only whether staged training helps in general.

The procedure for choosing the settings is also unclear. Sec. 4 describes the preferred reward weights as "the best performing combination", without stating what data were used to choose them, and no separate validation split is described. D.3 reports that none of the models trained with alternative reward weights exceeds 2.5 kcal/mol relative to T1(500). Some of those models still exceed frontier baselines, so this sensitivity concerns the size of the reported margin. The schedules in Table 1 use predefined switching points and task mixtures, and the authors report no clear cost-performance scaling law. A clear selection criterion would help readers assess whether the chosen setting is independent of the final evaluation and how to choose a schedule for a different model or training budget.

Questions for authors

Clarifications that could change my assessment:

Q1. Please report the model means and uncertainties behind Fig. A9, particularly on the 26 neutral systems. Does any trained model improve reliably over the strongest frontier comparator on this subset? This would clarify the scope of the aggregate advantage.

Q2. What explains the 26.1 kcal/mol mean score gain associated with removing an anionic group? Is it sensitive to the proton-transfer or solvation terms of Eq. A11? An independent check on a small set of the charged designs,using another scoring method or a free-energy calculation, would help establish whether these gains correspond to improved binding.

Q3. Please report results over valid rollouts alongside the zero-filled metric, together with each model's mean SQM score improvement over the seed. Does the frontier comparison change with a more lenient answer parser or a worked format example? The Tier-1 prompt includes a worked example on p. 39, while the evaluation prompt does not. This would help separate molecule quality from answer-format reliability.

Q4. What data were used to select the curriculum schedule and reward weights? In particular, was w_obj = 0.75 / w_constraint = 0.25 selected using the 40 SQM evaluation systems? If so, can the selected setting be checked on data independent of that choice? What criterion would guide the switching point and task mixture for a different training budget?

Q5. Were any recipes trained repeatedly with independent random seeds? If so, are the differences among the best recipes stable? Limited repetition of a few representative recipes would help assess the training variability that the evaluation error bars do not measure.

Q6. How was the tool-call penalty warmup handled in T2(500)? The reported collapse is a single-turn policy that does not iterate with tool calls, while B.3 reports conservative tool use when the warmup is omitted. The standard warmup spans 1000 group scoring steps. Was the collapse robust to a modest change in this schedule? This would help distinguish a failure caused by the training settings from difficulty learning Tier-2 tasks directly.

Suggestions for presentation, separate from the clarification requests above:

The title's emphasis on synthetic task scaling would be clearer if the paper explicitly distinguished corpus size from training duration. At 500 steps with 64 unique tasks per batch, a run uses at most 32,000 of the 200,000 Tier-1 tasks. Clarifying what is varied would help readers understand which scaling claim the experiments support.

The prompt text in Figure 1 is too small to read clearly at normal print size. Please enlarge the text or use a shorter excerpt so the task setup is easier to understand.

4 comments
AuthorsOpenReview ↗

Author rebuttal

We thank the reviewer for their feedback and suggestions and address the questions below, which also cover the weaknesses raised.

Q1. Reporting model means and uncertainties

Please report the model means and uncertainties behind Fig. A9, particularly on the 26 neutral systems. Does any trained model improve reliably over the strongest frontier comparator on this subset? This would clarify the scope of the aggregate advantage.

Response: The means and uncertainties for both the trained and zero-shot models are already included in Figure A9, but we provide the table for the means and uncertainties behind the Figure here for convenience:

ModelAnionic seeds (n=14)Neutral seeds (n=26)All systems (n=40)
T1(400)-T2(100)+3.41 ± 3.54+3.06 ± 0.94+3.18
T1(300)-T2(200)+5.81 ± 4.24+2.84 ± 2.06+3.88
T1(250)-Mix(250)+5.37 ± 4.60+2.72 ± 1.34+3.65
T1(350)-Mix(150)+2.15 ± 2.01+2.19 ± 0.59+2.17
T1(450)-Mix(50)+2.35 ± 2.48+2.28 ± 0.61+2.31
T1(450)-T2(50)+2.38 ± 1.64+1.91 ± 0.50+2.07
T1(350)-T2(150)+4.96 ± 3.06+0.83 ± 0.47+2.28
T1(300)-Mix(200)+2.69 ± 2.33+1.49 ± 0.44+1.91
T1(400)-Mix(100)-0.99 ± 1.54+1.69 ± 0.51+0.75
T1(250)-T2(250)+5.71 ± 3.77+1.18 ± 0.60+2.76
T1(500)0.00 (ref)0.00 (ref)0.00 (ref)
GPT-5.6 Sol-1.59 ± 1.61+3.08 ± 1.44+1.44
Claude Opus 5-5.24 ± 2.87+2.58 ± 0.67-0.16
Claude Opus 4.8-2.95 ± 2.60+2.20 ± 0.92+0.40
GPT-5.5-1.33 ± 2.13+1.18 ± 0.93+0.30
GLM-5.2-4.96 ± 3.67+1.77 ± 1.11-0.58
Qwen 3.6 (base)-7.40 ± 3.82-0.47 ± 1.05-2.89

These results further reinforce the trained models' large improvement in the cases with anionic seed structures when compared to the zero-shot models, outperforming the zero-shot baselines and leading to better average improvement across all 40 systems. For neutral seed structures specifically, we see that the trained models achieve competitive performance to the zero-shot frontier models, substantially improving upon the performance of the Qwen 3.6 (base) model and keeping pace with far stronger models such as GPT-5.6 Sol. While the degree of increase in the relative binding affinity improvement for the trained models on the neutral seed systems is lower than that for the anionic seeds, the fact that a weaker Qwen 3.6 (base) model can still be trained to match the performance of frontier models is a testament to the effectiveness of the curriculum training strategy investigated in this work.

Q2. 26.1 kcal/mol mean score gain

What explains the 26.1 kcal/mol mean score gain associated with removing an anionic group? Is it sensitive to the proton-transfer or solvation terms of Eq. A11? An independent check on a small set of the charged designs,using another scoring method or a free-energy calculation, would help establish whether these gains correspond to improved binding.

Response: The 26.1 kcal/mol mean score gain comes from how charged groups affect the gas-phase interaction energy ΔEint\Delta E_{\text{int}} and the change in the solvation free energy ΔΔGsolv\Delta \Delta G_{\text{solv}}, where for a charged ligand both of these terms in Equation A11 are large and opposite in sign, each on the order of tens of kcal/mol for a monovalent ion with the physical binding contribution being a small residual between these terms. Removing the anion reduces both the favorable interaction energy and the opposing solvation penalty, and when the reduction in the solvation penalty outweighs the loss of favorable interactions, we see a large gain in the score. To clarify this point, we have added the following text to the manuscript:

Appendix D.6, Page 33: Removing an anionic group can produce a large predicted improvement in binding when the reduction in the solvation penalty (ΔΔGsolv\Delta\Delta G_{\text{solv}}) outweighs the loss of favorable gas-phase interaction energy (ΔEint\Delta E_{\text{int}}). Because these large, opposing contributions substantially cancel, errors in their relative magnitudes can exaggerate the predicted improvement.

AuthorsOpenReview ↗

Author rebuttal (part 2)

Q3. Results under different criteria

Please report results over valid rollouts alongside the zero-filled metric, together with each model's mean SQM score improvement over the seed. Does the frontier comparison change with a more lenient answer parser or a worked format example? The Tier-1 prompt includes a worked example on p. 39, while the evaluation prompt does not. This would help separate molecule quality from answer-format reliability.

Response: While models do exhibit some failure to adhere to answer formatting, we note that our trained models and stronger frontier models like GPT-5.6 Sol are sufficiently competent such that these failures are rare and a non-issue. Furthermore, we note that failure to adhere to the correct answer formatting is a real failure mode that is corrected through the curriculum training procedure and should be factored into the reporting of the results.

Q4. Curriculum selection

What data were used to select the curriculum schedule and reward weights? In particular, was wobjw_{obj} = 0.75 / wconstraintw_{constraint} = 0.25 selected using the 40 SQM evaluation systems? If so, can the selected setting be checked on data independent of that choice? What criterion would guide the switching point and task mixture for a different training budget?

Response: The weighting factors wobj=0.75w_{\mathrm{obj}}=0.75 and wconstraint=0.25w_{\mathrm{constraint}}=0.25 used for the results presented in the main text were chosen based on the evaluation set performance, with the results for the other weighting combinations reported in the appendix. For a different training budget, the specific weighting factors would need to be determined empirically, but the task mixture should stay skewed towards the harder task during the curriculum training stage. A good assessment of whether a mixture can continue to deliver training signal without inducing catastrophic forgetting in the model is to examine the training dynamics of the rewards, as is shown in Figure A1 and Appendix D.1. Monitoring this quantity during tuning can help decide a more optimal mixture or curriculum structure. To clarify the selection process for the weighting factor, we have added the following text to the manuscript:

Page 5: We explored various wobjw_{\mathrm{obj}} and wconstraintw_{\mathrm{constraint}} and found wobj=0.5,wconstraint=0.5w_{\mathrm{obj}}=0.5,w_{\mathrm{constraint}}=0.5 for Tier-1 tasks and wobj=0.75,wconstraint=0.25w_{\mathrm{obj}}=0.75,w_{\mathrm{constraint}}=0.25 for Tier-2 tasks to be the best performing combination based on the evaluation set performance, and so we focus our analysis on this condition and its results ...

Q5. Seed repetition and statistics

Were any recipes trained repeatedly with independent random seeds? If so, are the differences among the best recipes stable? Limited repetition of a few representative recipes would help assess the training variability that the evaluation error bars do not measure.

Response: All recipes were trained using a single random seed due to constraints on the compute available for this work. While a single run cannot characterize the training variability with regards to the evaluation performance metric, the trends we uncover in terms of model performance are robust and the more important result. To clarify this point, we have added the following text:

Page 5: We train each model using a single seed due to compute constraints, and while we do not obtain a clear "scaling law" where increased cost leads to increased performance, we do see that the more expensive curriculum training approaches outperform the cheaper Tier-1 only training, demonstrating robust trends which are the focus of this work.

AuthorsOpenReview ↗

Author rebuttal (part 3)

Q6. Tool call penalty warmup

How was the tool-call penalty warmup handled in T2(500)? The reported collapse is a single-turn policy that does not iterate with tool calls, while B.3 reports conservative tool use when the warmup is omitted. The standard warmup spans 1000 group scoring steps. Was the collapse robust to a modest change in this schedule? This would help distinguish a failure caused by the training settings from difficulty learning Tier-2 tasks directly.

Response: The tool call penalty warmup was handled in the same way for T2(500) as for all other curriculum experiments, as we found the choice of 1000 scoring steps to be robust for ensuring models learned to economically use the tools provided without becoming exploitative across all of the settings we investigated. We did find from early experiments that T2(500) without a tool penalty similarly does not work, reflecting that the model collapse is a consequence of the task difficulty as opposed to a miscalibrated training setting. To clarify this point, we have added the following text to the manuscript:

Appendix D.2, Page 29: Additionally, we find that while removing the tool call penalty can help the model exploit tools more, this alone did not correct the failure mode noted in Figure A2, indicating that the failure is due to the task difficulty and not a miscalibrated tool penalty warmup during training.

Clarification about task consumption

The title's emphasis on synthetic task scaling would be clearer if the paper explicitly distinguished corpus size from training duration. At 500 steps with 64 unique tasks per batch, a run uses at most 32,000 of the 200,000 Tier-1 tasks. Clarifying what is varied would help readers understand which scaling claim the experiments support.

Response: To clarify this point, we have added the following text to the manuscript:

Page 4: A single 500-step run consumes 32000 of these, at 64 unique tasks per step, so the pool is sized to supply non-repeating tasks rather than to be exhausted.

Fixing prompt text size in Figure 1

The prompt text in Figure 1 is too small to read clearly at normal print size. Please enlarge the text or use a shorter excerpt so the task setup is easier to understand.

Response: We will enlarge the text in the revised version of the paper.

Thank you for the replies and the subgroup table. I continue to find the transfer from cheap synthetic tasks to the SQM evaluation useful. The reported check that removing the tool-call penalty did not correct the Tier-2-only collapse addresses my specific concern about the penalty. The clarification of the task-pool size is also helpful.

My main concerns remain:

  1. On the neutral systems, the reported means are +3.06 kcal/mol for T1(400)-T2(100) and +3.08 kcal/mol for GPT-5.6 Sol, relative to T1(500). Their mean performance is similar. The overall advantage of the best recipes over GPT-5.6 Sol is concentrated in the anionic systems. The explanation of the opposing interaction and solvation contributions clarifies how a large score gain can arise. However, it does not establish whether the proposed designs bind more strongly. The added discussion acknowledges that errors in these contributions can exaggerate the predicted gain, and the reply provides no independent check of the designs. This remains my main concern about the interpretation of the gains.

  2. Thank you for confirming that the reward weights were chosen using evaluation-set performance. The reported comparison is therefore not independent of that choice. Reporting the other weight settings is helpful, but results on data not used for selection would be needed to assess the selected margin independently. I also understand the compute constraints behind using one training seed per recipe. This leaves variation across training runs, including the stability of the recipe ordering, unmeasured.

  3. I agree that reliability should be included in the main results and that formatting failures are rare for GPT-5.6 Sol. However, the original Table A3 lists 2 formatting failures and 17 scoring failures out of 200 rollouts for this model. My question also concerned these scoring failures. The requested results over valid rollouts and mean SQM-score improvement over the seed are not provided in the reply. Reporting these alongside the main metric would help clarify how much of the comparison reflects successful task completion and how much reflects score improvement.

The rebuttal clarifies several points, but my main reservations about the chemical interpretation of the gains and the independence of the evaluation remain. I therefore maintain my rating of 2.

Reviewer_jmucOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the question of whether training language models for molecule generation on cheaper synthetic tasks in an easy-to-hard curriculum can improve their performance on tasks which are more expensive to score.

The work trains an MoE LLM, Qwen 3.6 with 35 B parameters using the curriculum training. Tier 1 tasks are about generating new molecules with improved molecular properties while preserving a required core structure and satisfying other constraints, and the tier 2 harder tasks are about generating new molecules with changed docking scores with some target protein.

The model is tested on 40 protein-molecule systems, and the results show that the post-training substantially improves upon the base model and the model achieves higher average evaluation scores than models such as GPT-5.6 Sol and Claude Opus 5 on this benchmark.

Critical strengths

The work shows that post-training with reinforcement learning on synthetic tasks which are cheaper to evaluate can improve the performance on tasks evaluated on expensive high-fidelity methods. This gives a useful recipe for training LLMs for molecule generation.

Critical weaknesses

The experimental results support that training on both types of tasks is better than training on a single type. However, it does not show that sequencing in two stages is better than just one stage with both types of tasks mixed from the start. A key baseline of training a mix of T1 and T2 tasks from the start with comparable volume is missing. Without such a baseline, it is not clear that two-stage curriculum is better than just one-stage reinforcement learning on both types of the synthetic tasks. Since one central claim is that the recipe of gradually incorporating more challenging tasks is useful, missing this mixture from the start experiment is a significant weakness.

Questions for authors

How does the two-stage schedule compare with training on a mixture of both types of tasks from the start, using comparable numbers of examples from each type? This would help isolate the value of task order.

3 comments
AuthorsOpenReview ↗

Author rebuttal

We thank the reviewer for their feedback and suggestions and address the main weaknesses and questions below.

W1. Mixture of tasks baseline

However, it does not show that sequencing in two stages is better than just one stage with both types of tasks mixed from the start. A key baseline of training a mix of T1 and T2 tasks from the start with comparable volume is missing. Without such a baseline, it is not clear that two-stage curriculum is better than just one-stage reinforcement learning on both types of the synthetic tasks. Since one central claim is that the recipe of gradually incorporating more challenging tasks is useful, missing this mixture from the start experiment is a significant weakness.

Response: Based on the T2(500) performance where we trained models using only Tier-2 tasks starting from the Qwen 3.6 (base) model, it is evident that training using only the difficult tasks leads to model mode collapse, resulting in degenerate policies that do not exhibit any meaningful multi-turn optimization behavior. Because the mixture of tasks is dominated by Tier-2 tasks in the curriculum with a ratio of 25% Tier-1 and 75% Tier-2 (Table 1), training with this mixture from scratch would lead to suboptimal performance of the model. Furthermore, there is a significant asymmetry in the amount of data available for each tier of task, with 200000 possible training tasks for Tier-1 and only 999 training tasks for Tier-2, so extensive training with Tier-2 tasks would hurt model generalization beyond the training set.

Q1. Training on both task types from the start?

How does the two-stage schedule compare with training on a mixture of both types of tasks from the start, using comparable numbers of examples from each type? This would help isolate the value of task order.

Response (same response as to W1): Based on the T2(500) performance where we trained models using only Tier-2 tasks starting from the Qwen 3.6 (base) model, it is evident that training using only the difficult tasks leads to model mode collapse, resulting in degenerate policies that do not exhibit any meaningful multi-turn optimization behavior. Because the mixture of tasks is dominated by Tier-2 tasks in the curriculum with a ratio of 25% Tier-1 and 75% Tier-2 (Table 1), training with this mixture from scratch would lead to suboptimal performance of the model. Furthermore, there is a significant asymmetry in the amount of data available for each tier of task, with 200000 possible training tasks for Tier-1 and only 999 training tasks for Tier-2, so extensive training with Tier-2 tasks would hurt model generalization beyond the training set.

Thank you for the response. I acknowledge the reported collapse of T2-only training and the concern that repeatedly sampling from the smaller T2 dataset could cause overfitting. However, these arguments do not establish that mixed training from the start would necessarily perform poorly. Including T1 tasks from the start could help prevent collapse.

The requested baseline does not need to use 75% T2 tasks throughout training. Instead, the comparison could keep the total number of training examples from each tier the same, while changing when they are used. For example, an 80% T1 / 20% T2 mixture from the start over 500 steps would approximately match the total exposure to each task type in T1(400)-T2(100). This would avoid increasing total T2 exposure while testing the effect of task order. Without this comparison, the advantage of the two-stage schedule over mixed training from the start remains unestablished.

Thank you for the response and suggestion. We agree that a fixed 80% Tier 1 / 20% Tier 2 mixture over 500 steps would match total exposure to each tier in T1(400)–T2(100) and test whether delaying Tier 2 provides an additional benefit.

However, we note that our two-stage schedule is well grounded in prior work on easy-to-hard RL post-training (Parashar et al., Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning, ICLR 2026, Table 2, and NVIDIA’s Nemotron 3 Nano Technical Report, 2025, Section 3.2.2 and Figure 7). We explored a subset of possible recipes, and others, including the proposed fixed mixture, could perform as well or better. The paper’s central finding holds regardless of which schedule performs best. Training on a recipe involving cheaper synthetic tasks improves performance on the more expensive held-out SQM task. Tier 1 training improves performance over the base model, and adding Tier 2 improves mean performance over Tier 1 alone across all ten staged recipes in Table 2.

The discussion window closes in a few hours, and the additional training run takes about two days. We’d be happy to run the comparison and include the results in the camera-ready version.

Reviewer_odpjOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether large language models can acquire transferable molecular optimization capabilities through reinforcement-learning post-training on relatively inexpensive synthetic tasks.

The authors construct a curriculum involving low-cost molecular property objectives and more demanding structure-based optimization tasks, with the goal of improving performance on expensive molecular evaluation objectives that are not directly used during training.

Experiments on structure-based lead optimization suggest that curriculum-based training produces better downstream performance than training exclusively on either simpler tasks or more expensive objectives. The resulting model also compares favorably with substantially larger general-purpose language models.

Critical strengths
  1. Relevant problem with a convincing practical motivation.

The paper addresses an important limitation of reinforcement learning for molecular design: realistic molecular evaluation functions can be too computationally expensive for extensive online optimization.

Learning transferable optimization strategies from cheaper tasks is a meaningful research direction, particularly if it reduces dependence on expensive molecular evaluations.

  1. Substantial experimental evidence supporting the proposed training strategy.

The authors investigate multiple training curricula and evaluate their transfer to a more expensive molecular scoring function. The comparison between different training mixtures provides useful evidence that the composition and ordering of post-training tasks matter.

The evaluation across multiple protein systems strengthens the empirical contribution beyond a single illustrative example.

Critical weaknesses
  1. The contribution is primarily empirical, and the mechanism of transfer remains insufficiently characterized.

Curriculum learning and training on inexpensive proxy objectives are established ideas. The novelty therefore rests primarily on demonstrating that this strategy transfers effectively to expensive molecular optimization.

However, the experiments do not fully establish which capabilities are transferred or why the proposed curriculum is effective. It remains unclear whether the improvement results from learning general molecular optimization principles, improved familiarity with the chemical search space, or adaptation to related scoring functions.

This distinction matters because the broader claim of transferable molecular design capabilities is stronger than the demonstrated improvement on the selected evaluation setting.

  1. Generalization beyond the selected computational evaluation pipeline remains uncertain.

The evaluation relies on computational molecular scoring, including docking and a more expensive scoring function. Although these objectives are more realistic than simple molecular descriptors, they remain imperfect proxies for experimentally measured binding affinity and drug-development success.

The conclusions would be more convincing with evidence that the learned improvements transfer across substantially different scoring mechanisms, protein families, or independent experimental measurements.

Questions for authors
  1. What evidence distinguishes the learning of transferable molecular optimization strategies from adaptation to correlations shared by the training and evaluation scoring functions?
  2. How does the proposed curriculum compare with simpler alternatives using the same total training compute and molecular evaluation budget?
  3. How sensitive are the conclusions to the choice of evaluation oracle and the distribution of protein targets?
3 comments
AuthorsOpenReview ↗

Author rebuttal (weaknesses)

We thank the reviewer for their feedback and suggestions and address the weaknesses below, with a following comment addressing the specific questions.

W1. Mechanism of transfer is insufficiently characterized

However, the experiments do not fully establish which capabilities are transferred or why the proposed curriculum is effective. It remains unclear whether the improvement results from learning general molecular optimization principles, improved familiarity with the chemical search space, or adaptation to related scoring functions.

Response: In addition to the evaluation time performance of our trained models on the SQM optimization task, we characterize many additional aspects of the capability transfer instilled by the curriculum approach through our chemical analysis and the additional results provided in Appendix D. Specifically, we find through our analysis that:

  1. Models trained with our approach adopt a specific design strategy around molecular size and halogenation, deviating from the behaviors demonstrated by the frontier models (Figure 4) while also showing specific behaviors around functional group addition and removal (Figure 5).
  2. Training with the curriculum approach leads models to arrive at more efficient design strategies where models effectively leverage tool feedback throughout the longer horizon tasks (Table 2, Figures A7 and A8), and models make more grounded decisions during these multi-turn design settings (Figure A6).
  3. The curriculum training approach improves the model's familiarity with the chemical search space and syntax, enabling the models to more consistently generate valid molecules and results compared to frontier models (Table A3).

Thus, we characterize both the emergence and transfer of molecular design strategies throughout the training process as well as auxiliary considerations such as how the curriculum training impacts the model's multi-turn reasoning behavior and its adaptation to the chemical design space, providing a systematic picture of the impacts our approach has on model behavior. Regarding the effectiveness of the curriculum, we clearly demonstrate throughout the manuscript that training with the curriculum dramatically improves the model's performance at this molecular design task compared to zero-shot frontier and open-weight models (Figure 2, Table 2) and we note that efforts to side-step the curriculum (e.g., by training directly on Tier-2 tasks from the start) lead to clear mode collapse in the model's design behavior (Appendix D.2 and Figure A2).

W2. Generalization beyond the evaluation pipeline

The conclusions would be more convincing with evidence that the learned improvements transfer across substantially different scoring mechanisms, protein families, or independent experimental measurements.

Response: We note that there is already a very strong notion of transfer demonstrated in the manuscript by considering the evaluation performance of models trained only on Tier-1 tasks, i.e. T1(500). As shown in Figure 2 of the main text, T1(500) already closes the gap in performance between the Qwen 3.6 (base) model and the stronger frontier models such as GPT-5.5 despite being trained on a set of tasks that are very different in terms of fidelity and scoring mechanism compared to the evaluation setting, where Tier-1 tasks only use RDKit and OpenEye oracles for scoring instead of SQM2.20. Thus, the improvement in performance just after Tier-1 tasks shows that there is clear transfer across a very large gap in fidelity. Furthermore, while the Chemgauss4 scoring function used in the Tier-2 tasks and the SQM2.20 scoring function are both binding affinity scoring functions, they vary dramatically in definition, accuracy, and cost (Appendix B.2), so the further improvements that come from curriculum training with Tier-2 tasks also point to learned improvements transferring across a large fidelity gap. Finally, we carry out evaluations on the test set of the PLINDER dataset, which has been explicitly curated to be highly diverse, both in the protein families present and the ligands bound, and distinct from the train set.

AuthorsOpenReview ↗

Author rebuttal (questions)

Q1. Transferable strategies vs. shared scoring correlations

What evidence distinguishes the learning of transferable molecular optimization strategies from adaptation to correlations shared by the training and evaluation scoring functions?

Response: The auxiliary characterizations that we have provided in the text regarding trends in molecular design strategies, efficiency of design trajectories from the trained models, and robustness to the chemical representation (see response to Weakness 1) show that there are tangible improvements and changes obtained from the curriculum training that go beyond adaptation shared by training and evaluation scoring functions. Furthermore, the performance of T1(500) on the SQM-based design task shows that even in a setting where there is no correlation in the scoring function or task objective between training and evaluation, there is still a significant gain in performance.

Q2. Comparison at matched compute and evaluation budget

How does the proposed curriculum compare with simpler alternatives using the same total training compute and molecular evaluation budget?

Response: The main text already provides results on how T1(500) performs without any curriculum mixture or additional stages of training. We also tried training with just the Tier-2 tasks (T2(500), see Table 1) and we find that the models do not improve, instead mode collapsing to a degenerate single-turn policy which does not perform any optimization. This demonstrates the importance of having a curriculum-based approach for ensuring robust and smooth acquisition of design competencies.

Q3. Sensitivity to the evaluation oracle and target distribution

How sensitive are the conclusions to the choice of evaluation oracle and the distribution of protein targets?

Response: In our discussion in Section 5.1, we find that there is high variance in the performance between different targets in the evaluation set, motivating the use of the T1(500) performance as a baseline to isolate the effect of the curriculum itself. This reduces the sensitivity of the findings to the distribution of the protein targets. While the error bars in Figure 2 for the standard error are wide and at times overlapping, the trend of improved performance using the curriculum is robust and the more important finding of this work.

Thank you for the detailed responses. The added analyses of molecular design strategies and optimization trajectories, along with the PLINDER evaluation, strengthen the empirical characterization. I will maintain my rating of 3 (Weak Accept).

Some of my concerns remain. The Tier-1 results and the comparisons between Chemgauss4 and SQM2.20 provide evidence of transfer across scoring settings. However, using different scoring functions does not by itself rule out shared correlations in the chemical search space or task. The behavioral analyses offer useful clues about what changes during training, but the mechanism of transfer remains only partially established.

The T1(500) and T2(500) results are also informative, particularly the reported mode collapse with T2-only training. My question about matched training compute and molecular-evaluation budgets remains only partly addressed, since the response does not specify how those budgets compare quantitatively.

Finally, PLINDER broadens the target distribution, while the evaluation still relies on SQM2.20. The authors also note that error bars are wide and sometimes overlap. I would therefore describe the generalization evidence as promising, with the robustness of the gains still requiring cautious interpretation.

These limitations temper the strength of the transfer and generalization claims. I nevertheless find the practical problem and empirical study valuable enough to retain my weak-accept recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.