acceptodds
Under review as a conference paper at ICLR 2027

A Large Scale Investigation of Scaling Limits in Chemical Language Models

22 views

Abstract

Chemical Language Models (CLMs) are increasingly used in *de novo* drug design, driven by recent growth in model scale, compute, and dataset size. However, the relationship between design choices, training dynamics, and downstream generation quality remains poorly understood. We present a compute-controlled scaling study of CLMs comprising more than 30,000 experiments across molecular representations (SMILES, SELFIES, SAFE), tokenizations (atom-level and byte-pair encoding), model scales (0.5M–1B parameters), leakage-controlled datasets (MOSES, ChEMBL, PubChem, ZINC-22), and architectures (decoder-only and encoder–decoder). By fitting IsoFLOP profiles, we establish clear scaling trends in pretraining loss, but find that these improvements do not translate into comparable gains in goal-directed molecular design. Layer-wise probing and sparse autoencoder analysis reveal continued development of chemical representations: chemical syntax saturates early, while semantic properties emerge more slowly and become increasingly accessible with further training and model scale. These representational gains coexist with diminishing improvements in goal-directed generation under the evaluated protocols and oracle budgets. Our resulting suite of models, NovoMolGen, achieves state-of-the-art results, outperforming prior CLMs and specialized generative models in goal-directed molecular generation across drug discovery tasks. These findings expose a disconnect between chemical representation learning and downstream molecular design, motivating the development of pretraining paradigms that more directly learn chemical semantics.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Scaling Laws, De Novo Molecule Generation, Chemical Language Models, Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XVv2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether scaling chemical language models (CLMs) improves downstream goal-directed molecular generation. The authors train decoder-only and encoder-decoder CLMs across model sizes, molecular representations, tokenizations, and pretraining corpora using self-supervised sequence modeling, then fine-tune them with reinforcement learning for targeted generation.

They evaluate PMO and protein-ligand docking tasks, considering both generated-molecule quality and sample efficiency. The main result is that pretraining loss and several representation probes continue to improve with scale, while downstream generation performance shows only marginal gains beyond relatively small models.

Critical strengths
  • The paper presents a large and systematic study across model sizes, architectures, molecular representations, tokenizations, and datasets.
  • The analysis is thorough, covering pretraining loss, OOD behavior, unconstrained generation, linear probes, sparse autoencoders, PMO, and docking.
  • The observation that representation-related metrics continue improving while downstream generation saturates is interesting.
  • The SAE analysis adds an interpretability angle beyond standard scaling and probing experiments.
Critical weaknesses

1. The central research question is not convincingly answered

The paper asks a difficult question: whether scaling the pretrained CLM itself improves goal-directed molecular generation. I am not convinced that the chosen experimental design can answer this question cleanly, because the measured downstream performance reflects the full pipeline: pretrained model + RL fine-tuning + oracle + oracle budget. Therefore, a plateau in PMO or docking performance does not necessarily imply a plateau in the useful capability of the pretrained model itself. The following two points explain why I find this difficult to interpret.

2. The downstream evaluation may not reliably reflect the benefit of a better pretrained model

The RL stage relies on computational oracle scores. This is understandable, since obtaining experimental labels during iterative RL would be impractical, but it creates a risk that the agent mainly learns to exploit the behavior of the oracle rather than benefiting from richer pretrained chemical knowledge.

The authors partly acknowledge this issue by noting that current benchmarks may mainly measure oracle exploitation and by suggesting harder tasks, unseen targets, and stricter oracle budgets as better tests of whether larger models help.

3. Sample efficiency mixes pretrained-model quality with RL adaptation difficulty

I do think the PMO AUC metric provides a useful insight: sample efficiency changes only modestly beyond a certain model size. However, this does not by itself show whether the larger pretrained model is better or not.

A larger model may provide a better pretrained prior, but it also introduces a larger trainable policy that may require more feedback to adapt efficiently. In other words: sample efficiency depends on both prior quality and RL adaptation difficulty. Therefore, similar sample efficiency between a 5M and 1B model could mean either that scaling does not help, or that the benefit of the larger pretrained model is offset by a higher fine-tuning cost.

For me, this leaves the central research question unresolved. The paper convincingly demonstrates limited scaling gains under the specific RL-and-oracle evaluation setup, but I am less convinced that this is sufficient to conclude that scaling the pretrained CLM itself does not improve goal-directed molecular generation. One could instead evaluate downstream quality using more standard predictive metrics such as MAE or F1 score, but this immediately runs into the limitation discussed in Point 2. That's why I think the research question posed in the paper is inherently difficult to answer, and the current experimental design does not resolve it convincingly.

4. Some claims are stronger than the evidence supports

The paper describes larger models as learning "richer chemical representations." The probe and SAE results do show that selected properties and common functional groups become more accessible, but this does not necessarily establish a generally richer or more transferable molecular representation. The SAE functional-group analysis is also limited to only ten common groups.

A more direct way to evaluate representation quality is to use pretrained models at different scales as molecular representation extractors and test whether those representations improve downstream molecular property prediction. Recent work has already studied this setup. For example, How Well Do Large-Scale Chemical Language Models Transfer to Downstream Tasks? evaluates CLMs at different scales using both frozen linear probes and full fine-tuning on molecular property prediction tasks, and finds that lower pretraining loss does not consistently translate into better downstream performance. Similarly, Unveiling Scaling Behaviors in Molecular Language Models: Effects of Model Size, Data, and Representation studies scaling across pretraining and downstream transfer and reports that downstream scaling behavior depends strongly on model size, data, and molecular representation. These results make me cautious about interpreting improved probe performance in the current paper as evidence that larger CLMs necessarily learn broadly better molecular representations.

Similarly, the finding that pretraining loss follows compute-optimal scaling is useful characterization, but qualitatively expected for transformer language models trained with next-token objectives.

Minor:

  • The first sentence in the abstract: “Chemical Language Models (CLMs) are increasingly used in de novo drug design, driven by recent growth in model scale, compute, and dataset size.” is conceptually imprecise. The causal link is too loose. Growth in scale, compute, and data may enable better CLMs, but those are not themselves the direct reason CLMs are increasingly used. A sharper sentence would connect adoption to demonstrated utility, while scale/compute/data are enabling factors.
  • Code is promised for release, but I would appreciate it if the authors could provide the code before or during rebuttal.
Questions for authors

Clarification on representation scaling.
The paper argues that larger models learn "richer chemical representations," supported by improved linear probes and SAE analyses. How do the authors reconcile this with recent work reporting mixed or setup-dependent downstream transfer when scaling chemical language models and data (especially How Well Do Large-Scale Chemical Language Models Transfer to Downstream Tasks?)?

3 comments
AuthorsOpenReview ↗

Response to Reviewer XVv2

Thank you for the careful review. All three reviewers recognise the rigour and systematic design of our study and highlight the disconnect between pretraining improvements and downstream generation. We also appreciate your recognition of the interpretability analysis.


W1 & W3: A plateau in PMO or docking performance does not necessarily imply a plateau in the useful capability of the pretrained model itself. ... Sample efficiency mixes pretrained-model quality with RL adaptation difficulty.

Our evaluation addresses whether improved pretraining translates into better goal-directed generation, and our experiments show that larger policies that require more feedback alone cannot explain the limited gains. Figure 1 captures the central result: increasing model size 200-fold, from 5M to 1B, improves aggregate PMO only from 16.45 to 16.68 and total docking score from −65.8 to −66.2 kcal/mol, despite lower pretraining loss and greater accessibility of chemical information, as shown by layer-wise probes and sparse autoencoder analysis. These findings support our hypothesis that current benchmarks are largely saturated with respect to the benefits of additional pretraining.

Zero-shot evaluation removes policy adaptation entirely. We sampled directly from each pretrained prior without RL, using the same 10,000-call oracle budget:

Model sizePrior-only AUC Top-10 (22 tasks)After AHC fine-tuningAdded by RL
0.5M10.0616.02+5.96
5M10.7316.45+5.72
32M10.9716.48+5.51
157M11.4016.54+5.14
300M10.8316.64+5.81
1B10.8016.68+5.88

Prior-only performance peaks at 157M and subsequently declines; removing adaptation does not reveal sustained gains across the full size range. Task-specific optimisation contributes substantially larger improvements. The discrepancy also occurs within a fixed 32M policy: extending pretraining from 500 to 75,000 steps improves measured representations but leaves downstream PMO unimproved (16.80 versus 16.48; Figures 5b and I3), showing that increased policy size cannot be the sole explanation.

Providing five times more oracle feedback does not reveal a consistent larger-model advantage across the two tokenisations tested. We increased the budget on Perindopril_MPO from 10,000 to 50,000 calls:

ModelTokenisationAUC Top-10 @ 10kAUC Top-10 @ 50k
32MAtom-level0.637 ± 0.0500.675 ± 0.012
32MBPE0.635 ± 0.0180.706 ± 0.027
300MAtom-level0.668 ± 0.0110.724 ± 0.063
300MBPE0.669 ± 0.0490.691 ± 0.051

Additional feedback improves every configuration, but at 50k the 300M model leads with atom-level tokenisation while the 32M model leads with BPE. Although this experiment covers a single task, it directly examines whether additional feedback consistently yields an advantage for the larger pretrained model. We have incorporated both additional experiments into Section 3.5 and Appendix I and clarified the benchmark-saturation hypothesis in Section 4.

We agree that separating the benefits of pretraining from downstream optimisation is a difficult question; addressing it is central to this study. We consider fixed-budget generation, complemented by prior-only evaluation and increased-budget experiments, the most practical setup for assessing whether additional pretraining improves molecular design. Establishing that current benchmarks capture little of this additional benefit is itself a valuable finding for the community. We would welcome your suggestions for a concrete alternative setup that could more effectively isolate these benefits.


W2: The agent mainly learns to exploit the behavior of the oracle rather than benefiting from richer pretrained chemical knowledge.

Our results support the concern that current benchmarks reward optimisation of the supplied oracle more than the additional chemical information acquired through further pretraining. A compact or early-trained prior already supports strong performance, while improved pretraining loss and measured chemical representations provide little additional downstream benefit. This limited sensitivity is central to our interpretation of benchmark saturation in Section 4.

Here, "oracle exploitation" means optimising the supplied scoring function; our experiments do not establish that the agent exploits flaws in that function. Higher oracle scores also do not necessarily imply better experimental activity. We have clarified this distinction in Section 4 and connected it to our recommendations for function-informed pretraining and downstream evaluations that better test the value of pretrained chemical knowledge.

AuthorsOpenReview ↗

Response to Reviewer XVv2 (continued)

W4 & Q1: How do the authors reconcile this with recent work reporting mixed or setup-dependent downstream transfer when scaling chemical language models and data?

Our findings are consistent with mixed downstream scaling, which we already discuss in the introduction through Xu et al. (2026) and which directly motivates our study. Improved accessibility of the properties measured by our probes does not imply uniformly better downstream transfer, consistent with Sagawa & Kojima (2026).

Our additional evaluation on 22 TDC ADMET tasks confirms this distinction: pretrained representations outperform width-matched untrained models on 18–19 tasks, but matched-width evaluation shows no consistent improvement beyond 5M parameters. We have added these results to Section 3.4 and Appendix D and clarified that our representation claims concern the specific chemical information measured by our probes.


W4: The SAE functional-group analysis is also limited to only ten common groups.

The ten functional groups provide a targeted substructure evaluation. Our interpretability analysis also examines 36 property targets across layers and training checkpoints. We selected common, SMARTS-defined groups to compare their accessibility in SAE features and residual-stream neurons using five-fold stratified cross-validation. This follows the concept-based evaluation approach of Cohen et al. (2025), who examined 14 functional groups. SAE features outperform residual-stream neurons on seven of our ten groups (Table D2). Together with the property analyses, these experiments characterise which chemical information becomes accessible during pretraining, while the additional ADMET experiments separately assess downstream transfer.


W4: Compute-optimal pretraining scaling is qualitatively expected for transformers.

The contribution is the measured chemical scaling frontier and its failure to predict comparable gains in generation. IsoFLOP analysis quantifies the allocation of compute between parameters and tokens, while bits per molecule enables comparison across encodings. Connecting that frontier to probes, training checkpoints, PMO, and docking reveals where the expected pretraining improvements become less useful for the evaluated generation tasks.


Minor: The abstract's opening makes an imprecise causal link between adoption and growth in scale, compute, and data.

We have replaced the opening with: "Chemical language models are increasingly used in de novo molecular design to generate candidates with desired properties." Growth in model size, data, and compute is discussed separately as the motivation for studying scaling.


Minor: I would appreciate it if the authors could provide the code before or during rebuttal.

We have already provided the code for pretraining, IsoFLOP fitting, probing, SAE training, and downstream RL evaluation in the supplementary material.


We hope these clarifications and additional experiments address your concerns, and we would appreciate your consideration of a higher score. Please let us know if any points remain unresolved; we would be happy to clarify them further.

Thank you for the response and additional experiments. I appreciate the authors' efforts to address my concerns.

Points addressed:

  • The prior-only evaluation, fixed-size pretraining comparison, and increased oracle budget provide useful evidence regarding W1 and W3.
  • The clarification about oracle exploitation helps clarify W2.
  • The additional ADMET evaluation and matched-width controls provide useful evidence regarding W4.

Remaining concerns:

  1. Separating pretraining benefits from RL adaptation (W1 and W3). The additional experiments strengthen the findings, but they do not fully separate pretrained-model quality from downstream optimization difficulty. The increased-budget experiment is limited to one task, and the results remain dependent on the selected RL method. I agree that the evidence supports diminishing returns under the evaluated setup, but broader conclusions about the limits of scaling pretrained CLMs remain difficult to establish.

  2. Interpretation of benchmark saturation (W2). I appreciate the clarification that optimizing an oracle does not necessarily mean exploiting its flaws. However, it remains unclear whether the observed saturation reflects limited benefits from additional pretraining or limitations of the downstream optimization and evaluation setup.

  3. Representation scaling and novelty (W4). The additional ADMET results raise a further concern about the paper's central contribution. The paper emphasizes a disconnect between increasingly rich chemical representations and saturating goal-directed generation. However, the matched-width controls and ADMET results show that larger models do not consistently improve transferable representation quality either. This suggests that pretraining loss improves with scale, while downstream performance shows limited or mixed improvements in both representation transfer and molecular generation.

    Similar observations about the limited or inconsistent downstream benefits of scaling CLMs have already been reported in the prior work cited in my original review. This makes the claimed disconnect less convincing as a distinct finding. Could the authors clarify what new insight this work provides beyond extending previously observed downstream scaling limitations to PMO and docking benchmarks?

Reviewer_7TJkOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study examines scaling laws for chemical language models, particularly with respect to pre-training loss, molecular representation learning, and downstream molecular optimization performance. Their primary conclusion is that while pre-training loss and chemical representation learning improve as a model scales up in size, this does not necessarily translate to appreciable gains in molecular optimization performance. The authors address these points through a comprehensive scaling study across different model sizes, architectures, chemical representations, and datasets.

Critical strengths

I found the authors did a good job of being rigorous about preventing data leakage through their exclusion of molecules similar to actives in the PMO benchmarks. Their observation about bits/token instead of bits/molecule was convincing and supported by their data. Most importantly, the degree of exploration across many different axes in their scaling study was incredibly beneficial in making this a much more consequential study than just a simple power law.

Critical weaknesses

While I found their observation of downstream performance on molecular optimization tasks to be interesting, there just wasn't that many comparisons to baseline methods in a field that has many to compare to. In particular, I'd be interested to see if their claims of a plateau hold up with a different optimization technique other than REINVENT. I also think there could be a dimensionality confound in the probing experiments, as the hidden dimension doesn't appear to be accounted for even though it varies given model size. The presentation of the paper overall felt a bit unconventional to me, with the discussion of related work not being very extensive and the presentation of equations being somewhat buried in the text. Figure 6, which a core claim of the paper about the plateau in performance on downstream tasks rests on, does not have any notion of error bars or statistical significance. Presenting Figure 6 with heatmaps instead of tables also does hamper the readability and ability to present such significance tests.

Questions for authors

Clarifying questions:

  • Which RL configuration was selected at each model size, and did the selected lambda and sigma change across the 0.5M - 1B parameter range?
  • Has the change in model dimension across sizes been accounted for in the probe experiments? If not, it should be.
  • Does the PMO total for REINVENT and f-RAG account for the one task that was removed?

Suggestions:

  • What are the error bars on any of the metrics in Figure 6?
  • Can you compare to more baseline methods than just REINVENT and f-RAG?
  • Can you try a different RL algorithm besides REINVENT on the model to see if this affects downstream performance?
  • This is more of a comment than a question, but I think the paper would need a significant change in presentation for a camera-ready version for acceptance. Removing the findings boxes, a more extensive discussion of related work, reworking Figure 6, and decreasing the reliance on inline equations would greatly improve the manuscript's readability.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 7TJk

Thank you for the careful review. All three reviewers recognise the rigour of our systematic study and highlight the disconnect between pretraining improvements and downstream generation. We also appreciate your recognition of our leakage controls and bits-per-molecule evaluation.


W1 & S2: Can you compare to more baseline methods than just REINVENT and f-RAG?

Our docking comparison includes MOOD and GEAM alongside REINVENT and f-RAG, and NovoMolGen outperforms all four on every docking target. Figure 6 and Appendix Table J1 report these comparisons. In PMO, we originally selected f-RAG as a state-of-the-art baseline because it had demonstrated improvements across a broad range of molecular generation methods. The revised PMO comparison now includes additional methods, making the results easier to assess.


Q3: Does the PMO total for REINVENT and f-RAG account for the one task that was removed?

All reported PMO totals for REINVENT, f-RAG, and NovoMolGen use the same 22 tasks, excluding Valsartan_SMARTS. Appendix Tables I1–I2 provide the per-task scores. We have stated this explicitly in the main comparison and Figure 6 caption.


W2 & S3: Can you try a different RL algorithm besides REINVENT on the model to see if this affects downstream performance?

The weak scaling trend is already present without RL: removing policy adaptation does not reveal sustained improvements in prior-only PMO performance. Under the same 10,000-call budget, prior-only performance peaks at 157M and then decreases at 300M and 1B. Thus, the limited benefit of scale cannot be attributed solely to the RL update rule.

Model sizePrior-only AUC Top-10 (22 tasks)After AHC fine-tuningAdded by RL
0.5M10.0616.02+5.96
5M10.7316.45+5.72
32M10.9716.48+5.51
157M11.4016.54+5.14
300M10.8316.64+5.81
1B10.8016.68+5.88

Scores are aggregate AUC Top-10 over the same 22 PMO tasks. AHC adds 5.14–5.96 points, and the fine-tuned 5M model exceeds every prior-only model. We have included the pretrained prior-only evaluation in Section 3.5 and Appendix I.

We want to clarify that our pipeline uses Augmented Hill-Climb (AHC), combined with a penalty term and experience replay (Section 2.4; Appendix E), which performs competitively against more complex policy optimisers such as PPO (Bou et al., 2024; Schulman et al., 2017). It builds upon REINVENT's prior–agent formulation, but the policy updates use the highest-reward fraction of each batch.


Q1: Which RL configuration was selected at each model size, and did the selected lambda and sigma change across the 0.5M–1B parameter range?

The selected configuration was identical at all three tuned sizes, with lambda = 500 and sigma = 500 throughout the 0.5M–1B evaluation. At 32M, 157M, and 300M, we selected configurations based on aggregate performance across two tuning tasks per category over three seeds: Perindopril_MPO and Zaleplon_MPO for PMO; fa7 and jak2 for docking. The remaining settings were batch size 64, learning rate 5e-4, top-k fraction 0.1, and replay batch size 16. We reused the 32M configuration for 0.5M and 5M, and the 300M configuration for 1B. Appendix E now reports the selected settings alongside the search grids.


W3 & Q2: Has the change in model dimension across sizes been accounted for in the probe experiments?

Yes. We ran two additional experiments: randomly initialised models matching each pretrained model's width, and PCA projections of all representations to 64 dimensions, the smallest model's width. The table averages results across property targets and layers.

Model sizedmodeld_\mathrm{model}Untrained R2R^2Learned gain in R2R^2 (trained − untrained)Learned gain (PCA, 64 dims)
0.5M640.482+0.100+0.100
5M1920.599+0.074+0.093
32M3840.639+0.071+0.091
157M7680.673+0.070+0.095

Pretraining improves probe R2R^2 across all model sizes, even after matching dimensionality. However, the untrained baseline also improves with width, and the matched-width gains do not increase consistently with model size. Regularisation is selected by five-fold cross-validation on the training split, with evaluation on OOD held-out molecules. We have therefore revised Section 3.4 and Appendix D to distinguish higher raw probe accuracy from the gain attributable to pretraining.

AuthorsOpenReview ↗

Response to Reviewer 7TJk (continued)

W5 & S1: What are the error bars on the metrics in Figure 6?

We want to point out that Appendix Tables I1 and I2 already report the per-task means and standard deviations across three independent seeds for the PMO benchmark, with the best and second-best models highlighted (based on the Wilcoxon signed-rank test), and that Table J2 reports means and standard deviations across three runs for the docking results. We have brought the PMO uncertainty from Tables I1–I2 into the main comparison.


W4 & S4: Remove the findings boxes, expand related work, rework Figure 6, and reduce inline equations.

We have revised the presentation throughout: removed the findings boxes, expanded the main-text related work, replaced the Figure 6 heatmaps with readable tables, and moved the equations into numbered displays. The revised comparison makes task coverage and uncertainty across model seeds easier to read.


We appreciate these suggestions and hope the additional comparisons, experiments, and presentation changes address your concerns. In light of these revisions, we would be grateful if you could consider raising your score.

Reviewer_7TJkOpenReview ↗

Response to Author Responses

Thank you for the detailed response and for running the additional experiments. The width-matched probing controls and the prior-only evaluation are both substantive. I still have further clarifying questions before I am willing to raise my score.

Uncertainty on the PMO and docking totals

Tables I1–I2 report per-task means and standard deviations, but the totals carry none. Please report a propagated standard deviation or bootstrap confidence interval for each total, and a paired Wilcox signed-rank test across the 22 tasks for 5M versus 1B specifically. The same applies to the docking totals, where the spread across sizes is 0.2 kcal/mol.

Oracle calls to a fixed threshold as a function of model size

Figures I5 and J1 already contain the reward trajectories. Could you re-report these as the number of oracle calls required to reach a fixed top-10 reward against model size?

RL algorithm (W2/S3)

I would like to note that your table is non-monotonic in both columns. Some comment on that pattern would be helpful.

Hyperparameters (Q1)

Thank you for addressing this. λ = 500 being selected identically at all three tuned sizes largely addresses my concern about drift. One remaining concern is that σ = 500 is the lowest value in your grid, and it's possible that a lower one may actually be more appropriate. A single additional point (σ = 100 or 250) would be of interest to me.

Reviewer_R5CBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a large-scale investigation of scaling laws in Chemical Language Models (CLMs), conducting over 30,000 experiments across model sizes, molecular representations, tokenizations, datasets, and architectures. The results reveal that while pretraining loss and learned chemical representations consistently improve with scale, downstream goal-directed molecular generation exhibits diminishing returns beyond approximately 5M parameters. The proposed NovoMolGen model family achieves state-of-the-art performance on molecular optimization and docking benchmarks, highlighting the disconnect between representation learning and practical molecular design.

Critical strengths
  • Conducts over 30,000 experiments across diverse model configurations, datasets, and architectures, providing extensive insights into CLM scaling behavior.
  • Identifies a significant disconnect between improvements in pretraining loss, chemical representation quality, and downstream molecular generation performance.
Critical weaknesses
  • The work primarily investigates existing architectures and training strategies without introducing fundamentally new modeling techniques.
  • Although the paper identifies a disconnect between improved chemical representations and saturated downstream generation performance, it lacks controlled experiments to establish the underlying causes of this discrepancy.
Questions for authors

please refer to the weakness part

4 comments
AuthorsOpenReview ↗

Response to Reviewer R5CB

Thank you for the review. All three reviewers recognise the rigour and systematic design of our study and highlight the disconnect between pretraining improvements and downstream generation.


W1: The work primarily investigates existing architectures and training strategies without introducing fundamentally new modeling techniques.

We introduce NovoMolGen, a family of 42 CLMs that outperforms the evaluated baselines on aggregate PMO and all five docking targets, together with a controlled account of why further scaling yields little downstream return. At only 5M parameters, NovoMolGen exceeds f-RAG on PMO (16.45 versus 16.30) and achieves a better total docking score (−65.8 versus −59.8 kcal/mol). This result comes from a systematically evaluated training and optimisation pipeline spanning molecular encoding, tokenisation, the corpus, the architecture, and the RL configuration.

The paper contributes more than an architecture comparison. We establish compute-controlled scaling relationships across molecular encodings, using bits per molecule to compare likelihoods across different sequence lengths and IsoFLOP analysis to identify the parameter–token allocation that minimises pretraining loss. Connecting this analysis to generation yields a practical result: increasing NovoMolGen from 5M to 1B parameters improves aggregate PMO by only 0.23 points. A compact model therefore captures nearly all the observed downstream benefit, despite continued improvements in pretraining loss. Keeping established backbone families fixed makes these scaling comparisons interpretable.

Our layer-wise probes and BatchTopK SAEs also connect the development of chemical features to downstream optimisation. Chemical syntax emerges early, while measured properties and functional-group features become accessible later, with little additional PMO benefit. Leakage-controlled OOD evaluation, fixed-size checkpoint comparisons, and the added prior-only control examine where improvements in likelihood and chemical feature accessibility fail to improve generation. Together, these analyses provide evidence about what current benchmarks reward and how to allocate training resources. We have made these methodological and empirical contributions explicit in the introduction.


W2: Although the paper identifies a disconnect between improved chemical representations and saturated downstream generation performance, it lacks controlled experiments to establish the underlying causes of this discrepancy.

Our controlled experiments support benchmark saturation as an explanation for this discrepancy. Additional pretraining provides little downstream benefit, even with a fixed policy size, and removing RL does not restore sustained gains from scaling. The submitted manuscript already includes a fixed-size experiment in which the 32M checkpoint pretrained for only 500 steps achieves 16.80 PMO after fine-tuning, compared with 16.48 after 75,000 steps (Figure I3). Property representations develop during this additional pretraining (Figure 5b), yet generation does not improve. Because the policy size is unchanged, increasing the number of trainable parameters cannot account for this result.

The manuscript also varies molecular representation, tokenisation, and architecture individually under matched pretraining budgets. Diminishing downstream returns recur across these choices (Figure I1), establishing that the finding extends beyond a single configuration. To further isolate the role of RL adaptation, we evaluated every pretrained prior without RL, using the same 10,000-call oracle budget:

Model sizePrior-only AUC Top-10 (22 tasks)After AHC fine-tuningAdded by RL
0.5M10.0616.02+5.96
5M10.7316.45+5.72
32M10.9716.48+5.51
157M11.4016.54+5.14
300M10.8316.64+5.81
1B10.8016.68+5.88

Scores are aggregate AUC Top-10 over the same 22 tasks. Removing RL does not reveal sustained gains across model sizes. In contrast, task-specific optimisation adds 5.14–5.96 points, and the fine-tuned 5M model exceeds every prior-only model.

AuthorsOpenReview ↗

Response to Reviewer R5CB (continued)

Together, these experiments support our hypothesis that current benchmarks are largely saturated with respect to the benefits of additional pretraining. Once a compact or early-trained prior supports effective task-specific optimisation, improvements in pretraining loss and measured chemical representations yield little additional improvement in generation performance. This limited sensitivity helps explain the discrepancy and raises a substantive question about whether current benchmarks can distinguish advances in pretrained chemical knowledge. Establishing this limitation is a central contribution of our study and motivates the more demanding evaluations discussed in Section 4. We have added the prior-only experiment to Section 3.5 and Appendix I and made this interpretation explicit in the discussion.


We hope this clarifies the methodological contributions and the experimental evidence underlying our explanation of the discrepancy. We would appreciate hearing whether these address your concerns, and we would be grateful if you would consider revising your score.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer R5CB,

As the discussion period closes on November 18, we would appreciate knowing whether our rebuttal resolved your concerns or if there is anything further we can clarify before the deadline.

If you find that our responses have addressed your concerns, we would sincerely appreciate it if you could update your score accordingly.

Thank you for your time and consideration.

Best regards,

The Authors

Thanks for the clarification and additional experiments to address my concerns. I will raise my score to 3.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Reviewer XVv2 (additional clarifications)

Thank you for the follow-up questions and code audit. We address each point below:


Incorrect reward assignment

Our RL implementation assigns each oracle reward to the correct molecule. The claimed misalignment does not occur during training. RL scoring uses filter=False, which preserves the input batch's length and ordering. Invalid strings remain at their original indices and receive reward 0.0. Thus, [molecule A, invalid string, molecule B] receives [reward A, 0.0, reward B], with no index shift.

The filter=True option removes invalid entries by applying a Boolean mask to evaluate validity and is modified to False during RL training. Its behaviour, therefore, does not support the claim of an incorrect reward assignment. We ask that this claim be corrected. If a separate issue remains, please identify the specific training code and input that demonstrate the mismatch.


Docking reward and evaluation protocol

Our reported docking evaluation adapts the protocol from Tartarus (Nigam et al.; NeurIPS, 2023) and applies the stated drug-likeness and novelty criteria. During RL fine-tuning, training uses batch-min-max-normalised docking scores alongside the drug-likeness constraints (QED > 0.5 and SA < 5). During final evaluation, the novelty threshold (Tanimoto similarity < 0.4 to ZINC-250k) is applied alongside these drug-likeness criteria when calculating the reported docking scores and hit ratios. The absence of novelty filtering in the training reward, therefore, does not mean it was omitted from evaluation. We will clarify these stages in the documentation. If a discrepancy remains, please identify the reported metric and the implementation step that differs from its stated definition.


IsoFLOP analysis, RL configurations, interpretability, and dataset documentation

The supplementary material included only a minimal reference implementation due to size constraints imposed by large datasets and checkpoints. The updated release will include the full IsoFLOP grid, loss-surface fitting and bootstrap scripts, replay-buffer initialisation and sizes, regularisation settings, pooling, and SAE training configurations.

The release will also include the matched-width probing and TDC ADMET experiments added during the discussion. Syntax probing (a validity property of the dataset) was already implemented as a binary classification between corrupted and uncorrupted SMILES and will be included in the README. The docstring reference concerned only the expected data format, using a publicly available dataset as an example. We will also publicly release all pretrained checkpoints and full datasets.

AuthorsOpenReview ↗

Response to Reviewer XVv2 (additional clarifications)

Separating pretraining quality from RL fine-tuning

Our original and additional experiments consistently demonstrate diminishing returns to scaling pretraining in the evaluated setup, rather than a universal limit on pretrained CLMs. The zero-shot experiment reported in our previous response evaluates samples directly from each pretrained model, with no RL, policy updates, or downstream adaptation. It therefore measures the pretrained prior's generation performance separately from the difficulty of RL optimisation. The absence of scaling gains in this experiment cannot be attributed to larger policies requiring more feedback.

The fixed-size pretraining comparison and increased oracle-budget experiments provide compelling evidence. The latter now covers both Perindopril_MPO and Zaleplon_MPO, showing no consistent advantage for 300M over 32M across the two tokenisations at 50,000 calls (Table I6).

TaskModel / TokenisationAUC@10kAUC@50k
Perindopril_MPO32M / Atom0.637 ± 0.0500.675 ± 0.012
Perindopril_MPO32M / BPE0.635 ± 0.0180.706 ± 0.027
Perindopril_MPO300M / Atom0.668 ± 0.0110.724 ± 0.063
Perindopril_MPO300M / BPE0.669 ± 0.0490.691 ± 0.051
Zaleplon_MPO32M / Atom0.579 ± 0.0230.631 ± 0.020
Zaleplon_MPO32M / BPE0.611 ± 0.0270.627 ± 0.058
Zaleplon_MPO300M / Atom0.599 ± 0.0000.618 ± 0.022
Zaleplon_MPO300M / BPE0.574 ± 0.0100.591 ± 0.041

A different optimiser could expose additional benefits, but that possibility does not contradict the demonstrated limited return from scaling under these protocols. Since you agree that the evidence supports this finding, could you identify which specific conclusion in the revised manuscript remains unsupported, and what additional experiment would resolve the remaining concern?


Interpreting benchmark saturation

Uncertainty and sample-efficiency analyses (Tables I3–I4, J3–J4; Figures I7, J2) confirm benchmark saturation. From 5M to 1B, changes in total docking and PMO scores stay within single-run SD and vanish under BPE. Scaling 200x beyond 5M yields only a 15% gain in sample efficiency. Once a compact prior (~5M) generates valid, diverse molecules, next-token pretraining does not allocate additional budget to target-specific actives, thereby making RL the bottleneck and motivating harder benchmarks.


Representation scaling and novelty

Our contribution is to show that chemical features develop during pretraining without corresponding improvements in goal-directed generation, even when model size and representation width remain fixed. The matched-width and ADMET experiments clarify this finding. Pretraining provides useful chemical information, but larger models do not consistently improve its transfer learning capabilities. At a fixed size of 32M, property probes and SAE analyses monitor the development of chemical features during pretraining (Figures 5b and 5e). Notably, PMO achieves a score of 16.80 after just 500 pretraining steps, whereas it reaches only 16.48 after 75,000 steps (Figure I3). This discrepancy cannot be attributed to an increase in representation width or policy size.

In contrast to static transfer studies (Xu et al., 2026; Sagawa & Kojima, 2026), our research establishes a connection between compute allocation, the learning of more complex chemical features, and the performance of de novo molecule generation through matched-budget comparisons and experiments that vary pretraining duration, reinforcement learning adaptation, and oracle budget. The key insight derived from this is that achieving competitive goal-directed generation performance within these protocols necessitates significantly less pretraining than ongoing improvements in likelihood and assessed chemical features would suggest.


We hope these additions address your concerns and kindly ask you to reconsider your score.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer XVv2,

As the discussion period closes on November 18, we would appreciate knowing whether our rebuttal resolved your concerns or if there is anything further we can clarify before the deadline.

If you find that our responses have addressed your concerns, we would sincerely appreciate it if you could update your score accordingly.

Thank you for your time and consideration.

Best regards,

The Authors

AuthorsOpenReview ↗

Summary of Contributions and Revisions

Dear Area Chair and Reviewers,

We sincerely thank you for the careful evaluation and feedback, which strengthened the paper. We are grateful that reviewers R5CB and 7TJk found our revisions convincing and raised their scores.

We appreciate reviewers R5CB, 7TJk, and XVv2 for recognising the breadth and systematic design of our scaling study, as well as the disconnect between pretraining improvements and de novo goal-directed generation. Reviewer R5CB noted the insights into CLM scaling from over 30,000 experiments. Reviewer 7TJk recognised our pretraining data leakage control and bits-per-molecule evaluation, while Reviewer XVv2 valued our thorough analysis and SAE interpretability. These cover our main contributions:

  • We establish compute-controlled scaling relationships across molecular representations, using bits per molecule to compare likelihoods across sequence lengths and IsoFLOP analysis to identify loss-minimising parameter-token allocations.

  • Connecting this to goal-directed generation yields a practical result: scaling NovoMolGen from 5M to 1B improves aggregate PMO by only 0.230.23 points and sample efficiency by only 15%. Once a compact prior (~5M) generates valid, diverse molecules, next-token pretraining does not allocate budget to target-specific actives, making RL the bottleneck and motivating harder benchmarks.

  • We show that chemical features develop during pretraining without corresponding gains in goal-directed generation.

  • Unlike static transfer studies (Xu et al., 2026; Sagawa & Kojima, 2026), we connect compute allocation, chemical feature learning, and de novo generation via matched-budget experiments varying pretraining duration, RL adaptation, and oracle budget.

The reviewers' questions highlighted areas for strengthening controls, statistical reporting, and presentation. We ran additional experiments, clarified claims, and added these revisions:

Reviewer R5CB: Methodological contributions and controlled evidence for the representation–generation disconnect.

  1. We stated our methodological and empirical contributions clearly, connecting the scaling frontier to layer-wise probes, BatchTopK SAEs, and generation (Section 1).

  2. We added a prior-only (zero-shot) evaluation across sizes (10k-call budget): prior-only PMO peaks at 157M (11.4011.40) and declines at 300M (10.8310.83) and 1B (10.8010.80), while AHC adds 5.145.14--5.965.96 points and fine-tuned 5M exceeds every prior-only model (Section 3.5, Table I5).

  3. We connected this with our 32M checkpoint experiment (16.8016.80 PMO at 500 steps vs 16.4816.48 at 75k steps despite developing property representations; Figures 5b, I3) to support benchmark saturation (Section 4).

Reviewer 7TJk: Baselines, significance, sample efficiency, width confounds, RL hyperparameters, and presentation.

  1. We expanded PMO baselines and stated that all totals use the same 22 tasks, excluding Valsartan_SMARTS (Section 3.5, Tables 1 and I1--I2).

  2. We reported propagated SDs, 95% bootstrap CIs, and paired Wilcoxon/tt-tests for 5M vs 1B (Tables I3--I4, J3--J4), confirming gains stay within single-run SD and vanish under BPE, and recomputed oracle calls to fixed thresholds (∼15%\sim 15\% gain from 200×200\times scaling; Figures I7, J2).

  3. We controlled for width via width-matched untrained baselines and 64D PCA. Pretraining improves probe R2R^2 while matched-width gains plateau across scale (Section 3.4, Table D1).

  4. We ablated σ∈{100,250,500}\sigma \in \{100, 250, 500\}, confirming performance improvement to σ=500\sigma=500, and explained prior-only non-monotonicity as sampling variance under leakage control (Table E2, Appendix I.3).

  5. We removed findings boxes, expanded related work (Section 1), replaced the former Figure 6 heatmaps with tables including uncertainty (Table 1), and moved equations to numbered displays (Equations 1--3).

Reviewer XVv2: Isolating pretraining from RL, oracle exploitation, transfer studies, and abstract clarity.

  1. We added a 5×5\times oracle-budget experiment (50k calls) on Perindopril_MPO and Zaleplon_MPO (32M vs 300M, atom-level and BPE), showing no larger-model advantage (Section 3.5, Table I6).

  2. We evaluated 22 TDC ADMET tasks against width-matched untrained baselines. Pretrained model representations improve performance on most tasks, but plateau beyond 5M (Section 3.4, Table D3).

  3. We distinguished optimising a scoring function from exploiting oracle flaws, linked saturation to function-informed pretraining and harder evaluations. Our code response explains reward assignment for invalid strings and the docking novelty filter.

The reviewer feedback has significantly strengthened the experiments, statistical rigour and presentation of our study. We have incorporated these analyses into the revised manuscript and will publicly release all checkpoints, the full codebase and leakage-controlled datasets.

Thank you again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.