- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether scaling chemical language models (CLMs) improves downstream goal-directed molecular generation. The authors train decoder-only and encoder-decoder CLMs across model sizes, molecular representations, tokenizations, and pretraining corpora using self-supervised sequence modeling, then fine-tune them with reinforcement learning for targeted generation.
They evaluate PMO and protein-ligand docking tasks, considering both generated-molecule quality and sample efficiency. The main result is that pretraining loss and several representation probes continue to improve with scale, while downstream generation performance shows only marginal gains beyond relatively small models.
Critical strengths
- The paper presents a large and systematic study across model sizes, architectures, molecular representations, tokenizations, and datasets.
- The analysis is thorough, covering pretraining loss, OOD behavior, unconstrained generation, linear probes, sparse autoencoders, PMO, and docking.
- The observation that representation-related metrics continue improving while downstream generation saturates is interesting.
- The SAE analysis adds an interpretability angle beyond standard scaling and probing experiments.
Critical weaknesses
1. The central research question is not convincingly answered
The paper asks a difficult question: whether scaling the pretrained CLM itself improves goal-directed molecular generation. I am not convinced that the chosen experimental design can answer this question cleanly, because the measured downstream performance reflects the full pipeline: pretrained model + RL fine-tuning + oracle + oracle budget. Therefore, a plateau in PMO or docking performance does not necessarily imply a plateau in the useful capability of the pretrained model itself. The following two points explain why I find this difficult to interpret.
2. The downstream evaluation may not reliably reflect the benefit of a better pretrained model
The RL stage relies on computational oracle scores. This is understandable, since obtaining experimental labels during iterative RL would be impractical, but it creates a risk that the agent mainly learns to exploit the behavior of the oracle rather than benefiting from richer pretrained chemical knowledge.
The authors partly acknowledge this issue by noting that current benchmarks may mainly measure oracle exploitation and by suggesting harder tasks, unseen targets, and stricter oracle budgets as better tests of whether larger models help.
3. Sample efficiency mixes pretrained-model quality with RL adaptation difficulty
I do think the PMO AUC metric provides a useful insight: sample efficiency changes only modestly beyond a certain model size. However, this does not by itself show whether the larger pretrained model is better or not.
A larger model may provide a better pretrained prior, but it also introduces a larger trainable policy that may require more feedback to adapt efficiently. In other words: sample efficiency depends on both prior quality and RL adaptation difficulty. Therefore, similar sample efficiency between a 5M and 1B model could mean either that scaling does not help, or that the benefit of the larger pretrained model is offset by a higher fine-tuning cost.
For me, this leaves the central research question unresolved. The paper convincingly demonstrates limited scaling gains under the specific RL-and-oracle evaluation setup, but I am less convinced that this is sufficient to conclude that scaling the pretrained CLM itself does not improve goal-directed molecular generation. One could instead evaluate downstream quality using more standard predictive metrics such as MAE or F1 score, but this immediately runs into the limitation discussed in Point 2. That's why I think the research question posed in the paper is inherently difficult to answer, and the current experimental design does not resolve it convincingly.
4. Some claims are stronger than the evidence supports
The paper describes larger models as learning "richer chemical representations." The probe and SAE results do show that selected properties and common functional groups become more accessible, but this does not necessarily establish a generally richer or more transferable molecular representation. The SAE functional-group analysis is also limited to only ten common groups.
A more direct way to evaluate representation quality is to use pretrained models at different scales as molecular representation extractors and test whether those representations improve downstream molecular property prediction. Recent work has already studied this setup. For example, How Well Do Large-Scale Chemical Language Models Transfer to Downstream Tasks? evaluates CLMs at different scales using both frozen linear probes and full fine-tuning on molecular property prediction tasks, and finds that lower pretraining loss does not consistently translate into better downstream performance. Similarly, Unveiling Scaling Behaviors in Molecular Language Models: Effects of Model Size, Data, and Representation studies scaling across pretraining and downstream transfer and reports that downstream scaling behavior depends strongly on model size, data, and molecular representation. These results make me cautious about interpreting improved probe performance in the current paper as evidence that larger CLMs necessarily learn broadly better molecular representations.
Similarly, the finding that pretraining loss follows compute-optimal scaling is useful characterization, but qualitatively expected for transformer language models trained with next-token objectives.
Minor:
- The first sentence in the abstract: “Chemical Language Models (CLMs) are increasingly used in de novo drug design, driven by recent growth in model scale, compute, and dataset size.” is conceptually imprecise. The causal link is too loose. Growth in scale, compute, and data may enable better CLMs, but those are not themselves the direct reason CLMs are increasingly used. A sharper sentence would connect adoption to demonstrated utility, while scale/compute/data are enabling factors.
- Code is promised for release, but I would appreciate it if the authors could provide the code before or during rebuttal.
Questions for authors
Clarification on representation scaling.
The paper argues that larger models learn "richer chemical representations," supported by improved linear probes and SAE analyses. How do the authors reconcile this with recent work reporting mixed or setup-dependent downstream transfer when scaling chemical language models and data (especially How Well Do Large-Scale Chemical Language Models Transfer to Downstream Tasks?)?
3 comments
Response to Reviewer XVv2 (continued)
W4 & Q1: How do the authors reconcile this with recent work reporting mixed or setup-dependent downstream transfer when scaling chemical language models and data?
Our findings are consistent with mixed downstream scaling, which we already discuss in the introduction through Xu et al. (2026) and which directly motivates our study. Improved accessibility of the properties measured by our probes does not imply uniformly better downstream transfer, consistent with Sagawa & Kojima (2026).
Our additional evaluation on 22 TDC ADMET tasks confirms this distinction: pretrained representations outperform width-matched untrained models on 18–19 tasks, but matched-width evaluation shows no consistent improvement beyond 5M parameters. We have added these results to Section 3.4 and Appendix D and clarified that our representation claims concern the specific chemical information measured by our probes.
W4: The SAE functional-group analysis is also limited to only ten common groups.
The ten functional groups provide a targeted substructure evaluation. Our interpretability analysis also examines 36 property targets across layers and training checkpoints. We selected common, SMARTS-defined groups to compare their accessibility in SAE features and residual-stream neurons using five-fold stratified cross-validation. This follows the concept-based evaluation approach of Cohen et al. (2025), who examined 14 functional groups. SAE features outperform residual-stream neurons on seven of our ten groups (Table D2). Together with the property analyses, these experiments characterise which chemical information becomes accessible during pretraining, while the additional ADMET experiments separately assess downstream transfer.
W4: Compute-optimal pretraining scaling is qualitatively expected for transformers.
The contribution is the measured chemical scaling frontier and its failure to predict comparable gains in generation. IsoFLOP analysis quantifies the allocation of compute between parameters and tokens, while bits per molecule enables comparison across encodings. Connecting that frontier to probes, training checkpoints, PMO, and docking reveals where the expected pretraining improvements become less useful for the evaluated generation tasks.
Minor: The abstract's opening makes an imprecise causal link between adoption and growth in scale, compute, and data.
We have replaced the opening with: "Chemical language models are increasingly used in de novo molecular design to generate candidates with desired properties." Growth in model size, data, and compute is discussed separately as the motivation for studying scaling.
Minor: I would appreciate it if the authors could provide the code before or during rebuttal.
We have already provided the code for pretraining, IsoFLOP fitting, probing, SAE training, and downstream RL evaluation in the supplementary material.
We hope these clarifications and additional experiments address your concerns, and we would appreciate your consideration of a higher score. Please let us know if any points remain unresolved; we would be happy to clarify them further.
Thank you for the response and additional experiments. I appreciate the authors' efforts to address my concerns.
Points addressed:
- The prior-only evaluation, fixed-size pretraining comparison, and increased oracle budget provide useful evidence regarding W1 and W3.
- The clarification about oracle exploitation helps clarify W2.
- The additional ADMET evaluation and matched-width controls provide useful evidence regarding W4.
Remaining concerns:
-
Separating pretraining benefits from RL adaptation (W1 and W3). The additional experiments strengthen the findings, but they do not fully separate pretrained-model quality from downstream optimization difficulty. The increased-budget experiment is limited to one task, and the results remain dependent on the selected RL method. I agree that the evidence supports diminishing returns under the evaluated setup, but broader conclusions about the limits of scaling pretrained CLMs remain difficult to establish.
-
Interpretation of benchmark saturation (W2). I appreciate the clarification that optimizing an oracle does not necessarily mean exploiting its flaws. However, it remains unclear whether the observed saturation reflects limited benefits from additional pretraining or limitations of the downstream optimization and evaluation setup.
-
Representation scaling and novelty (W4). The additional ADMET results raise a further concern about the paper's central contribution. The paper emphasizes a disconnect between increasingly rich chemical representations and saturating goal-directed generation. However, the matched-width controls and ADMET results show that larger models do not consistently improve transferable representation quality either. This suggests that pretraining loss improves with scale, while downstream performance shows limited or mixed improvements in both representation transfer and molecular generation.
Similar observations about the limited or inconsistent downstream benefits of scaling CLMs have already been reported in the prior work cited in my original review. This makes the claimed disconnect less convincing as a distinct finding. Could the authors clarify what new insight this work provides beyond extending previously observed downstream scaling limitations to PMO and docking benchmarks?
Response to Reviewer XVv2
Thank you for the careful review. All three reviewers recognise the rigour and systematic design of our study and highlight the disconnect between pretraining improvements and downstream generation. We also appreciate your recognition of the interpretability analysis.
Our evaluation addresses whether improved pretraining translates into better goal-directed generation, and our experiments show that larger policies that require more feedback alone cannot explain the limited gains. Figure 1 captures the central result: increasing model size 200-fold, from 5M to 1B, improves aggregate PMO only from 16.45 to 16.68 and total docking score from −65.8 to −66.2 kcal/mol, despite lower pretraining loss and greater accessibility of chemical information, as shown by layer-wise probes and sparse autoencoder analysis. These findings support our hypothesis that current benchmarks are largely saturated with respect to the benefits of additional pretraining.
Zero-shot evaluation removes policy adaptation entirely. We sampled directly from each pretrained prior without RL, using the same 10,000-call oracle budget:
Prior-only performance peaks at 157M and subsequently declines; removing adaptation does not reveal sustained gains across the full size range. Task-specific optimisation contributes substantially larger improvements. The discrepancy also occurs within a fixed 32M policy: extending pretraining from 500 to 75,000 steps improves measured representations but leaves downstream PMO unimproved (16.80 versus 16.48; Figures 5b and I3), showing that increased policy size cannot be the sole explanation.
Providing five times more oracle feedback does not reveal a consistent larger-model advantage across the two tokenisations tested. We increased the budget on
Perindopril_MPOfrom 10,000 to 50,000 calls:Additional feedback improves every configuration, but at 50k the 300M model leads with atom-level tokenisation while the 32M model leads with BPE. Although this experiment covers a single task, it directly examines whether additional feedback consistently yields an advantage for the larger pretrained model. We have incorporated both additional experiments into Section 3.5 and Appendix I and clarified the benchmark-saturation hypothesis in Section 4.
We agree that separating the benefits of pretraining from downstream optimisation is a difficult question; addressing it is central to this study. We consider fixed-budget generation, complemented by prior-only evaluation and increased-budget experiments, the most practical setup for assessing whether additional pretraining improves molecular design. Establishing that current benchmarks capture little of this additional benefit is itself a valuable finding for the community. We would welcome your suggestions for a concrete alternative setup that could more effectively isolate these benefits.
Our results support the concern that current benchmarks reward optimisation of the supplied oracle more than the additional chemical information acquired through further pretraining. A compact or early-trained prior already supports strong performance, while improved pretraining loss and measured chemical representations provide little additional downstream benefit. This limited sensitivity is central to our interpretation of benchmark saturation in Section 4.
Here, "oracle exploitation" means optimising the supplied scoring function; our experiments do not establish that the agent exploits flaws in that function. Higher oracle scores also do not necessarily imply better experimental activity. We have clarified this distinction in Section 4 and connected it to our recommendations for function-informed pretraining and downstream evaluations that better test the value of pretrained chemical knowledge.