acceptodds
Under review as a conference paper at ICLR 2027

SMolLM: Small Language Models Learn Small Molecular Grammar

Abstract

Language models for molecular design have scaled to hundreds of millions of parameters, yet how they learn chemical grammar is poorly understood. We train SMolLM, a 53K-parameter weight-shared transformer, to generate novel SMILES with 95% validity on the ZINC-250K drug-like-molecule benchmark, outperforming a standard GPT with 10 times more parameters. Mechanistically, the same block resolves SMILES constraints across passes in a fixed hierarchy: brackets first, rings second, and valence last, as shown by error classification and linear probing, with ablation isolating the bracket-matching head. Together, these results yield a compact, mechanistically interpretable molecular generator and a testbed for studying iterative computation in formal-language domains.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, weight sharing, looped transformers, molecular generation, SMILES, causal ablation, formal languages

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iTVvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper applies a looped Transformer to SMILES generation, reporting 95.3% validity on ZINC-250K with 53K parameters. It also analyzes errors, internal representations, and attention-head ablations to argue that repeated computation resolves brackets, rings, and valence in a fixed order.

Critical strengths

The 53K model achieves high validity with fewer parameters than the reported unshared baselines. The novelty and diversity checks, together with the additional architectural controls, make this a useful empirical demonstration of compact SMILES generation.

Critical weaknesses
  1. Limited contribution and an unclear central concept. The paper does not clearly explain what "small" means in "Small Molecular Grammar." It neither defines grammar size nor connects this phrase to a measurable research question. The demonstrated result is primarily a favorable parameter–validity trade-off on ZINC-250K. Although the additional generation metrics are useful, it remains unclear what broader insight they provide into molecular grammar. This distinction could also be made clearer in the presentation.

  2. The claimed bracket–ring–valence computational order is not sufficiently supported. Table 1 shows that all three error types decrease sharply between depths 1 and 2. This supports the narrower observation that bracket errors become rare earlier, while valence errors remain more prevalent and continue to improve, but does not establish that the model first processes brackets, then rings, and finally valence. Table 7 suggests timing differences but does not establish a computational dependency. Since each invalid string is assigned to a single error category, the classification procedure also needs clarification. Head ablation provides a targeted intervention for bracket-related behavior, but does not test the proposed three-stage order. The single-head localization is also reported for the 206K model, whereas the abstract emphasizes the 53K model without clearly distinguishing the scope of these results.

Questions for authors
  1. Could the authors clarify what "small" means in "Small Molecular Grammar," and what insight into grammar learning is supported beyond parameter-efficient generation on this dataset?

  2. What evidence distinguishes a fixed computational order from different rates of error reduction?

2 comments

Thank you for your review and for recognizing that the 53K model achieves high validity with fewer parameters than the unshared baselines and that our novelty, diversity, and architectural controls provide a useful empirical demonstration of compact SMILES generation. New multi-label error, dependency, prefix-count probe, and 53K/206K ablation controls test each concern directly and calibrate our claim to a depth- and pass-dependent progression.

1. Error classification and computational order vs. error-reduction rates (Weakness 2, Question 2)

We agree that depth error rates and probe timing alone do not distinguish a fixed computational order from different rates of error reduction, which requires causal dependency across passes and decodability above prefix token counting. To test all three points directly, we replaced single-label buckets with a multi-label classifier, intervened on pass-1-to-pass-4 dependency, and added prefix-count probe controls. Together, these tests show pass-specific ring computation beyond surface error rates without uniform stage-to-stage dependency across seeds. We revised the abstract and Section 4 to replace the fixed three-stage order claim with this depth- and pass-dependent progression.

  • Multi-label error classification (Section 4, Table 2). We now score branch, ring, parse, termination, and chemistry failures independently to remove single-label precedence bias (syntax on all 3,000 strings and chemistry on the 1,924→2,984/3,000 unsanitized-RDKit-parseable subset). Across three seeds, validity rises from 34.67 ± 1.36% at depth 1 to 89.30 ± 0.61% at depth 2 (branch 21.40% → 2.10%, ring 18.40% → 2.13%, parse 35.87% → 3.93%), and to 96.97 ± 0.58%, 97.90 ± 0.60%, and 98.50 ± 0.17% at depths 4, 8, and 16, with chemistry failures mostly kekulization/aromaticity, and early-pass final-head readouts truncate (96.33%, 62.20%, 28.07% at passes 1–3 in WS-53K).
  • Direct causal test of bracket-to-ring dependency (Section 4, Appendix I). If the model resolves constraints in a dependent chain rather than independently at different rates, ablating the pass-1 bracket head on a fixed prefix should degrade pass-4 ring closure. Across all six ZINC checkpoints (53K/206K, three seeds each), pass-4 ring-closing log-probability drops as predicted in 4/6 runs (4/5 localizing seeds) vs. KL-matched random controls, but shifts positively in 2/6 (including non-localizing WS-53K seed 42, pooled +0.00429 nats, 95% CI [−0.0912, +0.1149]). Because two seeds shift positively and the pooled CI spans zero, we drop the fixed stage-to-stage causal claim.
  • Probes with prefix-count controls (Section 4, Table 3). While the cross-stage causal chain is inconclusive, within-model probes show that later passes compute ring state beyond surface token counting. Prefix token counts nearly match pass-3 bracket-depth probing (macro-F1 0.912 vs. 0.915/0.916 in WS-53K/WS-206K), whereas pass-3 open-ring state (0.930/0.906 vs. 0.413) and pass-8 next-ring-closure (0.939/0.951 vs. 0.608) far exceed token counting.

2. Scope of causal head localization across 53K and 206K (Weakness 2)

We resolved both points directly by running the full 32-cell per-head, per-pass ablation on the 53K model across three seeds on ZINC and ChEMBL and by restricting our head-ablation claim to bracket matching (having tested stage-to-stage dependency above). Single-head pass-1 bracket localization replicates in the 53K model (5/6 WS-53K runs, +10 pp bracket errors on ZINC and +18.4 pp on ChEMBL) at the same rate as the 206K model (5/6 WS-206K runs, +23 pp on ZINC and +19.3 pp on ChEMBL), with one non-localizing seed per size (ZINC WS-53K seed 42, ChEMBL WS-206K seed 43). We revised the abstract and Section 4.3 (Table 4, Appendix H Table 14) to report the 53K and 206K models side by side, state this 10/12 localization scope explicitly, and restrict our causal ablation claim to bracket matching.

3. “Small-molecule grammar” and contribution beyond compactness (Weakness 1, Question 1)

“Small-molecule” refers to the standard drug-like compound class (MW ≤ 500 Da, as in ZINC, MOSES, and ChEMBL) rather than formal grammar size, and we hyphenated the title to make this explicit. Beyond compact generation across ZINC, MOSES, and ChEMBL, reusing the same weights at every pass separates computation across depth from layer-specific weights, showing rising ring-state and pass-8 ring-closure decodability above prefix-count controls and a reproducible early bracket head. We revised the title to “SMolLM: Small Language Models Learn Small-Molecule Grammar” and Section 1 to define this term directly.

We hope the multi-label classifier, dependency test, probe controls, and 53K/206K localization resolve your concerns, and we welcome your feedback.

Thank you for the response. My main remaining concern is the novelty and significance of the contribution. With the original claim of a fixed bracket-ring-valence ordering withdrawn, I find it difficult to identify the central new insight into grammar learning beyond applying an established weight-shared architecture to SMILES and analyzing its intermediate representations. The revised abstract and introduction also remain difficult to follow, and I find the term "decodability" hard to understand. Given the acknowledged use of AI writing assistance, I suggest that the authors review the revised text more carefully.

Reviewer_rz7GOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SMolLM, a 53K-parameter weight-shared (looped) transformer block that is applied for eight forward passes per emitted token and trained on ZINC-250K SMILES. The generation claim is that this model reaches 95.3% RDKit validity, beats an identically trained 527K-parameter unshared GPT (87.6%) at 10× fewer parameters, comes within 5 points of a 3.2M GPT, and dominates the sub-megaparameter validity/parameter Pareto frontier. The second half is a mechanistic interpretability study arguing that the shared block resolves SMILES constraints in a fixed order across passes: brackets by pass 2, rings by pass 4, valence by pass 8.

Critical strengths

The strongest aspect is the conceptual setup, which is well motivated. SMILES combines two properties that are rare together: every intermediate state can be checked symbolically and exactly, and the constraints form a natural scope hierarchy from local bracket matching to long-range ring-digit pairing to global valence consistency.

Critical weaknesses

(1) The first critical weakness is that the compact-generation claim has no external coordinate system and its one fair internal control is deferred. 95.3% validity is not a notable number in this literature: in MOSES, the character-level RNN baseline reaches 97.5% and the VAE 97.7%, with structurally constrained generators at 100%. Every comparison in the paper is against GPTs the authors trained themselves, so Figure 2's "Pareto frontier" is a self-constructed curve whose position relative to the field is unknown.

(2) Relative to prior work, the residual increment is narrow. Looped and weight-shared transformers are well established, from the Universal Transformer's shared transition function with variable depth through ALBERT, Subformer, Huginn and Ouro, and the paper itself cites concurrent work on per-iteration specialization; SAE-based circuit analysis of SMILES transformers already exists, with branch-balancing and ring-closure circuits recovered in a 25M-parameter model.

Questions for authors

(1) Please report SMILES length, molecular weight, ring count, and heteroatom distributions for generated versus training molecules. How much of the validity advantage is attributable to emitting shorter or simpler strings?

(2) Is there a token-embedding-only (or bag-of-tokens) probe baseline? Given that bracket depth and ring state are near-computable from the prefix by counting, how do the probe accuracies distinguish model computation from input decodability?

(3) Why is there no comparison to standard MOSES/GuacaMol baselines (CharRNN 97.5%, VAE 97.7%) under a common protocol? Where does 95.3% sit on that map?

1 comment

Thank you for your supportive review and for highlighting how our conceptual setup combines exact symbolic checking at every intermediate state with constraints spanning bracket matching, ring-digit pairing, and valence consistency. New MOSES, GRU, length-matched, and prefix-count controls verify compact generation and ring computation while calibrating baseline scope.

1. Common-protocol MOSES and baseline comparisons (Weakness 1, Question 3)

We agree that Figure 2 was an internal architecture frontier rather than a field-wide map. To position WS-53K against standard baselines, we trained WS-51K, GPT-521K, and the canonical 11.93M-parameter, three-layer CharRNN on official MOSES splits (molsets==0.3.1, 3 seeds, 30,000 samples/cell) alongside matched ZINC GRUs.

MetricWS-51KGPT-521KPaired WS−GPTCharRNN-11.93M
Validity97.221 ± 0.233%95.593 ± 0.033%+1.628 ± 0.210 pp99.019 ± 0.247%
Unique@10K99.847 ± 0.031%99.727 ± 0.015%+0.120 ± 0.036 pp99.923 ± 0.025%
FCD/Test ↓0.93996 ± 0.006550.97127 ± 0.00202−0.03131 ± 0.007990.05790 ± 0.00194
Novelty90.204 ± 0.437%84.982 ± 0.165%+5.222 ± 0.575 pp81.511 ± 1.343%

Under canonical T=1.0 full-softmax decoding, CharRNN-11.93M leads validity, Unique@10K, and FCD/Test, while WS-51K uses 233x fewer parameters, trails validity by 1.80 pp, leads novelty by 8.69 pp, and beats 10.20x-larger GPT-521K across all four metrics under matched top-k=40, T=0.8 decoding. GPT leads WS-51K on SNN/Test, Scaf/Test, Filters, and some property distances, and retrained VAE/GuacaMol panels remain future work. Matched ZINC GRUs (GRU-53K 91.3 ± 2.0%, GRU-206K 96.9 ± 0.4%) still trail WS-53K (95.3%) and WS-206K (98.0%). We revised Section 3 (Table 1, Figure 2) and Appendix D to report all three MOSES models and the GRU baselines.

2. SMILES length and molecular-property distributions (Question 1)

No, WS-53K does not emit shorter or simpler strings. We compared 60,000 ZINC outputs (WS-53K and GPT-527K, three seeds, 10,000 samples/cell) against 224,509 ZINC training molecules.

QuantityWS-53KGPT-527KTraining reference
Token length40.7841.2340.46
Character length45.1245.6946.45
Molecular weight*329.71331.14332.08
Ring count*2.6222.6352.756
Heteroatom fraction24.77%25.09%26.39%
Maximum branch depth2.7442.7222.702

*Molecular weight and ring count use valid molecules, whereas other rows use all strings.

WS-53K matches GPT-527K across these descriptors, and binning both models into training quartiles across length, atom counts, branch depth, and ring labels leaves the +8.85 ± 0.76 pp WS−GPT gap intact (+8.48 to +9.00 pp). We revised Section 3 and Appendix F Table 12 to report these distributions and quartile-reweighted controls.

3. Token-embedding and prefix-count probe baselines (Question 2)

We agree that bracket-depth accuracy matches prefix counting (0.912 vs. 0.915/0.916), so we dropped it as an emergent claim. We reran all probes with molecule-level splits, macro-F1, raw embeddings, and a non-leaky prefix-count baseline (Section 4, Table 3), showing ring state (0.930/0.906 vs. 0.413 prefix, 0.189 raw) and pass-8 closure (0.939/0.951 vs. 0.608, 0.635) far exceed both baselines.

TargetPrefix countRaw embeddingHidden-state (WS-53K / WS-206K)
Bracket depth0.9120.222pass 3: 0.915 / 0.916
Ring state0.4130.189pass 3: 0.930 / 0.906
Next token is ring-close0.6080.635pass 8: 0.939 / 0.951
Token type0.3401.000≈1.000 at every pass

Ring decodability stays correlational while far exceeding both controls. We revised Section 4 (Table 3) to add both baselines and drop bracket depth as an emergent hidden-state claim.

4. Novelty relative to prior work (Weakness 2)

We clarify how our 53K shared block adds insight beyond applying looped architectures to SMILES, contrasting it with unshared SMILES models and large looped LLMs. We revised the Abstract and Sections 1 and 6 to foreground this testbed role.

  • Unshared SMILES models (Section 6, 25M circuit study). Unshared layers confound depth with layer-specific weights, whereas reusing one 53K block keeps weights fixed while beating 10x-larger GPT across ZINC, MOSES, and ChEMBL.
  • Looped LLMs (Sections 1 and 6, Universal Transformers, ALBERT, Subformer, Huginn, Ouro). Large looped LLMs lack exact symbolic checks per step, whereas SMILES rules let us score each pass and localize bracket matching.

We hope the MOSES evaluation, complexity check, and probe controls address your concerns, and we welcome your assessment.

Reviewer_8KMSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a small weight-shared Transformer for SMILES generation and analyzes how repeated passes relate to bracket, ring, and valence constraints. The 53K-parameter model reaches 95.3% SMILES validity on ZINC-250K.

Critical strengths

The paper combines compact molecular generation with a detailed mechanistic analysis using error classification, probing, SAEs, and causal ablation.

Critical weaknesses
  1. The methodological novelty is limited. Weight sharing and looped Transformers are established architectures; the main novelty is their application to SMILES generation and the analysis of intermediate computation.
  2. Parameter efficiency does not imply computational efficiency. The shared block is applied eight times, and inference FLOPs scale with the number of passes. No compute- or latency-matched generation comparison is reported.
  3. The causal evidence for the claimed computation hierarchy is incomplete. Causal ablation identifies a bracket-matching head, whereas ring is supported by errors, probes, and SAE correlations, and valence mainly by output error trends; neither receives comparable causal intervention.
  4. The empirical scope is narrow. All main results are based on ZINC-250K SMILES, so it is unclear whether the observed hierarchy generalizes to other molecular distributions or representations.
Questions for authors
  1. How does SMolLM compare with unshared models under matched inference FLOPs?
  2. Is there causal evidence for the proposed ring and valence stages?
  3. Does the pass-wise hierarchy persist on other molecular datasets?
1 comment

Thank you for your thoughtful review and for noting our combination of compact generation with error classification, probing, SAEs, and ablation. New ChEMBL, MOSES, FLOP/latency, and dependency controls replicate compact generation and bracket localization while calibrating compute and causal claims.

1. Generalization beyond ZINC-250K (Weakness 4, Question 3)

We agree ZINC-250K alone cannot establish generalization across distributions, protocols, and passes. To test all three, we ran curated ChEMBL models, the official MOSES suite, and ChEMBL probing and head ablation. ChEMBL and MOSES replicate compact generation and ChEMBL replicates bracket localization, while prefix-controlled ChEMBL probes, a ChEMBL depth sweep, and SELFIES remain open. We revised Sections 3–4 (Table 1) and Appendices C–D and H to report these results and limits.

  • ChEMBL corpus (Section 3.1, Appendix C Table 8). Over 3 seeds on 249,455 MW- and vocab-matched molecules, WS-53K beats GPT-527K on validity (93.7 ± 0.7% vs. 78.0 ± 1.1%) and FCD (3.59 ± 0.11 vs. 4.12 ± 0.08) at 9.8x fewer params, and WS-206K hits 98.5 ± 0.1%.
  • MOSES protocol (Section 3.1 Table 1, Appendix D Table 9). Over 3 seeds (molsets==0.3.1, 30K samples/cell), WS-51K beats GPT-521K on validity (97.22% vs. 95.59%), Unique@10K, FCD/Test (0.940 vs. 0.971), and novelty (90.20% vs. 84.98%), while 233x-larger CharRNN-11.9M (T=1.0) hits 99.02% validity, 0.058 FCD, and 81.51% novelty (GPT leads similarity/filters).
  • ChEMBL pass-wise transfer (Sections 4.2–4.3, Appendix H Table 14). Ring-state probe accuracy rises across passes (0.74 to 0.95 in WS-53K, 0.79 to 0.95 in WS-206K, descriptive without prefix-count controls), and head ablation localizes pass-1 brackets in 5/6 ChEMBL runs (10/12 overall).

2. Comparison under matched inference FLOPs and latency (Weakness 2, Question 1)

We agree parameter efficiency does not imply compute efficiency. To test matched FLOPs and latency, we computed fixed-length FLOPs, timed CPU decoding, and ran a 0.1%-matched-FLOP pair. Weight sharing saves 4x to 9.85x parameter memory at similar FLOPs while trading off 3.4x slower CPU sampling and 1.6 pp equal-FLOP validity. We revised Section 3.1 and Appendix E (Tables 10–11).

  • Matched forward FLOPs (Appendix E Table 11). At a 0.1% match (203.3M vs. 203.5M learned-weight FLOPs, plus 33.6M attention at length 128), 4-pass WS-206K uses ~4x fewer params than GPT-813K (205,568 vs. 813,440) at 97.1% vs. 98.7% validity.
  • Memory vs. CPU speed (Section 3.1, Appendix E Table 10). At length 128, WS-53K uses fewer learned-weight FLOPs (101.5M vs. 129.8M, plus 33.6M vs. 18.9M attention) and 9.85x less memory (209 KB vs. 2,058 KB) than GPT-527K, yet runs 3.4x slower on CPU (99 vs. 337 tok/s) over 8 serial passes.

3. Causal evidence for ring and valence stages (Weakness 3, Question 2)

We agree causal evidence applies specifically to brackets. To test cross-stage dependency and prefix-count confounds, we ran a 32-cell ablation sweep, a 6-checkpoint dependency test, controlled probes, and multi-label scoring. No causal locus or dependency is established for ring or valence (10/12 bracket localization, dependency inconclusive). We revised the Abstract, Section 4 (Tables 2–4), and Appendices H–I to frame a depth- and pass-dependent progression.

  • Head-ablation sweep (Section 4.3 Table 4, Appendix H Table 14). Across 2 sizes, 3 seeds, and 2 datasets (n=2,000/condition), zeroing heads isolates a pass-1 bracket head in 10/12 runs with no ring or valence locus.
  • Dependency test (Appendix I Table 15). On fixed ZINC prefixes, pass-1 bracket-head ablation lowers pass-4 ring log-probability vs. KL-matched controls in 4/6 runs (4/5 localizing seeds) but raises it in 2/6 (pooled +0.00429 nats, 95% CI [−0.0912, +0.1149]), leaving results inconclusive.
  • Errors and probes (Sections 4.1–4.2, Tables 2–3). Trained depth lowers syntax and chemistry errors, and pass-3 ring-state (0.930/0.906 vs. 0.413) and pass-8 closure probes (0.939/0.951 vs. 0.608) beat prefix counts, while ring/chemistry interventions remain open.

4. Contribution relative to prior work (Weakness 1)

We clarify how our 53K shared block adds insight beyond applying looped models to SMILES, contrasting it with unshared SMILES models and looped LLMs. We revised the Abstract and Sections 1 and 6 to foreground this testbed role.

  • Unshared SMILES models (Section 6, 25M circuit study). Unshared layers confound depth with layer-specific weights, whereas reusing one 53K block keeps weights fixed while beating 10x-larger GPT across ZINC, MOSES, and ChEMBL.
  • Looped LLMs (Sections 1 and 6, Universal Transformer, ALBERT, Subformer, Huginn, Ouro). Large looped LLMs lack exact symbolic checks per step, whereas SMILES rules let us score each pass and causally localize bracket matching.

We hope these ChEMBL, MOSES, compute, and dependency controls resolve your concerns, and we welcome your feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We thank all three reviewers for their detailed reviews and the area chair for overseeing the discussion. New controls rule out alternative explanations while calibrating our compute and causal claims.

  1. External benchmarks and alternative explanations (8KMS W4/Q3, rz7G W1/Q1/Q3). To test generalization beyond ZINC-250K and rule out protocol, RNN, or length artifacts, we ran ChEMBL, MOSES, GRU, and complexity controls, showing weight-shared models keep leading 10x-larger GPT while ChEMBL probe controls and SELFIES remain open.
  2. Parameter storage vs. inference compute (8KMS W2/Q1). To separate storage from compute, we tested FLOPs, CPU latency, and matched-FLOP validity, showing weight sharing saves 4x to 9.85x memory at comparable FLOPs while trading off CPU speed and 1.6 pp validity. We revised Section 3 and Appendix E Tables 10-11 to report this memory, FLOP, and latency trade-off.
  3. Mechanistic evidence vs. causal scope (8KMS W3/Q2, rz7G Q2, iTVv W1/W2/Q1/Q2). To separate hidden computation from token counting and cross-stage causality, we tested multi-label errors, prefix-controlled probes, 53K/206K ablations, and cross-pass dependency, showing pass-wise ring computation and bracket localization while cross-stage dependency remains open. Every soundness concern raised in the reviews proved directly testable and is addressed via new controls and calibrated claims. Errors drop with depth, ring probes beat prefix counting, and bracket heads localize in 10 of 12 ZINC and ChEMBL checkpoints, while 6-checkpoint pass-1-to-pass-4 shifts are inconclusive. We revised the Title, Abstract, and Sections 1 and 4 to define "small-molecule" as compounds up to 500 Da and frame a depth- and pass-dependent progression.
  4. Scientific role relative to prior work (8KMS W1, rz7G W2). To clarify how our 53K looped model adds insight beyond Universal Transformers, ALBERT, Subformer, Huginn, Ouro, and the 25M SMILES circuit study, we revised the Abstract and Sections 1 and 6. They show that unshared SMILES models conflate depth with layer-specific weights and looped LLMs lack exact symbolic checks, whereas reusing one 53K block under exact rules isolates what repeated computation alone resolves.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.