acceptodds
Under review as a conference paper at ICLR 2027

MolToken: Scaling a Molecular Foundation Model with Unified Graph Tokenization across Chemical and Biological Space

Abstract

A central goal of molecular foundation modeling is to jointly model chemical and biological space. However, existing molecular foundation models often retain class-specific representations and encoding schemes even in joint modeling, while scaling behavior under a unified chemical representation remains underexplored. We introduce MolToken, a Molecular foundation model with unified graph Tokenization across chemical and biological space. MolToken maps small molecules, peptides, proteins, DNA, and RNA into a unified atom-level graph space. Strictly reversible Frequency Dual Depth First Search (FDDFS) serialization and Byte Pair Encoding (BPE) yield a unified vocabulary for joint pre-training with a single autoregressive Transformer. BPE reduces the total length of FDDFS sequences by a factor of approximately 71.8, making atom-level modeling more tractable. We systematically compare joint and single-domain pre-training across model sizes and data scales. Within the evaluated range, compute-optimal model and data scales follow power laws, while optimal loss decreases approximately linearly with log compute. Out-of-sample experiments support loss prediction at substantially larger compute. Cross-domain comparisons reveal domain-dependent benefits and trade-offs of joint training and suggest that single-domain scaling cannot substitute for cross-domain data coverage. Downstream fine-tuning further demonstrates the adaptability of the unified pretrained backbone to molecular generation and design.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Molecule, Tokenizer, Foundation Model, Scaling Laws

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UWf3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes MolToken, a unified molecular foundation model that represents small molecules, peptides, proteins, DNA, and RNA using a shared atom-level graph representation. The authors introduce a unified graph tokenization approach based on FDDFS serialization and BPE, enabling joint pretraining with a single autoregressive Transformer. The paper primarily investigates the scaling behavior of the proposed model, particularly the relationship between joint and single-domain pretraining as model size and computational budget increase. Several downstream tasks are also evaluated to demonstrate the applicability of the pretrained model across different molecular domains

Critical strengths
  1. The paper provides an interesting approach to representing different molecular domains within a unified atom-level graph framework. The combination of reversible graph serialization and a shared vocabulary enables joint modeling without relying on domain-specific encoders.
  2. The authors conduct a relatively comprehensive analysis of the scaling behavior of MolToken, including compute-optimal scaling and comparisons between joint and single-domain pretraining. These experiments provide useful empirical observations about how different molecular domains respond to increased model and training scales.
Critical weaknesses
  1. The discussion of existing all-atom molecular foundation models could be more comprehensive. For example, ATOMICA (Fang et al., 2025) is a relevant prior work that learns unified atom-level representations across different molecular modalities. Although ATOMICA focuses on 3D intermolecular interactions and differs from MolToken in its modeling objectives, a discussion of this work would help clarify the novelty and positioning of MolToken relative to existing unified molecular representation approaches.
  2. While the proposed approach provides a unified representation for multiple molecular domains, the experimental results do not sufficiently demonstrate whether this unification leads to improved cross-domain knowledge sharing.
  3. While the reported 71.8× compression is useful, sequence compression is an expected benefit of BPE. Its impact on actual training efficiency and downstream performance remains unclear.
Questions for authors
  1. The paper provides detailed observations of how joint and single-domain pretraining behave at different scales. However, it is not entirely clear what general insights these results offer for the development of unified molecular foundation models. For example, how could these findings inform future model design, training strategies, or data allocation across molecular domains?
  2. Since one of the motivations for unified molecular modeling is to enable learning across different molecular domains, evaluating whether joint pretraining improves downstream performance over single-domain pretraining would help strengthen the paper's conclusions.
8 comments
AuthorsOpenReview ↗

Related Work and Cross-Domain Knowledge Sharing

We thank the reviewer for the thoughtful comments on related work and cross-domain knowledge sharing.

1. Comparison with ATOMICA

We appreciate the reviewer for highlighting ATOMICA, an important related work. We believe ATOMICA represents a valuable exploration of unified biomolecular representation learning and provides important insights into modeling molecular structures and interactions across different molecular types.

Although both approaches investigate unified modeling across molecular types, they differ substantially in their research objectives, modeling targets, and technical approaches.

ATOMICA primarily focuses on learning representations of intermolecular interactions. It models molecular complexes and their interactions using three-dimensional geometric information, employing graph neural networks (GNNs) to learn structural and interaction features.

In contrast, MolToken focuses on unified representation and scalable generative modeling of individual molecules across heterogeneous chemical and biological domains. We convert small molecules, peptides, proteins, DNA, and RNA into two-dimensional atomic graphs and construct a shared discrete token space through FDDFS+BPE. This enables autoregressive pretraining of a single standard Transformer across all five domains and allows us to further investigate scaling laws within this unified framework.

Thus, ATOMICA and MolToken explore cross-molecular-type modeling from complementary perspectives: geometric interaction representation and unified tokenization for scalable generative learning, respectively.

We thank the reviewer for this suggestion and will discuss ATOMICA and related work in the revised manuscript to clarify the connections and distinctions between these research directions.

2. Evidence of Cross-Domain Knowledge Sharing

The Bitter Lesson emphasizes the importance of general-purpose learning methods that can leverage increasing data and computational resources. We believe this perspective offers an important motivation for molecular foundation models: establishing a shared representation and learning space enables heterogeneous chemical and biological data to contribute to scalable pretraining within a single model.

Following this principle, MolToken converts small molecules, peptides, proteins, DNA, and RNA into unified atomic-level graph representations and constructs a shared token space through FDDFS+BPE tokenization, allowing all five domains to participate in parameter learning within the same Transformer.

To further evaluate the practical benefits of this framework, we conducted additional pretraining ablations during the rebuttal period. We controlled model size (approximately 41M parameters) and nominal pretraining compute (5×10175\times10^{17} FLOPs) and compared random initialization, small-molecule-only pretraining, and five-domain joint pretraining.

TaskRandomSmall-moleculeJoint
MOSES Validity ↑0.91150.98780.9692
DNA Promoter Accuracy ↑0.82870.84000.8966
RNA MRL Pearson² ↑0.39430.61900.6387
Protein GB1 Spearman ↑0.21120.27170.2292

On DNA promoter classification, the three settings achieve accuracies of 0.8287, 0.8400, and 0.8966, respectively. On RNA translation-efficiency prediction, their Pearson² scores are 0.3943, 0.6190, and 0.6387, respectively.

These results demonstrate that unified pretraining enables a shared model to learn representations applicable to downstream tasks across different molecular domains. Since joint pretraining also includes target-domain data, these experiments do not independently quantify the contribution of knowledge transferred from other domains, but provide an experimental basis for further investigating cross-domain knowledge sharing.

3. Practical Value of BPE Compression

A unified atomic-level representation provides a common foundation for modeling different molecular types, but also introduces longer sequences. BPE plays an important role in MolToken by reducing the token cost of this unified graph representation, allowing heterogeneous molecular data to be processed more efficiently by a shared autoregressive model.

Token compression itself provides direct practical value. The computational and memory costs of Transformers depend strongly on sequence length. Reducing the number of tokens therefore directly lowers sequence-processing costs and allows more molecular data to be processed under the same compute budget. The 71.8× compression ratio reported in our paper demonstrates a substantial reduction in token count relative to the original FDDFS symbol sequence, significantly improving the computational scalability of unified atomic-level representations.

To further evaluate this effect on actual pretraining data, we analyzed all 323,079,504 training records. After BPE, the mean sequence length is 30.62 tokens, with a 99th percentile of 133 tokens. Under the 512-token context limit, only 437 records (0.000135%) require runtime truncation.

These results demonstrate that FDDFS+BPE compresses atomic-level representations across all five molecular domains into sequences suitable for a shared Transformer. The 71.8× token compression is not merely a representation-level statistic but a practical technical advantage supporting efficient training and continued scaling of unified molecular models. We will include these statistics in the revision and clarify the distinction between token compression and end-to-end training throughput.

AuthorsOpenReview ↗

Scaling Implications and Downstream Benefits

We thank the reviewer for the questions regarding the broader implications of our scaling analysis and the downstream benefits of joint pretraining.

1. Implications of Scaling Laws for Unified Molecular Models

A key contribution of MolToken is demonstrating that heterogeneous chemical and biological data can be jointly modeled within a unified framework while exhibiting predictable scaling behavior. This aligns with the central insight of The Bitter Lesson: general-purpose learning frameworks capable of leveraging increasing data and computation provide a promising path toward scalable intelligence.

Using 67 model configurations across 10 compute budgets, we identify compute-optimal scaling trends for unified molecular pretraining. The scaling exponents for model size and training tokens are approximately 0.543 and 0.457, respectively. Guided by these relationships, we trained a 0.55B model, achieving a predicted Test Loss of 4.0315 versus a measured 4.0482, with an absolute error of only 0.0166.

Importantly, the emergence of predictable scaling laws under joint pretraining provides empirical evidence that our unified representation supports systematic and scalable learning across heterogeneous molecular domains. Beyond guiding model-size and compute allocation, this demonstrates the potential of MolToken as a general-purpose framework for integrating increasingly diverse molecular data and computational resources.

Comparisons between joint and single-domain pretraining further reveal domain-dependent scaling behavior, providing a basis for future optimization of model capacity, training budgets, and domain sampling proportions.

2. Downstream Comparison Between Joint and Single-Domain Pretraining

As reported in our response on Cross-Domain Knowledge Sharing, we conducted additional downstream ablations comparing random initialization, small-molecule-only pretraining, and five-domain joint pretraining under matched model sizes and pretraining compute budgets.

The results show that joint pretraining achieves better performance on DNA promoter classification (0.8966 vs. 0.8400) and RNA translation-efficiency prediction (0.6387 vs. 0.6190) than small-molecule-only pretraining. Meanwhile, small-molecule-only pretraining performs better on MOSES validity (0.9878 vs. 0.9692). On protein GB1, joint pretraining has not demonstrated consistent improvements.

These findings demonstrate the downstream adaptability of unified pretraining across molecular domains while revealing task-dependent performance trade-offs. We will include the complete ablation results in the revised manuscript and further discuss the relationship between domain composition and downstream performance.

Reviewer_NfPhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MolToken proposes a single representation for five molecular domains (small molecules, peptides, proteins, DNA, RNA) by mapping all of them to attributed heavy-atom graphs over a shared attribute space (6 node attributes, 3 edge attributes), serializing those graphs with a two-pass frequency-guided DFS (“FDDFS”) that is strictly reversible up to attributed-graph isomorphism, and then learning a single BPE vocabulary (17,207 entries) over the resulting symbol sequences. A decoder-only Transformer is pretrained autoregressively on the joint corpus (323M graphs, 9.90B BPE tokens).

Contributions as claimed: (i) the unified tokenizer, with reversibility argued in Proposition B.1 and built on Turán’s (1984) spanning-tree/non-tree-edge encoding plus a frequency-guided traversal order; (ii) a corpus-level compression of 71.78× from FDDFS base symbols to BPE tokens (710.8B → 9.90B), which is what makes atom-level modeling of biopolymers tractable; (iii) an IsoFLOP scaling study across four train/eval settings (1-to-1, m-to-m, 1-to-m, m-to-1) over 6M–458M parameters and 3.5×10¹⁷–3.5×10¹⁸ FLOPs, yielding N* ∝ C^0.543, D* ∝ C^0.457, and L* = −0.1988 log₁₀C + 8.036; (iv) an out-of-sample check in which a 0.55B model trained at 37.5× the largest fitting budget attains 4.0348 against a predicted 4.0370; (v) cross-domain findings that joint training reduces target-domain token requirements (most visibly for peptides, 0.17% of joint tokens) while the joint loss of single-domain models does not improve with compute; (vi) fine-tuning on eight downstream tasks (MOSES, GuacaMol, RAbD, TCR, Rfam-17, protein generation, DNA generation, PepBench-Gen).

Validation is entirely empirical apart from the reversibility proposition, which is a constructive encoder/decoder argument rather than a hard theoretical result.

Critical strengths

Atom-level fidelity for macromolecules no longer costs a prohibitive sequence length: proteins compress to ~50 tokens for 64–256-residue windows, and the 99th-percentile length in every domain fits inside the 511-token content budget. This matters because the standing objection to atom-level biopolymer modeling has been sequence length, and the paper removes it with a reversible encoding.

Critical weaknesses
  1. The cross-domain claim rests on an out-of-support metric and is near-tautological. With a 17,207-entry vocabulary, uniform prediction scores ln(17207) ≈ 9.75 nats/token, yet the single-domain joint losses in Fig. 6 and Table E.5 run ~7.5–20 — mostly worse than uniform. Table E.5 is clearest: the 41.0M RNA model’s in-domain loss falls 4.084 → 3.858 while joint loss rises 15.706 → 17.844. That is calibration, not scaling: a sharpening RNA-only model assigns ever-lower probability to tokens it never emits, so cross-domain NLL must increase. Since this grounds a claim in the abstract, intro, and conclusion, it needs a metric meaningful off-support — per-domain perplexity after a fixed light adaptation budget, or transfer as fine-tuning efficiency on held-out domains.

  2. The headline extrapolation is fragile on three counts. (a) It uses terminal training loss under repetition (~3.87 passes, §A.2), justified by an argument that assumes the infinite-data regime the run is not in; the train/test calibration in Table D.2 comes from a different setting, and its MAE (0.003) exceeds the reported extrapolation error (2.24×10⁻³). (b) L*(C) = u log₁₀C + v is unbounded below and must break; the standard L = E + A/N^α + B/D^β is never fit as a comparison, so a reader cannot tell a real trend from a locally linear segment across 1.5 decades. (c) One extrapolation point, whose configuration was chosen from the fit, cannot distinguish a correct law from a lucky one.

  3. Downstream results undercut the thesis, and Table 2 is unrepresentative. In the appendix: protein pLDDT 35.15/34.35 vs DPLM150M 81.88 and natural UniRef50 63.79, exact-length rates 18.75%/27.50% (F.12); DNA strict 200-bp validity 39.0%/37.9%, with F.15’s motif comparison blank for both MolToken models; Rfam-17 behind RfamGen and GCVAE (5.04 vs 6.76, 5.96) at 74.8% coverage vs ~99%; TCR behind TCRT5 on Top-20. MolToken clearly leads on 1 of 8 tasks (RAbD prefix-L AAR, 38.28 vs 34.81), yet Table 2 selects three datasets including two of the strongest. A main-text-only reader forms a materially different impression — and a model that cannot reliably emit a valid 200-bp sequence or hit a requested protein length does not yet support the paper’s framing.

Questions for authors

Figure 6 reports joint losses up to ~20 nats/token against a uniform baseline of ~9.75. Is the increase in joint loss with compute anything other than single-domain models sharpening away probability mass from unseen tokens? Can you show the result holds under a measure that is defined off-support — e.g. transfer efficiency under a fixed adaptation budget? Both downstream backbones (0.55B, 1.44B) consume the same 39.57B tokens; by your own fit, the compute-optimal N at 3.4×10²⁰ FLOPs is ≈0.9B, so the 1.44B model is off-frontier. Do the downstream results test the scaling conclusions at all, or are they an independent capacity comparison? Table F.15 is empty for both models. Is the DNA motif-distribution evaluation unfinished, and is it required for the DNA claims?

7 comments
AuthorsOpenReview ↗

Scaling-Law Extrapolation and Alternative Fits

We thank the reviewer for the constructive suggestions regarding scaling-law extrapolation and alternative fitting formulations.

1. Extrapolation Using Test Loss

Following the reviewer's suggestion, we reevaluated the scaling-law predictions using the independent test set defined in Appendix A.1.

Specifically, we evaluated all 67 model configurations across the original 10 compute budgets on the same test set and refitted the IsoFLOP-optimal frontier. We then predicted the optimal loss at the compute budget of the 0.55B model without including its evaluation results in calibration.

EvaluationPredicted LossMeasured LossAbsolute ErrorRelative Error
Original: Training Loss4.037044.03480.002240.06%
New: Test Loss4.031544.048180.016640.41%

The Test Loss-based IsoFLOP frontier achieves an absolute extrapolation error of only 0.01664 nats/token (0.41%) on the high-budget model. This demonstrates that the observed scaling-law extrapolation accuracy is not confined to Training Loss and remains strong when both fitting and validation use independent test data.

We will include these results in the revised manuscript and clearly distinguish the Training Loss and Test Loss fitting and validation protocols.

2. Comparison with the Two-Dimensional Chinchilla Formulation

Due to practical computational constraints and the range of model sizes explored, our scaling-law calibration experiments cover approximately one order of magnitude in compute. Within this range, we adopt the IsoFLOP approach from Chinchilla, estimating the optimal model size and training-token count in stages and fitting the optimal loss using a lower-parameter semi-logarithmic formulation to reduce the complexity of joint fitting.

Following the reviewer's suggestion, we additionally fitted the standard two-dimensional power law, L(N,D)=E+A/Nα+B/DβL(N,D)=E+A/N^\alpha+B/D^\beta, on the same calibration data:

MethodPredicted LossMeasured LossAbsolute Error
IsoFLOP + semi-log4.03154.04820.0166
2D Chinchilla power law4.21094.04820.1627

Further analysis indicates that the limited and uneven coverage of calibration configurations makes the asymptotic term EE difficult to estimate reliably. Under the fixed-FLOPs IsoFLOP design, larger models receive fewer training tokens. Some very small models and large-model, short-training configurations strongly influence the fitted decay exponents, yielding E=4.1091E=4.1091, which exceeds the measured loss of the high-budget model.

To investigate this behavior without retraining models, we performed two fitting diagnostics. First, keeping the original 67 configurations' (N,D,C)(N,D,C) unchanged, we replaced and restored the losses of selected observations. Four very small-model configurations shifted EE from 3.8932 to 4.1054, while two large-model, short-training configurations shifted it from 4.0042 to 4.1091. Second, we added existing checkpoints from different training stages of large models as additional observations. Including earlier checkpoints yielded E=4.1662E=4.1662, whereas including final checkpoints yielded E=3.8584E=3.8584. These post-hoc diagnostics show that the coverage of model sizes and training-token counts substantially affects the estimation of the asymptotic term.

We do not argue that the two-dimensional power-law formulation is inherently unsuitable. Rather, under the current computational and calibration-data constraints, we selected the staged IsoFLOP semi-logarithmic approach because it can be estimated more stably and achieves better independent extrapolation accuracy. The semi-logarithmic formulation is a local empirical approximation within the studied compute range, and we will clarify its scope in the revision.

AuthorsOpenReview ↗

Cross-Domain Adaptation and Peptide Data Efficiency

We thank the reviewer for the suggestions regarding cross-domain loss and peptide data efficiency.

1. Cross-Domain Adaptation

To further validate the findings in Figure 6, we conducted an additional cross-domain transfer experiment under a fixed fine-tuning budget. Specifically, we evaluated RNA-only models pretrained with different compute budgets on the QM9 HOMO molecular property prediction task.

ModelQM9 NLL ↓HOMO MAE (meV) ↓
RNA-only Low19.073216.027
RNA-only Mid20.376241.037
RNA-only High21.425234.705
Random initialization9.814166.035
Joint pretraining3.949128.020

As RNA-only pretraining compute increases, cross-domain NLL consistently deteriorates, while downstream performance remains relatively poor. In contrast, joint pretraining achieves substantially lower cross-domain NLL and outperforms both RNA-only models and random initialization under the same fine-tuning budget.

These results demonstrate that the cross-domain modeling issue revealed in Figure 6 is not merely reflected in zero-shot probability predictions but is also accompanied by degraded downstream transfer performance. This further supports our conclusion that increasing single-domain pretraining compute alone cannot substitute for broader training-domain coverage.

2. Robustness of Peptide Data Efficiency

We appreciate the reviewer's detailed comments on the number of fitting points, fitting windows, and repeated training epochs in our peptide scaling analysis.

High-quality publicly available peptide data remain relatively limited, particularly diverse sequences suitable for large-scale self-supervised pretraining. This constraint makes peptide-only experiments more sensitive to dataset size and repeated training, while limiting the compute range available for reliable scaling-law fitting.

Nevertheless, we conducted additional analyses of 65 jointly pretrained models and 41 peptide-only models, evaluating them on a common test set containing 4,966 variable-length peptide sequences and 97,148 supervised tokens.

Model ConfigurationPeptide Training TokensTest Loss ↓
Joint (minimum target-domain tokens)1.01M4.8738
Peptide-only (lowest loss)114.82M5.0701

On this common test set, the joint model using the fewest peptide training tokens achieves a Test Loss of 4.8738 with approximately 1.01M peptide tokens. In comparison, the best-performing evaluated peptide-only model uses approximately 114.82M peptide tokens and achieves a Test Loss of 5.0701, corresponding to an approximately 114-fold difference in target-domain token usage.

Across the additional evaluation, the Test Loss ranges of joint and peptide-only models are 4.7647–4.8738 and 5.0701–7.2454, respectively, further supporting the target-domain data-efficiency advantage of joint pretraining.

We also examined sensitivity to fitting windows and repeated training epochs. The original scaling-law coefficients could be reproduced accurately. However, the limited number of effective compute budgets for peptide-only training means that removing certain budgets or further restricting training epochs reduces the number of points available for stable fitting. This primarily reflects the practical constraints of peptide dataset size and experimental compute budgets.

Importantly, data efficiency in our analysis refers to the utilization of target-domain peptide tokens, not total computational efficiency. Joint pretraining can leverage data from other molecular domains, but this does not imply a lower overall computational cost.

In summary, although the limited peptide dataset constrains the scaling-law fitting range, our additional experiments show that the target-domain data-efficiency advantage of joint pretraining is directly observable in the Test Loss of trained models, beyond the fitted scaling curves. We will incorporate these analyses and clarify the scope of this conclusion in the revised manuscript.

AuthorsOpenReview ↗

Compute Accounting and Model Selection

We thank the reviewer for the questions regarding compute accounting in Figure 5 and the relationship between downstream model selection and scaling laws.

1. Compute Budget Definition in Figure 5

We would like to clarify that using rkCr_kC as the horizontal axis in Figure 5 is an intentional choice motivated by our research objectives.

One important question in this work is how the amount of data allocated to each molecular domain affects its scaling behavior under unified pretraining. Accordingly, we use rkDr_kD to measure the number of training tokens from the target domain and rkCr_kC to represent the corresponding compute budget allocated in proportion to its token share. This provides a consistent analytical basis for comparing joint and single-domain pretraining.

Figure 5 is intended to examine modeling performance in relation to target-domain data and attributed compute. The domain-normalized compute metric therefore has a clear analytical purpose. For single-domain pretraining, rk=1r_k=1; for joint pretraining, rkr_k denotes the target domain's proportion of training tokens. Both settings use the same definition of attributed compute.

The reviewer's suggested total FLOPs, CC, measure the actual computational cost of training the entire model, whereas rkCr_kC measures the compute share attributed to the target domain. These represent two distinct analytical dimensions and cannot be directly substituted for one another.

For peptides in particular, the small value of rkCr_kC reflects their approximately 0.17% share of joint-pretraining tokens, precisely the cross-domain data-utilization phenomenon that Figure 5 is designed to illustrate.

We therefore maintain that the current horizontal-axis definition in Figure 5 is appropriate for its research purpose and effectively characterizes the relationship between domain-specific resource allocation and scaling behavior. Total computational cost and domain-attributed compute have different meanings, and Figure 5 explicitly analyzes the latter.

2. Selection of the 1.44B Model and Its Relationship to Scaling Laws

We would like to clarify that our scaling-law analysis and downstream evaluations serve different experimental purposes.

The scaling-law experiments use the IsoFLOP approach to study the optimal allocation of model parameters and training tokens under a given compute budget, with the 0.55B model serving as the high-budget loss-extrapolation validation point.

In contrast, the 1.44B model is primarily used to investigate how increased model capacity affects practical downstream performance, rather than serving as an additional validation point for compute-optimal scaling laws.

To this end, we keep the pretrained tokenizer, data composition, and number of training tokens consistent between the 0.55B and 1.44B models, increasing only the number of model parameters to examine the effect of model capacity on downstream performance.

Thus, the downstream experiments demonstrate the applicability of unified molecular pretraining at different model scales and complement the compute-optimal allocation analysis. We do not treat the 1.44B model as a compute-optimal configuration.

We will further clarify these distinct experimental objectives in the revised manuscript to avoid confusion.

Reviewer_TbhwOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

the paper uses a reversible atom-level tokenization (FDDFS + BPE) to put five molecular domains into one autoregressive transformer, and runs a joint vs. single-domain scaling study. The tokenization is neat, but the main claims don't hold up.

Critical strengths

Clean, reversible tokenization shared across all five domains. A large, carefully logged set of training runs.

Critical weaknesses
  • The headline claim is basically trivial. An RNA-only model doing badly on proteins is just out-of-distribution failure. On each domain's own loss, joint training doesn't help proteins or DNA, which undercuts the case for a unified model.
  • The scaling laws are fragile. The compute range is narrow, several fit points are weakly supported, extrapolation is checked with a single point, and training runs about 3.9 epochs, so data repetition is mixed into every fit.
  • The losses can't be compared across domains or models. Per-BPE-token NLL depends on the vocabulary, and there are no bits-per-atom numbers.
  • The downstream results are weak. It loses badly to TCRT5 (GIANA 3.1 vs 8.0), its MOSES FCD is worse than a plain VAE, and there's no comparison to ESM, Evo or RNA-FM. "Foundation model" isn't earned.
  • Ignores residue-level priors and 3D information. Does this hurt at large sequence lengths? What happens to very long proteins and genomes?
  • No data-mix ablation. Peptides are only 0.17% of tokens, so the trade-off conclusions may just reflect this one mixture. How sensitive are the conclusions to the domain weights?
Questions for authors

Please reference above.

4 comments
AuthorsOpenReview ↗

Unified Molecular Modeling and Scaling Reliability

We thank the reviewer for the thoughtful comments. We address the value of unified molecular modeling and the reliability of our scaling-law analysis below.

1. Value of Unified Molecular Modeling

Inspired by the principles of general-purpose learning and scalability emphasized in The Bitter Lesson, we believe an important direction for unified molecular foundation models is to bring data from diverse chemical and biological domains into a shared model space, enabling joint pretraining through shared parameters.

Through a unified atomic-graph representation and FDDFS+BPE tokenization, MolToken enables small molecules, peptides, proteins, DNA, and RNA to be modeled by a single Transformer. Figure 5 illustrates the scaling behavior of this unified training framework across different domains. While domain-specific models achieve lower in-domain loss for proteins and DNA, joint pretraining enables a single model to cover multiple molecular domains, demonstrating the capabilities and performance trade-offs of unified modeling.

Figure 6 further shows that increasing the compute budget consistently reduces the in-domain loss of domain-specific models, but does not necessarily improve their overall cross-domain loss. This indicates that model scaling and training-domain coverage are two important dimensions of unified molecular modeling.

We therefore believe that MolToken's value lies in establishing a unified pretraining framework that can jointly leverage heterogeneous molecular data. Our scaling analysis further reveals domain-specific performance and resource-allocation patterns, providing a foundation for expanding cross-domain data coverage and computational scale.

2. Reliability of Scaling Laws

We thank the reviewer for the suggestions regarding scaling-law reliability. We further examined the stability of our fitted results and conducted additional extrapolation validation using independent test loss.

Our study uses 67 model configurations across 10 compute budgets to estimate the IsoFLOP-optimal frontier. To assess sensitivity to the calibration data, we conducted a subsampling experiment in which 75% of the configurations (approximately 50) were randomly retained and the fitting procedure was repeated 100 times. Under this setting, the absolute extrapolation error for the 0.55B model was 0.0183 ± 0.00085, indicating that the predictions remained relatively stable even with fewer calibration points.

From the perspective of statistical learning theory, generalization bounds relate test loss to training loss through a generalization-error term, i.e., Ltest≤Ltrain+ϵgenL_{\mathrm{test}} \leq L_{\mathrm{train}} + \epsilon_{\mathrm{gen}}. When the generalization gap is small, training loss can serve as an effective approximation to test loss. Small generalization gaps are also commonly observed in large-scale language model pretraining, providing theoretical and empirical motivation for studying scaling laws using training loss.

To further verify this relationship for MolToken, we reevaluated all 67 calibration configurations on an independent test set and refitted the scaling relationship using test loss. Without including the 0.55B model's test results in calibration, the predicted test loss was 4.0315, compared with the measured value of 4.0482, yielding an absolute error of only 0.0166 (approximately 0.41% relative error). The model's compute budget was approximately 38.7× the upper limit of the calibration range, providing direct validation of high-budget extrapolation.

Regarding data repetition, the 0.55B model underwent approximately 3.9 training epochs. Under the training settings studied, the analyses based on training loss and test loss exhibit consistent scaling trends, and the accurate independent test-loss extrapolation further supports the stability of our empirical scaling findings.

We will incorporate the calibration-subsampling analysis and test-loss extrapolation results into the revised manuscript, while clarifying the calibration compute range and the scope of the single high-budget validation point.

AuthorsOpenReview ↗

Loss Comparability and Downstream Evaluation

We thank the reviewer for the comments regarding loss comparability and downstream evaluation.

1. Comparability of Loss Measurements

We would like to clarify that all scaling-law experiments in this work use the same FDDFS+BPE tokenizer and a shared vocabulary of 17,207 tokens. Therefore, there is no inconsistency in loss units arising from different tokenizers across models.

Moreover, our scaling-law analysis primarily examines how loss changes with model size and compute budget within the same molecular domain and under the same evaluation distribution. Joint-model comparisons are likewise conducted on a fixed five-domain evaluation mixture, ensuring a consistent evaluation protocol.

We agree that BPE tokens from different molecular domains may correspond to different numbers of atoms or structural units, and therefore absolute loss values across domains do not necessarily reflect equivalent modeling difficulty. However, this does not affect the scaling-law comparisons conducted under a fixed tokenizer and evaluation distribution.

We will further clarify the evaluation protocol and the scope of the loss metric in the revised manuscript.

2. Downstream Evaluation and Additional Ablations

We acknowledge the strengths of domain-specific models such as ESM and Evo on their respective tasks. MolToken aims to explore cross-domain pretraining and downstream adaptation through a unified atomic-graph representation.

To further evaluate this capability, we conducted additional ablation experiments during the rebuttal period, comparing random initialization, small-molecule-only pretraining, and five-domain joint pretraining. The pretrained models use approximately 41M parameters and the same pretraining compute budget of 5×10175 \times 10^{17} FLOPs, with a consistent downstream fine-tuning protocol.

Downstream TaskRandomSmall-moleculeJoint
MOSES Validity ↑0.91150.98780.9692
DNA Promoter Accuracy ↑0.82870.84000.8966
RNA MRL Pearson² ↑0.39430.61900.6387

The table reports results averaged over three random seeds.

Joint pretraining outperforms both random initialization and small-molecule-only pretraining on DNA promoter classification and RNA translation-efficiency prediction, demonstrating the cross-domain downstream adaptability of the unified model. Meanwhile, small-molecule-only pretraining achieves higher MOSES validity, illustrating the respective advantages of domain-specific and joint pretraining.

These additional experiments further support the feasibility of MolToken as a unified molecular foundation model. We will include the results in the revised manuscript and clarify the performance trade-offs relative to domain-specific models.

AuthorsOpenReview ↗

Biological Priors, Sequence Length, and Data Mixture

We thank the reviewer for the suggestions regarding biological priors, long-sequence modeling, and data mixture.

1. Biological Priors and Long-Sequence Modeling

MolToken follows the principles of general-purpose learning and scalability emphasized in The Bitter Lesson. Our goal is to enable models to leverage a broader range of chemical and biological data through a unified representation and data-driven learning, without requiring separate domain-specific representations.

Avoiding predefined residue-level priors is a deliberate design choice in MolToken. FDDFS represents molecules through atoms and chemical bonds, allowing structural patterns to be learned through data-driven tokenization. This enables small molecules, peptides, proteins, DNA, and RNA to be mapped into the same model space. Within the supported atom and bond types, this representation also naturally accommodates nonstandard residues and chemically modified structures, providing broader applicability.

Regarding three-dimensional information, we adopt two-dimensional molecular topology as the foundation for unified modeling because it provides a consistent and widely available structural representation across domains, without restricting training to data with 3D structural annotations.

To examine long-sequence modeling, we further analyzed the FDDFS+BPE sequence-length distribution of all 323,079,504 records used in pretraining:

StatisticValue
Training records323,079,504
Mean BPE sequence length30.62 tokens
Median21 tokens
99th percentile133 tokens
Records requiring runtime truncation437 (0.000135%)

The results show that, after unified tokenization, the vast majority of molecular records in our pretraining data have relatively short sequence lengths and can be efficiently processed by standard Transformers. This observation demonstrates that a unified atomic-level representation does not necessarily lead to unmanageable sequence lengths and provides a practical technical foundation for further extending the framework to longer molecular sequences.

We therefore believe that unified atomic-level representation, data-driven structural learning, and a scalable Transformer architecture are key design advantages of MolToken for cross-domain molecular modeling at scale.

2. Data Mixture and Scaling Analysis

The central research question of this work is how to integrate data from different chemical and biological domains into a unified model space and investigate the scaling behavior of this framework. Accordingly, we keep the five-domain data mixture fixed throughout joint pretraining to ensure consistent data distributions across model sizes and compute budgets.

The fixed data mixture is an intentional part of our scaling-law experimental design, not an omission in the experimental setup. Optimizing sampling weights across domains constitutes a separate data-allocation and multi-objective optimization problem, beyond the scope of our investigation into unified representation and scaling.

More importantly, our central conclusions do not depend on the current mixture being optimal. Even though peptides account for only approximately 0.17% of training tokens, jointly pretrained models achieve lower loss than all evaluated peptide-only models on a common test set, demonstrating the effectiveness of the unified framework under a highly imbalanced data distribution.

Therefore, systematic optimization of data-mixture proportions is not a prerequisite for validating our central contributions. It is a separate research direction that can be further explored using the unified framework established by MolToken.

Dear Reviewer Tbhw,

As the discussion period closes in one day, we would like to gently follow up and ask whether you have had a chance to review our rebuttal. We would be grateful for any further feedback.

Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their thoughtful evaluations and constructive feedback. Before addressing individual concerns, we would like to clarify the central motivation and contributions of MolToken and address several common concerns raised by the reviewers.

1. Core Motivation and Contribution

MolToken is deeply inspired by The Bitter Lesson [1], Richard Sutton's influential 2019 essay. Widely regarded as one of the most important perspectives on large-scale AI research, its central insight is that long-term progress in AI is often driven by general-purpose learning methods that can effectively leverage increasing computational resources, whereas excessive reliance on handcrafted, domain-specific knowledge and rules may limit scalability. Advances in deep learning and large language models have provided compelling empirical evidence for the importance of generality and scalability.

We believe this insight is equally relevant to molecular modeling. Despite the abundance of heterogeneous chemical and biological data, small molecules, proteins, DNA, and RNA are typically modeled using distinct representations and specialized frameworks, making it challenging to integrate these data into a shared learning space.

The central contribution of MolToken is the establishment of a unified molecular representation through atomic graphs and FDDFS+BPE tokenization, which maps small molecules, peptides, proteins, DNA, and RNA into a shared token space and enables a single standard Transformer to learn jointly across all five domains. This provides a unified foundation for integrating increasingly diverse molecular data while scaling model capacity and computational resources.

Building on this unified representation, we further investigate scaling laws for joint molecular pretraining. Our results provide initial evidence of predictable performance improvements with increasing model size, training data, and computational resources, offering empirical guidance for scaling unified molecular models.

2. Unified Pretraining and Downstream Applications

Regarding the reviewers' concerns about the benefits of joint pretraining and downstream performance, we emphasize that the key advantage of unified modeling lies in enabling heterogeneous molecular data to contribute to scalable learning within a shared model space, thereby expanding the data resources available for continued model scaling. Our experiments show that domain-specific training can achieve better in-domain performance in certain settings, whereas joint pretraining brings multiple molecular domains into a shared parameter space, creating broader opportunities for scaling with increasing data diversity, model capacity, and computation.

Our downstream evaluations are intended to explore the applicability of unified molecular pretraining across different domains and demonstrate its potential for generality and scalability. We evaluate molecular generation, protein-related tasks, and DNA/RNA modeling to investigate whether a shared pretrained model can be adapted to diverse applications. These experiments provide initial evidence of practical utility and motivate further exploration at larger scales and across a broader range of downstream tasks.

In response to the reviewers' suggestions, we have also conducted additional experiments and analyses, including test-loss extrapolation, alternative scaling-law fits, cross-domain adaptation and data efficiency, FDDFS ablations, and evaluation-data overlap audits. Detailed results are presented in our individual responses.

We believe MolToken offers a promising technical path toward more effectively integrating and leveraging large-scale heterogeneous chemical and biological data, advancing molecular modeling toward greater data scalability, computational scalability, and broader domain coverage.

[1] Sutton, R. S. (2019). The Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.