- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes MolToken, a unified molecular foundation model that represents small molecules, peptides, proteins, DNA, and RNA using a shared atom-level graph representation. The authors introduce a unified graph tokenization approach based on FDDFS serialization and BPE, enabling joint pretraining with a single autoregressive Transformer. The paper primarily investigates the scaling behavior of the proposed model, particularly the relationship between joint and single-domain pretraining as model size and computational budget increase. Several downstream tasks are also evaluated to demonstrate the applicability of the pretrained model across different molecular domains
Critical strengths
- The paper provides an interesting approach to representing different molecular domains within a unified atom-level graph framework. The combination of reversible graph serialization and a shared vocabulary enables joint modeling without relying on domain-specific encoders.
- The authors conduct a relatively comprehensive analysis of the scaling behavior of MolToken, including compute-optimal scaling and comparisons between joint and single-domain pretraining. These experiments provide useful empirical observations about how different molecular domains respond to increased model and training scales.
Critical weaknesses
- The discussion of existing all-atom molecular foundation models could be more comprehensive. For example, ATOMICA (Fang et al., 2025) is a relevant prior work that learns unified atom-level representations across different molecular modalities. Although ATOMICA focuses on 3D intermolecular interactions and differs from MolToken in its modeling objectives, a discussion of this work would help clarify the novelty and positioning of MolToken relative to existing unified molecular representation approaches.
- While the proposed approach provides a unified representation for multiple molecular domains, the experimental results do not sufficiently demonstrate whether this unification leads to improved cross-domain knowledge sharing.
- While the reported 71.8× compression is useful, sequence compression is an expected benefit of BPE. Its impact on actual training efficiency and downstream performance remains unclear.
Questions for authors
- The paper provides detailed observations of how joint and single-domain pretraining behave at different scales. However, it is not entirely clear what general insights these results offer for the development of unified molecular foundation models. For example, how could these findings inform future model design, training strategies, or data allocation across molecular domains?
- Since one of the motivations for unified molecular modeling is to enable learning across different molecular domains, evaluating whether joint pretraining improves downstream performance over single-domain pretraining would help strengthen the paper's conclusions.
8 comments
3. Practical Value of BPE Compression
A unified atomic-level representation provides a common foundation for modeling different molecular types, but also introduces longer sequences. BPE plays an important role in MolToken by reducing the token cost of this unified graph representation, allowing heterogeneous molecular data to be processed more efficiently by a shared autoregressive model.
Token compression itself provides direct practical value. The computational and memory costs of Transformers depend strongly on sequence length. Reducing the number of tokens therefore directly lowers sequence-processing costs and allows more molecular data to be processed under the same compute budget. The 71.8× compression ratio reported in our paper demonstrates a substantial reduction in token count relative to the original FDDFS symbol sequence, significantly improving the computational scalability of unified atomic-level representations.
To further evaluate this effect on actual pretraining data, we analyzed all 323,079,504 training records. After BPE, the mean sequence length is 30.62 tokens, with a 99th percentile of 133 tokens. Under the 512-token context limit, only 437 records (0.000135%) require runtime truncation.
These results demonstrate that FDDFS+BPE compresses atomic-level representations across all five molecular domains into sequences suitable for a shared Transformer. The 71.8× token compression is not merely a representation-level statistic but a practical technical advantage supporting efficient training and continued scaling of unified molecular models. We will include these statistics in the revision and clarify the distinction between token compression and end-to-end training throughput.
Scaling Implications and Downstream Benefits
We thank the reviewer for the questions regarding the broader implications of our scaling analysis and the downstream benefits of joint pretraining.
1. Implications of Scaling Laws for Unified Molecular Models
A key contribution of MolToken is demonstrating that heterogeneous chemical and biological data can be jointly modeled within a unified framework while exhibiting predictable scaling behavior. This aligns with the central insight of The Bitter Lesson: general-purpose learning frameworks capable of leveraging increasing data and computation provide a promising path toward scalable intelligence.
Using 67 model configurations across 10 compute budgets, we identify compute-optimal scaling trends for unified molecular pretraining. The scaling exponents for model size and training tokens are approximately 0.543 and 0.457, respectively. Guided by these relationships, we trained a 0.55B model, achieving a predicted Test Loss of 4.0315 versus a measured 4.0482, with an absolute error of only 0.0166.
Importantly, the emergence of predictable scaling laws under joint pretraining provides empirical evidence that our unified representation supports systematic and scalable learning across heterogeneous molecular domains. Beyond guiding model-size and compute allocation, this demonstrates the potential of MolToken as a general-purpose framework for integrating increasingly diverse molecular data and computational resources.
Comparisons between joint and single-domain pretraining further reveal domain-dependent scaling behavior, providing a basis for future optimization of model capacity, training budgets, and domain sampling proportions.
2. Downstream Comparison Between Joint and Single-Domain Pretraining
As reported in our response on Cross-Domain Knowledge Sharing, we conducted additional downstream ablations comparing random initialization, small-molecule-only pretraining, and five-domain joint pretraining under matched model sizes and pretraining compute budgets.
The results show that joint pretraining achieves better performance on DNA promoter classification (0.8966 vs. 0.8400) and RNA translation-efficiency prediction (0.6387 vs. 0.6190) than small-molecule-only pretraining. Meanwhile, small-molecule-only pretraining performs better on MOSES validity (0.9878 vs. 0.9692). On protein GB1, joint pretraining has not demonstrated consistent improvements.
These findings demonstrate the downstream adaptability of unified pretraining across molecular domains while revealing task-dependent performance trade-offs. We will include the complete ablation results in the revised manuscript and further discuss the relationship between domain composition and downstream performance.
Related Work and Cross-Domain Knowledge Sharing
We thank the reviewer for the thoughtful comments on related work and cross-domain knowledge sharing.
1. Comparison with ATOMICA
We appreciate the reviewer for highlighting ATOMICA, an important related work. We believe ATOMICA represents a valuable exploration of unified biomolecular representation learning and provides important insights into modeling molecular structures and interactions across different molecular types.
Although both approaches investigate unified modeling across molecular types, they differ substantially in their research objectives, modeling targets, and technical approaches.
ATOMICA primarily focuses on learning representations of intermolecular interactions. It models molecular complexes and their interactions using three-dimensional geometric information, employing graph neural networks (GNNs) to learn structural and interaction features.
In contrast, MolToken focuses on unified representation and scalable generative modeling of individual molecules across heterogeneous chemical and biological domains. We convert small molecules, peptides, proteins, DNA, and RNA into two-dimensional atomic graphs and construct a shared discrete token space through FDDFS+BPE. This enables autoregressive pretraining of a single standard Transformer across all five domains and allows us to further investigate scaling laws within this unified framework.
Thus, ATOMICA and MolToken explore cross-molecular-type modeling from complementary perspectives: geometric interaction representation and unified tokenization for scalable generative learning, respectively.
We thank the reviewer for this suggestion and will discuss ATOMICA and related work in the revised manuscript to clarify the connections and distinctions between these research directions.
2. Evidence of Cross-Domain Knowledge Sharing
The Bitter Lesson emphasizes the importance of general-purpose learning methods that can leverage increasing data and computational resources. We believe this perspective offers an important motivation for molecular foundation models: establishing a shared representation and learning space enables heterogeneous chemical and biological data to contribute to scalable pretraining within a single model.
Following this principle, MolToken converts small molecules, peptides, proteins, DNA, and RNA into unified atomic-level graph representations and constructs a shared token space through FDDFS+BPE tokenization, allowing all five domains to participate in parameter learning within the same Transformer.
To further evaluate the practical benefits of this framework, we conducted additional pretraining ablations during the rebuttal period. We controlled model size (approximately 41M parameters) and nominal pretraining compute (5×1017 FLOPs) and compared random initialization, small-molecule-only pretraining, and five-domain joint pretraining.
On DNA promoter classification, the three settings achieve accuracies of 0.8287, 0.8400, and 0.8966, respectively. On RNA translation-efficiency prediction, their Pearson² scores are 0.3943, 0.6190, and 0.6387, respectively.
These results demonstrate that unified pretraining enables a shared model to learn representations applicable to downstream tasks across different molecular domains. Since joint pretraining also includes target-domain data, these experiments do not independently quantify the contribution of knowledge transferred from other domains, but provide an experimental basis for further investigating cross-domain knowledge sharing.