acceptodds
Under review as a conference paper at ICLR 2027

Learning the Language of the Microbiome with Transformers

Abstract

Self-supervised pretraining has become central to biological machine learning, yet microbiome data remains comparatively underexplored in terms of both modeling approaches and evaluation frameworks. To address this gap, we present Atlas, a pretraining dataset of 539,308 microbiome datapoints from the MGnify database. Using Atlas, we train the Waypoint family of microbiome foundation models: a series of GPT-2 style causal language models ranging from 6M to 170M parameters. We also introduce Compass, a curated benchmark of eight predictive tasks spanning biome classification, drug-microbiome interactions, drug degradation, and infant gut development. Using this benchmark, we compare the performance of Waypoint models against classical baselines and the existing Microbial General Model (MGM) foundation model. Our results show that pretraining consistently improves downstream task performance, that both dataset scale and tokenization strategy affect model quality, that pretraining supports favorable scaling behavior and that representations learned during pretraining transfer between microbiome domains. Furthermore, pretrained transformer models outperform classical methods on Compass tasks with roughly 10,000 or more training examples, a number that is attainable for modern microbiome studies. Finally, we demonstrate that the Waypoint models achieve state-of-the-art performance among publicly available microbiome foundation models. Overall, our work highlights the importance of large-scale self-supervised pretraining and tokenisation choices in this domain and establishes Atlas, Compass, and the Waypoint models as valuable resources for the research community in this emerging field.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
microbiome representation learning, self-supervised learning, foundation models, taxonomic tokenisation, transfer learning, pretraining, large language models, biological language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_AzdSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a family of transformer-based foundation models (FMs) for microbiome data, named Waypoint, with sizes ranging from 6 to 170M parameters. The authors also present a benchmark of downstream tasks containing classification and regression tasks. The FMs are pretrained on ~540k microbiome compositional samples extracted from the MGnify database. The paper compares the proposed FMs at different model sizes to traditional ML-based models and an existing microbiome FMs (MGM). The results show that Waypoint models outperform the MGM model and are competitive against traditional models like XGBoost, even outperforming it for larger microbiome datasets (10k+ samples).

Critical strengths
  • This work presents a foundation model for microbiome compositional data which are traditionally represented as a normalized (CLR, for example) vector of the read counts per taxon. Embeddings from the FMs present a different way to represent microbiome data.
  • The paper builds on ideas from other recent papers, such as Zhang2026, and extend it by bringing in more pretraining data and the creation of a benchmark of downstream microbiome predictive tasks.
  • I appreciate the amount of information in supplementary material that help me better assess the work.
Critical weaknesses
  1. As stated earlier, the paper has limited methodological novelty. One of the few claimed methodological contribution is the “fallback” taxa selection in the tokenizer; however, the authors do not present a systematic evaluation of its effective performance contribution to the model.

  2. The data utilized for pretraining is a collection from many microbiome studies present in Mgnify. The paper does not discuss/address the bias present in individual datasets, which is a well-known characteristic of microbiome data. Each study follows an independent protocol, uses different sequencing platforms, distinct taxonomic classification pipelines, different quality and sequence filtering steps, etc; all of which have a strong influence in the final compositional abundance profile of the samples. Even though MGnify attempts to harmonizes the computational processing of data across studies, it does not guarantee that the resulting profiles are free of batch effects, given that uniformized computational processing cannot undo the differences introduced in the wetlab and data generation. Additional to that, the authors pulled data from different versions of MGnify, and different versions use different processing steps, classifier, and reference databases (as described in their website/papers). A simple check would be to plot the samples embedding and hue it by their MGnify version.

  3. Still related to the batch effect issues mentioned above, the order of the taxa in the input sequence, which is the only abundance information the model receives, is defined by z-scores computed with the mean and standard deviation of each taxon over the entire pretraining corpus (~485k samples). However, this corpus mixes 16S amplicon, shotgun and assembly data, MGnify pipeline versions v1.0 to v5.0 (with different classifiers and reference databases, etc), and over thousands of studies with different extraction protocols, primers and sequencing depths, all of which shift the relative abundances systematically by source. Then, the pooled mean and standard deviation are mixture statistics, and the z-score of a taxon partly measures how far a sample is from the average of the sources that dominate Atlas, not only its biology. This suggests that the token order, and consequently the next-token prediction objective, may encode technical provenance. The problem gets worse in the downstream tasks, where datasets processed with yet other pipelines are reordered using statistics they were never drawn from. This adds a second distribution shift on top of the vocabulary mismatch the authors already discuss . The paper does not discuss this, and it affects the interpretation of the main results. The biome structure observed in the embeddings in Figures 1C and 1D could partly reflect study, modality or pipeline. Additionally, the cross-domain transfer and scaling conclusions cannot be separated from technical confounding. The 10k crossover is also confounded, since the MGnify tasks (1 and 2) share the pretraining statistics while the other datasets do not. Again, simple checks would be to color the embeddings by modality, pipeline version and study. Other important analysis would be to compare the z-score ordering against a plain within-sample abundance ranking. Finally, the authors did not make it clear whether zeros are included when computing mu_t and sigma_t.

  4. The results presented do not show clear benefit of the proposed FMs when compared to XGBoost. The authors do not present a proper hypothesis test analysis. Additionally, the paper presents a series of models but only the best one, selected after the fact (after seeing the performance on the downstream tasks), is compared against XGBoost on different downstream task's sample size. It shows that the proposed 45M version of their FM beats XGboost on a datasets with 10k+ samples, which is impractical for most of the microbiome studies, where only hundreds of samples are usually available due to the cost and time to collect microbiome samples. See [1] for example, where all the 84 microbiome studies collected have less than 2k samples, with the large majority with a few hundreds of samples. Also, as mentioned above, from the 3 tasks with 10k+ samples, tasks 1 and 2 are from MGnify, so we cannot be sure whether the improvement over XGBoost for these tasks are due to sample size or due to be drawn from the pretraining dataset.

  5. The paper does not present clearly what’s the pretraining task until you read the supplementary material. The main paper should show how a microbiome sample is represented as sequence of tokens so it can be fed to a transformer-based model.

Minor points:

  • The fact that an approach is comparatively underexplored is not, by itself, a sufficient motivation. The paper should identify specific limitations of existing methods and explain why the proposed approach is suited to addressing them. This motivation should be grounded in the characteristics of microbiome data. For example, how could the transformer architecture or training objective account for the compositional nature of these data?
  • The relatively extreme low MLP results seems to indicate that something is wrong with its training. I suggest the authors to carefully revise the MLP experiments.

Refs: [1] Car Reen Kok, Nisha J Mulakken, James B Thissen, Jose Manuel Martí, Ryan Lee, Jacob B Trainer, Andre R Goncalves, Hiranmayi Ranganathan, Aram Avila-Herrera, Crystal J Jaing, Nicholas A Be, Meta2DB: curated shotgun metagenomic feature sets and metadata for health state prediction, Bioinformatics, Volume 42, Issue 7, July 2026, btag422, https://doi.org/10.1093/bioinformatics/btag422

Questions for authors

I don't have additional questions.

Reviewer_C481OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary This paper investigates self-supervised representation learning for taxonomic microbiome profiles and introduces three resources: Atlas, a corpus of 539,308 profiles collected from MGnify; Waypoint, a family of GPT-2-style causal transformers ranging from 6M to 170M parameters; and Compass, a benchmark containing eight prediction tasks from four microbiome datasets. Each microbiome is represented as a sequence of genus-level taxonomic tokens ordered by z-scored relative abundance. When genus-level assignments are unavailable, the tokenizer falls back to the most specific available higher rank. The models are pretrained with next-token prediction and subsequently fine-tuned end-to-end using the final non-padding token as the sample representation.

Critical strengths

1 Important Problem and Valuable Resources: Microbiome representation learning is substantially less standardized than protein or genomic language modeling. Atlas, Compass, and the released checkpoints could therefore be useful community resources. The corpus covers multiple sequencing modalities and more than 4,100 studies, and the paper provides concrete preprocessing and training details. 2 Broad Empirical Scope: The paper evaluates nine transformer configurations, pretrained and randomly initialized variants, four classical model families, and MGM. It also reports per-task macro-F1, ROC-AUC, PR-AUC, and balanced accuracy, rather than relying exclusively on the aggregate score. 3 Useful Pretraining Controls: Comparing pretrained models with architecture-matched random initializations gives clear evidence that the downstream improvements are not explained solely by transformer architecture. The comparison between MGM and the architecture-matched 6M-MGM configuration is also informative, even though it does not isolate all causal factors. 4 Attention to Vocabulary Mismatch: Table S4 shows that token coverage varies substantially, from complete coverage for MGnify tasks to only 59.8% total-token coverage for Roswall et al. Evaluating classical models both with and without out-of-vocabulary taxa is a thoughtful attempt to expose this important limitation. 5 Cross-Domain Investigation: The Only-human, Only-gut, Exclude-human, and Exclude-gut experiments go beyond a standard benchmark comparison. The 110,000-sample controlled setting is particularly useful because it partially separates corpus composition from corpus size. 6 Clear Exposition: The central method and conclusions are easy to follow. The limitations section appropriately acknowledges loss of quantitative abundance information, vocabulary mismatch, incomplete model comparisons, and the limited domain coverage of Compass.

Critical weaknesses

1 Potentially Severe Leakage from Non-Grouped Downstream Splits: The most important concern is the splitting protocol. The supplement states that datasets use random 80/10/10 splits and explicitly groups observations only for the Mastrorilli dataset. The Shi data contain multiple drug perturbations derived from the same source communities; random row-wise splitting can therefore place closely related versions of one community in both training and test sets. This is especially problematic for task 3, whose target is the source community itself. Similarly, the Roswall dataset is longitudinal, but the paper does not state that all samples from one infant are assigned to the same split. If an infant appears in both training and test sets, tasks 7 and 8 may exploit subject identity rather than generalize to unseen infants. These issues could substantially inflate several Compass results and must be resolved with group-based splits. 2 Pretraining and MGnify Evaluation Are Not Split at the Study Level: Atlas is randomly divided at the individual-profile level, while Compass tasks 1 and 2 use the held-out portion. Samples from the same MGnify study can consequently appear in pretraining and downstream test data. Study-specific sequencing protocols, taxonomic pipelines, locations, and batch effects may make biome labels predictable without demonstrating generalization to new studies. Deduplication and study-held-out evaluation are necessary, especially because the corpus combines several MGnify pipeline versions. 3 The Claimed Scaling Result Is Not Yet Well Established: The downstream improvement from 6M to 170M is very small, the 45M model is the best aggregate model, and Figure S7 shows flat or negative trends for several individual tasks. Only one pretraining run appears to be performed per architecture, with three downstream replicates that do not measure pretraining variance. Architectures, parameter count, stopping time, hardware, and effective optimization progress all vary together; the paper itself notes that smaller models have not converged. An ordinary least-squares line over these configurations is insufficient evidence for a general scaling law. 4 Randomly Initialized Models May Be Under-Tuned: All transformer sizes use essentially one fine-tuning recipe. Larger models trained from scratch may require different learning rates, regularization, warmup, or training duration. Their negative trend in Figure 4 may therefore reflect optimization choices rather than an inherent inability to benefit from scale without pretraining. Hyperparameter sweeps for representative pretrained and non-pretrained sizes are needed before drawing this conclusion. 5 Tokenization Contributions Are Not Cleanly Isolated: The comparison between MGM and 6M-MGM changes both the pretraining corpus and fallback tokenization, so it cannot identify which factor produces the improvement. The non-pretrained comparison provides some evidence about tokenization, but a full factorial experiment using the same architecture, training data, objective, and compute is missing. Moreover, genus versus species tokenization is selected from sequence-length statistics alone, without evaluating downstream performance. 6 The Sequential Language-Modeling Formulation Needs Stronger Justification: A microbiome profile is fundamentally a compositional set rather than a naturally ordered sequence. The imposed order depends on corpus-level z-scored abundance, yet the paper provides no permutation test, order ablation, or comparison with set-based architectures. Because exact abundance values are discarded, two communities with different compositions may map to the same rank ordering. Comparisons with DeepSets, Set Transformers, or abundance-aware token embeddings would clarify whether causal attention is responsible for the gains. 7 Baseline Evaluation Is Limited: The XGBoost search tunes only maximum depth while fixing the number of trees and learning rate. The study omits stronger microbiome-specific or compositional baselines, including elastic-net logistic regression with nested tuning, kernel methods, and modern abundance-aware/set-based neural models. Given that the aggregate advantage over XGBoost is roughly one percentage point and some classical baselines win on tasks 4 and 7, stronger tuning could change the main ranking. 8 The Aggregate Compass Score Is Difficult to Interpret: The score averages macro-F1 and an R² value clipped at zero, then gives equal weight to four datasets. These metrics have different meanings, and clipping negative R² conceals the magnitude of regression failures. The aggregate can also obscure the pronounced task-level variability visible in Figure S3. No confidence interval, paired statistical test, or uncertainty propagation is provided for the aggregate comparisons. 9 Cross-Biome Transfer Evidence Is Suggestive but Not Conclusive: The subset experiments use only the 6M model and apparently one pretraining run per condition. Differences between conditions frequently overlap downstream replicate variability, and no statistical test is reported. Ecological domain, sequencing modality, study composition, and taxon distribution remain confounded. The conclusion should therefore be framed as preliminary evidence rather than a definitive demonstration of domain-general representations. 10 Additional Reporting Gaps: The paper does not report how often the 510-token truncation affects samples, despite genus-level sequences reaching length 780. It also does not quantify duplicate or near-duplicate samples, study overlap across splits, class counts, or subject/community overlap. PCA and t-SNE visualizations are qualitative and do not establish representation quality.

Questions for authors

1 Were all samples from the same infant in Roswall et al. assigned to one split? If not, can the authors rerun tasks 7 and 8 using infant-grouped train/validation/test partitions? 2 For the Shi et al. data, are perturbations derived from the same source community shared across splits? How is leakage avoided in task 3 when the prediction target is source-community identity? 3 Can tasks 1 and 2 be evaluated with entire MGnify studies held out from both pretraining and fine-tuning, rather than using a random sample-level split? 4 What deduplication or near-duplicate detection was applied across MGnify studies and pipeline versions? 5 Can the authors provide a factorial ablation separating corpus size from fallback tokenization, using identical architecture, optimization, and compute? 6 How many samples are truncated at 510 taxa, and how does truncation affect biome- and task-specific performance? 7 Would the conclusions hold under random taxon permutations, raw-abundance ordering, absolute/relative abundance embeddings, or a permutation-invariant set encoder? 8 How sensitive are Figure 4 and the negative scaling result for non-pretrained models to size-specific learning-rate, regularization, and training-duration tuning? 9 Can the authors report confidence intervals or paired statistical tests for model comparisons and include independent pretraining seeds for selected configurations? 10 Why is negative R² clipped to zero in the aggregate score? Please report unclipped values and a sensitivity analysis under alternative aggregation or rank-based summaries. 11 Does the fixed full-corpus vocabulary used in cross-biome experiments introduce information from excluded domains, and what happens when each subset constructs its own vocabulary? 12 Can the authors compare against stronger abundance-aware and permutation-invariant baselines and perform a more comprehensive XGBoost search?

Reviewer_gsBJOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents Atlas, a pretraining dataset of about 539k microbiome samples collected from the MGnify database, and Waypoint, a family of GPT-2-style models (6M–170M parameters) pretrained on taxa sequences ordered by abundance. The authors also introduce a fallback tokenization that represents taxa at the genus level or the closest higher rank when the genus is unavailable, and Compass, a benchmark comprising eight tasks primarily focused on the gut microbiome.

Critical strengths

The paper addresses a relevant problem, namely the lack of large pretraining corpora and standardized benchmarks for microbiome foundation models. The authors release the dataset, benchmark, and pretrained models, which can be useful to the community.

Critical weaknesses
  1. The paper essentially proposes to scrape an existing public database (MGnify), train existing architectures (GPT-2) using the same input representation as previous work (taxa ordered by z-scored abundance, as in MGM), and evaluate them on a new collection of tasks. The only methodological element, the fallback tokenization (which consists of representing each taxon at the genus level and, when the genus is not available, using the most specific higher rank instead), is a minor preprocessing heuristic.

  2. As described in Section 2.1, the pretraining dataset and the held-out benchmark dataset (used for tasks 1 and 2) are split at the sample level rather than at the study level. This means that samples from the same study can easily be present in both the pretraining data and the benchmark. Since samples from the same study are typically very similar and share the same biome label, this could bias the overall assessment toward the pretrained models, which are likely to perform better in this setting.

  3. Several of the main findings are largely expected and do not bring new insights. For example: “Larger models achieve lower pretraining loss” (Section 3.1) is a well-known behaviour of language models; “Waypoint becomes more competitive as dataset size increases” (Section 3.3) is expected for transformers compared to tree-based methods such as XGBoost, and is moreover supported by only two or three tasks above the 10,000-example threshold; and the degradation of larger non-pretrained models (Section 3.4) is consistent with simple overfitting on small datasets. In addition, the gains are small: the advantage over the strongest classical baseline (XGBoost) is limited (Figure 2), and the improvement with model size after pretraining is only about 0.004 across a 30× increase in parameters (Figure 4).

Questions for authors
  1. The split between pretraining and benchmark data for tasks 1–2 appears to be random at the sample level rather than at the study level. Since samples from the same MGnify study are typically very similar and share the same biome label, near-duplicates of test samples may be present in the pretraining corpus, inflating the apparent benefit of pretraining on these tasks. Could the authors report results with a study-level split? Notably, how would the comparison with XGBoost change in this setting?

  2. What does this approach really bring compared to the classical setting, where models are trained directly on taxa abundances? The comparison with XGBoost involves different information in both directions: XGBoost uses exact taxon abundances, whereas the Waypoint models only see the rank order of taxa but benefit from large-scale pretraining. Since the overall advantage of the Waypoint models is small (Figure 2), could the authors train an XGBoost model on the same rank-based representation to separate the effect of the input representation from that of pretraining?

  3. Could the authors provide an ablation with the same pretraining data and model, with and without the fallback tokenization, to isolate its actual contribution?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.