acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Foundation Models for Microbiome Phenotype Prediction: What Actually Matters?

Abstract

Foundation models for the microbiome increasingly rely on large-scale pretraining and Transformer encoders, but it remains unclear whether these choices address the main difficulties of phenotype prediction from sparse abundance profiles. We revisit this assumption through a controlled study of inflammatory bowel disease (IBD) prediction that disentangles pretraining, architectural inductive bias, representation complexity, and cohort shift. A pretrained 34M-parameter MGM encoder and a 1.1M-parameter embedding model without any pretraining reach comparable held-out accuracy (92.8% versus 92.3%), indicating that neither scale nor pretraining alone determines downstream utility. Permutation-invariant models are similarly strong: DeepSets and a minimal learned-embedding model reach 91.9% and 92.3% in the same structural comparison. Under leakage-robust grouped cross-validation, however, classical models remain highly competitive: HistGradientBoosting reaches 0.979 AUROC, compared with 0.957 for the learned embedding, and a simple presence-vector MLP reaches 0.969. The decisive difficulty appears under cohort shift. When models trained on one cohort are evaluated on independent cohorts, performance drops sharply; on the balanced external cohort, AUROC ranges from 0.621 for the embedding model to 0.668 for Random Forest, far below the 0.974 in-family result. Cohort-probe analyses show that study identity is highly decodable from every representation (cohort accuracy 0.93–0.97 versus disease accuracy 0.76–0.82), and remains so after balancing disease composition, indicating that learned representations are dominated by cohort-associated structure. Crucially, adding a single additional cohort to the training set raises disease identification on an independent cohort from 32% to 49%, whereas gradient-reversal debiasing and model scaling do not. These results indicate that cross-cohort generalization in microbiome phenotype prediction is governed not by model capacity, pretraining, or explicit debiasing, but by the cohort coverage of the training distribution.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
microbiome representation, biological interpretation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_WuarOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Using binary IBD prediction from genus-level microbiome profiles as a test bed, the paper asks whether large-scale pretraining and Transformer encoders are what make microbiome representations useful for phenotype prediction. It compares the pretrained MGM foundation model (34M parameters) with classical models, task-trained models (FT-Transformer, DeepSets) and minimal models, including a mean-pooled taxon-embedding probe (SimpleEmb, 1.1M parameters).

Critical strengths

The evaluation escalates from a held-out split to grouped cross-validation to independent cohorts. Within the cohort, it benchmarks a pretrained foundation model against strong classical baselines: HistGradientBoosting, Random Forest, XGBoost, and SIAMCAT. The paper also reports results that run counter to its own model. SimpleEmb is not the best under-grouped CV, and Random Forest is among the most robust on the external cohorts. This matters because claims for microbiome foundation models often rest on random splits. With subject-level splits and the foundation model included in the external evaluation, this design could settle whether pretraining helps. The candid reporting makes the work a credible starting point.

The paper probes four representations for both disease and study identity: raw input, the learned embedding, pretrained MGM, and a randomly initialized MGM of the same architecture. It then repeats the cohort probe after balancing disease composition. Together, these controls show three things about the cohort signal:

  • it is already present in the raw input (0.933 cohort-probe accuracy);
  • it is not introduced by pretraining (0.930 for pretrained MGM versus 0.964 for random initialization);
  • it is not explained by disease composition alone.

This is a cheap, reusable diagnostic that microbiome benchmarks could adopt.

Critical weaknesses

The reported results can't be relied on as they stand

MGM results are contradictory. Figures 12, 14, and 15 report MGM at 47-51% accuracy, below the 55.7% majority baseline, and Figure 15 reports a 46% AUC. Table 1 reports 92.8% accuracy for the same model. Figure 14 is even titled Week 1.

Inconsistent statistics: Table 10's McNemar counts do not reconcile with Table 2's reported accuracies. For HistGradientBoosting, Table 10 favors SimpleEmb while Table 2 favors HistGradientBoosting.

Incomplete leakage control

Grouping removes exact-duplicate inputs but not same-subject samples, so subject-level leakage remains unverified

The merged -> clean transfer appears contaminated. The clean -> merged test set loses 105 overlapping samples, yet merged -> clean lists the full 3,350 training and 167 test samples

On the original split, 1-NN retrieval beats the trained classifier, consistent with near-duplicates across that split

Likely mis-mapped genus labels

The cohort's most prevalent genera include labels consistent with insect endosymbionts ("Candidatus Reg...", "Candidatus Ham...") and a plant genus (Hypseocharis)

Fig. 7E shows that 14 of 20 curated IBD genera are absent

If the mapping is wrong, the interpretation sections don't stand. If cohorts were also mapped inconsistently, the cohort effects in Section 6 could be inflated. This is critical because the paper's conclusions rest on these numbers. Stale figures alone could be a reporting fix, but the leakage and taxonomy issues likely require re-running substantial parts of the study

The paper claims cohort coverage, not capacity or pretraining, governs generalization. That claim rests on adding one cohort, with a clear gain on a single target (32% -> 49%). The gain is recall on disease-only cohorts. In a single-class cohort, recall cannot distinguish better discrimination from a shifted decision threshold.

Adding the cohort also changes three other things at once: it enlarges the training set; it changes the case (control mix, per Section 6.3's counts, and cohort 10317's ratio differs from that of cohort 2538); it introduces questionnaire-based diagnoses, because cohort 10317 is the American Gut Project, a citizen-science study. The paper does not mention this

Untested foundation-model conclusion: no cross-cohort prediction result is reported for MGM. The conclusion that pretraining and scaling don't help transfer is therefore untested in the setting where foundation models claim much of their value. Without it, what remains largely restates established findings.

Either weakness alone would justify rejection, and together they are unlikely to be resolved within a rebuttal. In my view, the experiments don't support the main claim

Questions for authors

Table 1 reports MGM+MLP at 92.8% accuracy, and Table 2 at 85.3%. Figures 12, 14, and 15 report 46.7-50.9% accuracy, and Figure 15 reports a 46.3% AUC. Which pipeline produced each number? Was the encoder fine-tuned or frozen? Were MGM's own tokenization and normalization used, and over how many seeds? Was MGM evaluated for prediction on the independent cohorts?

For paired predictions, the difference in pooled accuracy equals (B - C)/N. Table 10 reports B = 75 and C = 64 for HistGradientBoosting, and B - C = 9 for MGM+MLP. How do these counts reconcile with Table 2's accuracies? What is N, which SimpleEmb configuration was used, and were predictions from the three CV repeats pooled?

How many exact-duplicate groups exist in clean_2538, and why do duplicates occur? Do subjects contribute multiple samples? Do the within-cohort results hold under subject-level grouping, including the 97.0% kNN accuracy reported on the original split?

The clean -> merged test set shrinks from 838 to 733 after decontamination. Yet merged -> clean lists the full 3,350 training and 167 test samples. How many clean test samples were in merged's training set, and were any removed?

Figure 13D's 30 most prevalent genera (62–100% of samples) include labels consistent with insect endosymbionts ("Candidatus Reg...", "Candidatus Ham...") and a plant genus (Hypseocharis). Figure 7E marks 14 of 20 curated IBD genera as absent. How was the token-index-to-genus mapping verified? How were the cohorts mapped into the 1,226-taxon vocabulary, and were they processed with the same pipeline?

Which model was used, and how many cases and controls from cohort 10317 were added?

Did adding cohort 10317 also raise positive predictions on healthy samples not used for training (e.g., cross-validated controls)?

Does the gain for cohort 1939 persist when the total training size is held fixed?

Which cohorts defined the domains? Did training actually reduce cohort decodability? Please report cohort-probe accuracy after training, along with the hyperparameters.

What are cohorts 2538, 1939, 11484, 10283, 1629, and FTP in terms of population, sample type, sequencing protocol, and label source?

Cohort 10317 appears to be the American Gut Project, per its public Qiita record. Was it restricted to stool samples, and how were its IBD labels defined? Does it overlap with the separately listed AGP?

Do the source studies of the four disease-only cohorts include controls that would allow AUROC to be reported?

How many samples have fewer than five genera, and how does the label split them? Do the results change when they are excluded?

Reviewer_zBEjOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how pretraining, model architecture, and model complexity affect prediction performance and generalization in microbiome-based phenotype prediction. The authors compare pretrained foundation models, deep learning models, and classical machine learning methods for inflammatory bowel disease (IBD) prediction. The results show that large pretrained models and simpler models achieve similar performance within the same dataset. However, performance drops significantly on independent cohorts. Representation probing also shows that cohort information is more easily learned than disease-related information. Finally, adding more cohorts to training improves generalization in some cases, while gradient-reversal debiasing shows limited benefits.

Critical strengths
  1. The paper evaluates microbiome foundation models using grouped cross-validation and independent-cohort evaluation, rather than only within-dataset prediction accuracy. The results show that performance differences between cohorts are much larger than those between model architectures or pretraining strategies. This suggests that cross-cohort generalization is an important issue in microbiome phenotype prediction.

  2. The authors compare various models, including pretrained foundation models, classical machine learning, and simpler deep learning models. They also use representation probing and disease composition balancing to examine cohort-related information in learned representations. These analyses provide evidence that increasing model complexity alone may not be enough to improve generalization across cohorts.

Critical weaknesses
  1. The main claim of this paper is that cohort coverage is more important than model size, pretraining, or debiasing for cross-cohort generalization. However, the improvement from adding more cohorts is mainly observed in one external cohort (1939), while other cohorts show limited improvement. This makes the main conclusion less convincing.

  2. Adding more cohorts also increases the total amount of training data. Therefore, it is unclear whether the improvement comes from cohort diversity or simply from having more training samples. Without experiments controlling for training data size, the claim that cohort coverage is the key factor for generalization is not sufficiently supported.

  3. The authors show that cohort identity is more easily predicted than disease identity from learned representations. However, this does not explain what causes the cohort-specific signals. Disease composition balancing alone cannot rule out other factors, such as demographic differences or data processing methods.

  4. Some experimental results also need further explanation. In Table 9, RandomEmb performs better than learned SimpleEmb, while Table 8 shows higher mean absolute LOO importance for shuffled-label controls than for real data. In addition, some values in Table 5 do not match the description in Section 6.1. These issues make it difficult to interpret the results and assess the reliability of the conclusions.

Questions for authors
  1. It is unclear whether the improvement from adding more cohorts comes from increased cohort diversity or simply from more training samples. Can the authors compare single-cohort and multi-cohort training with the same number of samples? This is important to support the main conclusion of the paper.

  2. Why does adding more cohorts improve performance on cohort 1939 (0.322 to 0.489), but show little improvement on cohorts 10283 and 1629? Could this difference be related to the distributional similarity between cohorts? This needs further analysis before making a general conclusion about cohort coverage.

  3. Why does RandomEmb+MLP outperform SimpleEmb+MLP in Table 9? Also, why is the mean absolute LOO importance higher for shuffled labels than for real data in Table 8? These results raise questions about the benefits of learned representations and the reliability of the attribution analysis.

  4. Section 6.1 reports an AUROC of 0.773 when training on clean data and testing on merged data, while Table 5 reports 0.722 ± 0.118 for the same setting. A similar inconsistency appears when training on merged data and testing on clean data. Can the authors clarify whether these differences come from different evaluation settings or reporting errors?

Reviewer_oe4VOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper is an attempt to evaluate the different components in a microbiome foundation model, such as model architecture, pretraining, cohort used for training. With these evaluations the authors try to answer 1) whether FM embeddings provide a good representation of microbiome samples for predictive downstream tasks; 2) which inductive bias and representation complexities are needed and; finally 3) what component of these models are the limiting factors of model generalization. The analysis is performed by comparing ablations of the MGM model with simple bag-of-embeddings models (SimpleEmb), traditional ML models on abundance data; plus a couple of task-specific structural baselines. Results do not show much difference in the predictive performance on a IBD classification task among the more complex FMs and simple baselines. Also, the paper indicates that cohort coverage of the training data distribution is one of the most influential factors for cross-cohort generalization, particularly if compared to model capacity and pretraining.

Critical strengths

The paper attempts to disentangle the contributions of individual components of a microbiome foundation model (pretraining, architecture, and training data distribution) to model generalization and sample representation. These are important questions, and answering them can help guide the next steps in this line of research.

Critical weaknesses

The major weakness is that the analysis is limited to a single, balanced, and relatively easy downstream task for a particular disease. Even simple ML models trained directly on abundance profiles produced AUROC ~0.98. Any claim about the generalization of the analysis cannot be made. The paper also concludes that cohort coverage of the training distribution is a crucial factor for cross-cohort generalization, which to be honest is not unexpected and extremely well documented in the literature of several domains.

Questions for authors

I encourage the authors to expand their analysis on larger and more diverse set of downstream tasks and resubmit the paper. A starting point could be to use the curated and harmonized collection of labeled microbiome studies in [1].

[1] Goncalves AR, Ranganathan H, Valdes C, Zhu H, Zhang B, Kok CR, Martí JM, Mulakken NJ, Thissen JB, Jaing C and Be NA (2026) Beyond microbial abundance: metadata integration enhances disease prediction in human microbiome studies. Front. Microbiol. 16:1695501. doi: 10.3389/fmicb.2025.1695501

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.