- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Using binary IBD prediction from genus-level microbiome profiles as a test bed, the paper asks whether large-scale pretraining and Transformer encoders are what make microbiome representations useful for phenotype prediction. It compares the pretrained MGM foundation model (34M parameters) with classical models, task-trained models (FT-Transformer, DeepSets) and minimal models, including a mean-pooled taxon-embedding probe (SimpleEmb, 1.1M parameters).
Critical strengths
The evaluation escalates from a held-out split to grouped cross-validation to independent cohorts. Within the cohort, it benchmarks a pretrained foundation model against strong classical baselines: HistGradientBoosting, Random Forest, XGBoost, and SIAMCAT. The paper also reports results that run counter to its own model. SimpleEmb is not the best under-grouped CV, and Random Forest is among the most robust on the external cohorts. This matters because claims for microbiome foundation models often rest on random splits. With subject-level splits and the foundation model included in the external evaluation, this design could settle whether pretraining helps. The candid reporting makes the work a credible starting point.
The paper probes four representations for both disease and study identity: raw input, the learned embedding, pretrained MGM, and a randomly initialized MGM of the same architecture. It then repeats the cohort probe after balancing disease composition. Together, these controls show three things about the cohort signal:
- it is already present in the raw input (0.933 cohort-probe accuracy);
- it is not introduced by pretraining (0.930 for pretrained MGM versus 0.964 for random initialization);
- it is not explained by disease composition alone.
This is a cheap, reusable diagnostic that microbiome benchmarks could adopt.
Critical weaknesses
The reported results can't be relied on as they stand
MGM results are contradictory. Figures 12, 14, and 15 report MGM at 47-51% accuracy, below the 55.7% majority baseline, and Figure 15 reports a 46% AUC. Table 1 reports 92.8% accuracy for the same model. Figure 14 is even titled Week 1.
Inconsistent statistics: Table 10's McNemar counts do not reconcile with Table 2's reported accuracies. For HistGradientBoosting, Table 10 favors SimpleEmb while Table 2 favors HistGradientBoosting.
Incomplete leakage control
Grouping removes exact-duplicate inputs but not same-subject samples, so subject-level leakage remains unverified
The merged -> clean transfer appears contaminated. The clean -> merged test set loses 105 overlapping samples, yet merged -> clean lists the full 3,350 training and 167 test samples
On the original split, 1-NN retrieval beats the trained classifier, consistent with near-duplicates across that split
Likely mis-mapped genus labels
The cohort's most prevalent genera include labels consistent with insect endosymbionts ("Candidatus Reg...", "Candidatus Ham...") and a plant genus (Hypseocharis)
Fig. 7E shows that 14 of 20 curated IBD genera are absent
If the mapping is wrong, the interpretation sections don't stand. If cohorts were also mapped inconsistently, the cohort effects in Section 6 could be inflated. This is critical because the paper's conclusions rest on these numbers. Stale figures alone could be a reporting fix, but the leakage and taxonomy issues likely require re-running substantial parts of the study
The paper claims cohort coverage, not capacity or pretraining, governs generalization. That claim rests on adding one cohort, with a clear gain on a single target (32% -> 49%). The gain is recall on disease-only cohorts. In a single-class cohort, recall cannot distinguish better discrimination from a shifted decision threshold.
Adding the cohort also changes three other things at once: it enlarges the training set; it changes the case (control mix, per Section 6.3's counts, and cohort 10317's ratio differs from that of cohort 2538); it introduces questionnaire-based diagnoses, because cohort 10317 is the American Gut Project, a citizen-science study. The paper does not mention this
Untested foundation-model conclusion: no cross-cohort prediction result is reported for MGM. The conclusion that pretraining and scaling don't help transfer is therefore untested in the setting where foundation models claim much of their value. Without it, what remains largely restates established findings.
Either weakness alone would justify rejection, and together they are unlikely to be resolved within a rebuttal. In my view, the experiments don't support the main claim
Questions for authors
Table 1 reports MGM+MLP at 92.8% accuracy, and Table 2 at 85.3%. Figures 12, 14, and 15 report 46.7-50.9% accuracy, and Figure 15 reports a 46.3% AUC. Which pipeline produced each number? Was the encoder fine-tuned or frozen? Were MGM's own tokenization and normalization used, and over how many seeds? Was MGM evaluated for prediction on the independent cohorts?
For paired predictions, the difference in pooled accuracy equals (B - C)/N. Table 10 reports B = 75 and C = 64 for HistGradientBoosting, and B - C = 9 for MGM+MLP. How do these counts reconcile with Table 2's accuracies? What is N, which SimpleEmb configuration was used, and were predictions from the three CV repeats pooled?
How many exact-duplicate groups exist in clean_2538, and why do duplicates occur? Do subjects contribute multiple samples? Do the within-cohort results hold under subject-level grouping, including the 97.0% kNN accuracy reported on the original split?
The clean -> merged test set shrinks from 838 to 733 after decontamination. Yet merged -> clean lists the full 3,350 training and 167 test samples. How many clean test samples were in merged's training set, and were any removed?
Figure 13D's 30 most prevalent genera (62–100% of samples) include labels consistent with insect endosymbionts ("Candidatus Reg...", "Candidatus Ham...") and a plant genus (Hypseocharis). Figure 7E marks 14 of 20 curated IBD genera as absent. How was the token-index-to-genus mapping verified? How were the cohorts mapped into the 1,226-taxon vocabulary, and were they processed with the same pipeline?
Which model was used, and how many cases and controls from cohort 10317 were added?
Did adding cohort 10317 also raise positive predictions on healthy samples not used for training (e.g., cross-validated controls)?
Does the gain for cohort 1939 persist when the total training size is held fixed?
Which cohorts defined the domains? Did training actually reduce cohort decodability? Please report cohort-probe accuracy after training, along with the hyperparameters.
What are cohorts 2538, 1939, 11484, 10283, 1629, and FTP in terms of population, sample type, sequencing protocol, and label source?
Cohort 10317 appears to be the American Gut Project, per its public Qiita record. Was it restricted to stool samples, and how were its IBD labels defined? Does it overlap with the separately listed AGP?
Do the source studies of the four disease-only cohorts include controls that would allow AUROC to be reported?
How many samples have fewer than five genera, and how does the label split them? Do the results change when they are excluded?