acceptodds
Under review as a conference paper at ICLR 2027

WAVEBench: Assessing Generalization of Genome Language Models in Wastewater Virus Detection

Abstract

Genome language models (gLMs) are trained on DNA sequences to illuminate the relationship between genotype and phenotype. This goal is only useful if models are able to extend their reasoning beyond annotated data, or generalize out-of-distribution. However, this evaluation is confounded, since all biological sequences descend from the same origin. We introduce WAVEBench, a benchmark that withholds whole taxa, at ranks from species to class, from the labelled folds used to train a downstream probe. This way, each held-out taxon is novel to the probe. To test generalization, we devised a fold-assignment algorithm that, given any hierarchy of relatedness, produces a cross-validation partition where each fold's held-out data is stratified into progressively more distant novelty ranks from the remaining folds. We apply it to novel virus identification in wastewater metagenomics, a timely public-health surveillance task where the evolutionary distribution shift is intrinsic. We evaluated a logistic probe on 51 models from 11 gLM families and compared them to a 5-mer baseline. Median AUROC falls from species to class, but individual models do not decline uniformly; different ranks contain different clades, so these comparisons do not isolate an effect of evolutionary distance. Also, performance varies substantially relative to a k-mer composition baseline, with a subset of models achieving higher leaderboard AUROC. Overall, we contribute a fold-construction algorithm, a benchmark on a pressing public-health problem, and an assessment of gLMs' ability to generalize to taxa withheld from downstream training.

Primary area:
datasets and benchmarks
Keywords:
Genome Language Models, Out-of-Distribution Generalization, Taxonomic Holdout, Metagenomics, Wastewater Surveillance, Viral Detection

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_T5phOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces WAVEBench, a benchmark for assessing downstream out-of-taxon generalization of gLMs in wastewater viral detection. Specifically, it uses taxonomic hierarchy to do the data split, withholding whole taxa at five taxonomic levels, from species to class, to evaluate viral versus cellular classification on taxa withheld from downstream training. The work evaluates 51 models from 11 gLM families. Results show that only 19 of 51 models outperform the 5-mer baseline, suggesting that most gLMs have limited out-of-taxon generalizability.

Critical strengths

(1) Evaluation on unseen taxa is an interesting evaluation setup that reduces the influence of shared evolutionary ancestry between training and test data.

(2) The proposed hierarchical data split is designed to ensure taxonomic separation between downstream training and test sets at different ranks. The evaluation of 51 checkpoints from 11 gLM families with a simple baseline provides useful empirical insights into the generalization capabilities of current gLMs.

Critical weaknesses
  • Limited relevance to the broader machine learning community: The paper evaluates gLMs for virus detection in wastewater, a highly specialized application. The motivation for an ICLR audience is unclear. What do we learn from this evaluation beyond which models perform best on this highly niche task of wastewater virus detection?
  • Unseen during downstream training does not mean unseen from pretraining. Even if this work ensures that the held-out taxa are absent from downstream training, these taxa may have been seen during gLM pretraining. It remains unclear whether the observed performance reflects generalizable representations or prior exposure to related sequences.
  • The benchmark focuses on a relatively simple binary classification task, which may not reflect the generalization capabilities of gLMs. Also, the evaluation largely relies on the classifier. Even if pretrained embeddings contain generalizable features, the classifier may rely on only a subset of features that are predictive during training but fail to transfer to unseen taxa. Poor performance may only reflect limitations of the downstream classifier rather than the pretrained embeddings themselves.
Questions for authors

(1) The benchmark ensures that held-out taxa are absent from downstream training, but they may have been seen during the gLM pretraining. Could the authors provide an analysis of potential overlap between the samples from pretraining data and unseen taxa?

(2) What do we learn from this evaluation beyond which models perform best on this highly niche task of wastewater virus detection and what is the impact on the AI field? The contribution seems very minor, basically just training binary classifier on top of pretrained representations of gLMs on this very specific task.

(3) How is this transferable to more general evaluation of gLMs?

(3) The evaluation relies on logistic regression probes trained on known taxa. However, even when pretrained embeddings contain generalizable features, the classifier may rely on some specific features that perform well during training on seen taxa but fail to transfer to unseen taxa. Poor performance may reflect limitations of the downstream classifier rather than the pretrained embeddings themselves. How can it be guaranteed that the limitations are indeed in pretrained representations?

(4) Conceptually, I appreciate using taxonomic hierarchy to split the data, but its advantages over simpler taxonomic holdout strategies are not directly evaluated. Could the authors compare it against simple species- or genus-level holdout splits to directly show the advantage?

(5) The authors acknowledged that different taxonomic ranks contain different held-out clades, so the observed performance differences cannot isolate the effect of evolutionary distance. However, some conclusions in Section 5 still seem to attribute performance variation to evolutionary distance. For example, the authors state that "models vary in performance depending on the clades’ distance from the training set" (Section 5, paragraph Generalization). It is better to distinguish the effects of taxonomic novelty from the task's intrinsic difficulty.

Reviewer_TWSEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

WAVEBench evaluates how well frozen genome language model (gLM) embeddings separate viral from cellular short reads in wastewater metagenomics when whole clades are withheld from the downstream probe.

The main methodological contribution is a fold-construction algorithm that, given a taxonomic hierarchy, yields a single 5-fold cross-validation partition in which each fold holds out entire clades at five ranks (class, order, family, genus, species), so a linear probe can be scored at progressively coarser levels of novelty. Clades are assigned to folds so that statistical power is balanced across ranks and folds, after discounting for correlated reads within a clade.

The benchmark is built from 18 wastewater RNA-seq samples of a single study, labelled with Kraken2. For validation, 51 models from 11 gLM families are probed and compared with a k-mer composition baseline, DeepVirFinder and restricted BLAST, with results reported as clade-balanced AUROC per held-out rank and a transfer performance check on a non-balanced sample. The taxonomic holdout applies only to probe training, not to gLM pretraining.

Performance drops from species-level to class-level holdouts, fewer than half of the models beat the k-mer baseline.

Critical strengths

All biological sequences share common descent, so random or similarity-based train-test splits do not control the degree of relatedness between held-out and training data; WAVEBench addresses this by withholding whole clades at five taxonomic ranks within a single cross-validation partition. As a result, generalisation is reported separately for each rank at which clades are withheld, from species to class, rather than as a single score.

Critical weaknesses
  1. Presentation & description of the fold-construction algorithm The fold-construction algorithm is the main contribution but is difficult to follow as written. It mixes conventional terms (rank, clade, fold, partition) with paper-specific ones (assignment unit, candidate and confined clade, stratum, sister-class rule, floor) that are never illustrated together. Some of these terms are used before they are defined (candidate in Section 3.3, the sister-class exclusion in Section 3.1, both defined only in Appendix A.1), and floor is never explicitly defined.

  2. The definition of the fold-construction algorithm The fold-construction objective is not interpretable as a whole: its final scalar is a harmonic mean over 150 ratios per subset (viral, cellular) blending six within-clade correlation values that are neither estimated nor varied in a sensitivity analysis. It is also not stated how the two subset objectives are combined. The manuscript never reports the achieved objective value of the actually performed split nor a training trajectory. So the reader can not evaluate how optimal the resulting splits are. Finally, no comparison with a random split shows that the new hold-out changes model rankings.

Questions for authors

Clarification: Fig. 3c is cited in Section 5 for the claim that one clade holds half of the viral reads at every rank from class to genus. What does the panel add beyond that statistic. How does this relate to the clade floors?

Kraken2 assigns ambiguous reads at their LCA. This results in more ambiguous reads in higher ranks than in lower ranks. Please clarify how this might confound the finding presented in Fig. 6.

Suggestions:

  1. Compare a frozen gLM probe under a random train-test split with the same probe under your taxonomy-based clade hold-out split, and report the differences in model rankings and per-rank AUROC.

  2. Please provide insights into properties (diversity, organisms covered) of pretraining sets of the identified top ranking gFMs. This is relevant since your suggested clade split is only applied to probe training but performance differences might relate to the characteristics of the pretraining data.

  3. Since the fold construction is the main contribution, please annotate the existing figures (Fig. 1b, Fig. 2) with the terms used in the text (assignment unit, candidate and confined clade, stratum, sister-class rule, floor) and illustrate the algorithm itself (feasibility step, annealing moves), e.g. as a pseudocode box or a worked example. As currently described it is difficult to follow (see the critical weakness above).

  4. The abstract mentions only the 5-mer baseline; please also state that you compare against a per-fold retrained CNN (DeepVirFinder) and an alignment baseline (BLAST).

  5. Every interval in the paper conditions on one realised fold construction and one probe seed, yet conclusions are drawn from differences as small as +0.011 [+0.001, +0.021] between Evo 2 1B and METAGENE-1 (p. 7). Running the construction with a few additional seeds and reporting how far the leaderboard ordering and the "19 of 51" count move across splits would substantially strengthen the quantitative claims.

  6. The manuscript presents the question whether gLMs can generalize to unseen taxa as the central research question, yet the results can neither support strong positive findings (the models might have actually seen the supposedly unseen taxa in pretraining) nor negative findings (the probe might not be sensitive enough to exclude that a model can generalize beyond unseen taxa). Two suggestions result from this: First, adapting the presentation of the work, acknowledging these caveats very clearly in the main text, and second focusing the work more on demonstrating the added value and robustness of the algorithm.

  7. Consider clearer axis labels, i.e. Fig. 7a “Sensitivity” instead of “Viral reads recovered”, Fig. 7b “Representative sample FPR” instead of “On the representative sample”, Fig. 8a “Precision” instead of “Calls that are viral”

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.