- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces WAVEBench, a benchmark for assessing downstream out-of-taxon generalization of gLMs in wastewater viral detection. Specifically, it uses taxonomic hierarchy to do the data split, withholding whole taxa at five taxonomic levels, from species to class, to evaluate viral versus cellular classification on taxa withheld from downstream training. The work evaluates 51 models from 11 gLM families. Results show that only 19 of 51 models outperform the 5-mer baseline, suggesting that most gLMs have limited out-of-taxon generalizability.
Critical strengths
(1) Evaluation on unseen taxa is an interesting evaluation setup that reduces the influence of shared evolutionary ancestry between training and test data.
(2) The proposed hierarchical data split is designed to ensure taxonomic separation between downstream training and test sets at different ranks. The evaluation of 51 checkpoints from 11 gLM families with a simple baseline provides useful empirical insights into the generalization capabilities of current gLMs.
Critical weaknesses
- Limited relevance to the broader machine learning community: The paper evaluates gLMs for virus detection in wastewater, a highly specialized application. The motivation for an ICLR audience is unclear. What do we learn from this evaluation beyond which models perform best on this highly niche task of wastewater virus detection?
- Unseen during downstream training does not mean unseen from pretraining. Even if this work ensures that the held-out taxa are absent from downstream training, these taxa may have been seen during gLM pretraining. It remains unclear whether the observed performance reflects generalizable representations or prior exposure to related sequences.
- The benchmark focuses on a relatively simple binary classification task, which may not reflect the generalization capabilities of gLMs. Also, the evaluation largely relies on the classifier. Even if pretrained embeddings contain generalizable features, the classifier may rely on only a subset of features that are predictive during training but fail to transfer to unseen taxa. Poor performance may only reflect limitations of the downstream classifier rather than the pretrained embeddings themselves.
Questions for authors
(1) The benchmark ensures that held-out taxa are absent from downstream training, but they may have been seen during the gLM pretraining. Could the authors provide an analysis of potential overlap between the samples from pretraining data and unseen taxa?
(2) What do we learn from this evaluation beyond which models perform best on this highly niche task of wastewater virus detection and what is the impact on the AI field? The contribution seems very minor, basically just training binary classifier on top of pretrained representations of gLMs on this very specific task.
(3) How is this transferable to more general evaluation of gLMs?
(3) The evaluation relies on logistic regression probes trained on known taxa. However, even when pretrained embeddings contain generalizable features, the classifier may rely on some specific features that perform well during training on seen taxa but fail to transfer to unseen taxa. Poor performance may reflect limitations of the downstream classifier rather than the pretrained embeddings themselves. How can it be guaranteed that the limitations are indeed in pretrained representations?
(4) Conceptually, I appreciate using taxonomic hierarchy to split the data, but its advantages over simpler taxonomic holdout strategies are not directly evaluated. Could the authors compare it against simple species- or genus-level holdout splits to directly show the advantage?
(5) The authors acknowledged that different taxonomic ranks contain different held-out clades, so the observed performance differences cannot isolate the effect of evolutionary distance. However, some conclusions in Section 5 still seem to attribute performance variation to evolutionary distance. For example, the authors state that "models vary in performance depending on the clades’ distance from the training set" (Section 5, paragraph Generalization). It is better to distinguish the effects of taxonomic novelty from the task's intrinsic difficulty.