- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
LOCALE frames genomic sequence search as a "dense retrieval" problem. By learning sequence embeddings, it ensures that locally matching sequences are ranked ahead of irrelevant ones. The method employs an InfoNCE loss function to fine-tune a shared DNA encoder, while also incorporating synthetic mutations. Compared to existing k-mer-based benchmarks, this method significantly improves retrieval quality in the presence of sequence variations.
Critical strengths
-
Interesting solution for the task: Apply the genomic foundation model to the genomic sequence search problem.
-
Better performance compared to the baseline tools in AUPRC value: The trained model achieves better performance than ESA and LLM-ED and is partially better than MetaGraph.
Critical weaknesses
-
Novelty in the model training method: The training is a direct contrastive learning method; I do not see much novelty here.
-
Generalization capability at the taxonomic level has not yet been established. Although the accessions in the training and evaluation sets are distinct from one another, the extent of species-level overlap and sequence similarity between the training corpus (including reference genomes) and the evaluation data has not been fully characterized or analyzed. Given the potential presence of closely related sequences across these datasets, a partition based solely on individual accessions is insufficient to demonstrate that the model can generalize to unseen taxa. Stratifying model performance evaluation based on the degree of similarity to training sequences would help clarify the scope of the model’s generalization capability.
Questions for authors
-
Taxonomic generalization: To what extent is there overlap (at the species level or in terms of sequence similarity) between the training data and the evaluation data (including the reference genome corpora used for training)? Is it possible to perform a stratified analysis of model performance based on the degree of similarity to the training sequences? Relying solely on an evaluation approach that ensures "no overlap in sequence accessions" is insufficient to demonstrate that a model possesses the capability to generalize to previously unseen taxa.
-
What is the novelty of this paper, except for using a genomic foundation model for this task?
-
DNABERT-2 also have one contrastive learning version DNABERT-S, should aslo benchmark it.
3 comments
Response to reviewer s3wR (continued)
W1 & Q2
We appreciate the reviewer raising this point, as it gives us an opportunity to clarify where the novelty of LOCALE lies. We agree that contrastive learning itself is well established and our contribution is not a new contrastive loss.
The central challenge we address is considerably more fundamental: how to represent local sequence alignment in a metric vector space. Local alignment is particularly difficult to embed because it is not transitive: sequences (A) and (B) may have no similarity to each other, while each aligns strongly to a different region of a third sequence (C). This makes it non-trivial to construct a global geometry in which vector distance faithfully reflects local alignability.
LOCALE's key contribution is to show that such a geometry can nevertheless be learned. Rather than approximating global edit distance or relying on k-mer overlap as a proxy, LOCALE directly organizes the embedding space around local alignability. The crucial ingredient is the biologically informed construction of training pairs: sequences are positive when they share an alignable region, including under substitutions and indels. Contrastive learning is simply the optimization mechanism through which this structure is induced.
This changes what is computationally possible. Once local alignment is represented in a dense vector space, searching for locally alignable genomic sequences becomes an approximate-nearest-neighbor search problem. This allows mature vector-search infrastructure to be applied to genomic data and provides a path toward fast, robust search over repositories such as the SRA, as well as large-scale downstream ML analyses.
We therefore view the novelty of LOCALE not as a new training loss, but as a new representation and capability. LOCALE turns local sequence alignment, a problem that has traditionally required explicit sequence comparison or proxy similarities, into vector retrieval.
Q3
Thank you for your suggestion of this additional baseline, which is an important method to compare to. We provide results on the SRA500 benchmark for the DNABERT-S model:
AUPRC (0% / 5% / 10%): 0.394 / 0.191 / 0.112, versus 0.901 / 0.796 / 0.507 for LOCALE.
We evaluated the released DNABERT-S checkpoint off the shelf, substituted into the same indexing and retrieval pipeline as our other embedding baselines.
We note that the objective of the DNABERT-S paper was species differentiation, where the model must associate non-aligned regions of the same genome together, e.g. in metagenomic binning. In contrast, LOCALE seeks to associate locally-aligned sequences together, even if they are from different species, for the purposes of large-scale search. Our setting is also outside the range DNABERT-S was designed for: its authors report poor performance on inputs shorter than 256 bp, while our queries average 136 bp.
We will add DNABERT-S to the paper as a baseline.
Conclusion
Thank you again for your thoughtful review of our work, which has helped us sharpen our framing and bring new experiments to strengthen our empirical findings. We will upload a revised manuscript incorporating the changes described above during the discussion period, and will post a note here when it is available. If you feel we have addressed your concerns, we hope you will consider raising your score. We look forward to hearing from you for further discussion.
Thanks for the detailed response, and these solve my most concerns. I raise my score to 2.
Response to reviewer s3wR
Response to Reviewer s3wR
We thank the reviewer for taking the time to review our paper. You have highlighted several important points that will be added to our paper.
W2 & Q1: Taxonomic generalization
Thank you, s3wR, for bringing up the important point of species overlap between training and eval. Indeed, 7 training species account for 426 of the 500 eval accessions and as written the paper does not show taxa generalization. Here we provide evidence for generalization across taxa using 3 analyses inspired by your review: a species-stratified eval of the paper checkpoint, a new checkpoint trained without mouse or human data, and a study stratifying performance by alignment to training data.
Species-stratified eval
We first isolate the performance of our model on the 89 SRA500 queries from species outside the training corpus (we remove all queries whose species label matches the reference datasets, and also remove those with species "human metagenome"). On those, LOCALE scores AUPRC 0.922 / 0.790 / 0.552 vs 0.901 / 0.796 / 0.507 on all queries.
Leave taxa-out retrain
Next, we retrained a checkpoint with the same setup as the paper checkpoint, but with the human and mouse genomes removed from the training corpus. Over all SRA500 queries this model scores AUPRC 0.906 / 0.799 / 0.497, versus 0.901 / 0.796 / 0.507 for the paper checkpoint trained with human and mouse data, which is within error.
The more direct test is to split the retrained model's queries by whether it saw their species in training:
The 332 queries from species the model never trained on score within error of the remaining 168 at every variation rate, so retrieval does not depend on having trained on the query's species.
Some species among the 168 (e.g. zebrafish) remain in the training corpus, so this split should be read together with the species-stratified eval above: neither removing the dominant training species nor restricting to out-of-training organisms lowers accuracy.
Stratification analysis
We binned eval queries by alignment to the reference genome data, and give results for AUPRC on SRA500 at 0%/5%/10% variation, binned by best minimap2 identity to the training reference genomes.
Queries were aligned with minimap2 to the full RefSeq genomes of the seven training species. The unaligned bin is not purely out-of-training sequence, so we give its composition:
We therefore do not read the unaligned bin as a test of unseen taxa; that evidence is the 89-query subset (0.922 / 0.790 / 0.552) and the leave-taxa-out retrain. What the table does show is the direction of the relationship with training similarity. For LOCALE, both aligned bins score below the unaligned bin at every variation rate, and the queries most similar to training data (>=0.99 identity) score lowest at 5% and 10% (0.701 and 0.367). At 5% variation the same bin is also lowest for MetaGraph, which has no training process (0.713 -> 0.550), and for ESA. We therefore find no evidence that accuracy benefits from overlap with training data.
All three analyses will be added to the paper.