acceptodds
Under review as a conference paper at ICLR 2027

RAGenome: Scaling Retrieval-Based Genomic Language Models to Long Contexts

Abstract

The genome holds the blueprint that governs the biological properties of the cell. Consequently, advancing our knowledge of genomic function is crucial both for a broader understanding of biology and for continued biomedical advances. The success of large language models on natural language and protein sequences has motivated similar efforts on genomic data. However, standard genomic language models (gLMs) often require extremely large computational resources and still fall behind traditional methods on some downstream tasks. Recently, MSA-based pretraining has been proposed as an efficient alternative, but existing models are limited to short input contexts, restricting their use to short-range tasks, such as variant effect prediction. In this work, we present RAGenome, the first retrieval-based gLM that scales pretraining to longer contexts (100 longer than existing MSA-based gLMs), allowing it to capture both across-species evolutionary relationships and within-species longer-range interactions. Trained on whole-genome alignments from 100 vertebrates, RAGenome substantially improves the long-range capabilities of MSA-based gLMs, raising gene finding performance from 0.45 to 0.60, while remaining competitive on purely evolutionary-based tasks like prioritizing pathogenic variants. RAGenome provides competitive gLM performance at a fraction of the training cost, unifying evolutionary modeling and long-range capabilities within a single, flexible, scalable framework.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Genomic Language Models, Genomics, Evolutionary Biology, Self-Supervised Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_hAZ1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RAGenome, a retrieval-based genomic language model pretrained on the human-referenced whole-genome alignment of 100 vertebrates. The main motivation is that MSA-based models like GPN-Star work well on evolutionary tasks, but their context is limited to around 128 nt due to the large share of positions occupied by gap tokens. RAGenome removes these gaps and uses the original alignment column indices as RoPE positions to keep the alignment information. It also samples homologs based on phylogeny to control the input length. The model has 168M parameters and is trained with span MLM in four stages. On BEND gene finding, by using frozen embeddings, it improves MCC from 0.45 for GPN-Star to 0.60. For zero-shot ClinVar variant effect prediction, the AUROC is 0.87, which is still lower than that of GPN-Star, but better than several large sequence-only models. From the training-stage and inference-length ablations, longer context seems useful for gene finding, but it does not bring much improvement for variant effect prediction.

Critical strengths

The motivation is clear, and the paper addresses a real limitation of MSA-based genomic models. Removing WGA gaps while keeping the alignment column indices as RoPE positions is a reasonable idea. By integrating phylogeny-based homolog sampling and local/cross-species attention, RAGenome can use longer context beyond around 128 nt. The paper also reports the token savings from removing gaps.

The results on the two tasks show some benefits. Gene finding improves with longer context and performs better than GPN-Star (MCC 0.60 vs 0.45). ClinVar VEP is also better than some large sequence-only models, such as NT-MS (AUROC 0.87 vs 0.57). I think these results are promising.

Critical weaknesses

My main concern is that the evaluation is too limited for the broader claims made in the paper. Gene finding and ClinVar VEP are reasonable choices for testing long-range and evolutionary modeling, and the results are promising. However, gene finding is the only downstream task used to support the claim of long-range genomic understanding. It is still unclear whether the benefit extends to other tasks that require longer context. The paper also does not report variation across runs of the gene-finding probe or uncertainty estimates for the main results. I would like to see broader evaluation and a clearer assessment of how robust the improvements are. This limited evidence is the main reason for my recommendation.

Another concern is that the benefit of retrieval is not clearly established against a comparable model trained without it. Appendix A.5 does show that longer inference context improves gene finding at a fixed checkpoint, which is useful evidence. However, setting B=0 only at inference in A.6 tests a model that was never trained without retrieval, as the authors also acknowledge. This does not tell us how a model trained without retrieval would perform under comparable conditions. The comparison with GPN-Star also differs substantially in context length and reported training cost (16,032 vs about 1,000 GPU-hours in Table 1). A model trained without retrieval, with comparable context length, model size, training data, and compute, would help establish what retrieval adds beyond long-context sequence modeling. Without this control, the contribution of retrieval to the reported gains remains unclear.

Questions for authors
  1. Can you provide results on one or two additional tasks that require longer context, from BEND or a similar benchmark? This would help assess whether the improvement extends beyond gene finding. For gene finding, can you also report the mean and standard deviation across multiple runs of the downstream probe?

  2. Do you have results for a model trained without retrieval, using a similar model size, the same context length and training data, and comparable compute? The B=0 experiment in A.6 removes retrieval only at inference, while the model was trained with it. A training comparison would help clarify what retrieval adds beyond long-context sequence modeling.

  3. Please clarify the ClinVar evaluation setup, including the dataset version, filtering criteria, and the numbers of pathogenic and benign variants after filtering. Were all models evaluated on the same filtered set of variants? Please also clarify whether the baseline VEP scores were recomputed or taken from prior work.

3 comments

We thank the reviewer for the constructive feedback and we address each point below.

Q1: Additional tasks and gene finding variation

For additional tasks, we refer the reviewer to our response to Reviewer JQQB and to the context length ablation in our response to Reviewer Mygv.

For gene finding, we repeated the downstream probe training with 5 different seeds and report the mean and standard deviation of the test MCC for the three models below:

ModelReported in manuscriptMean ± std over 5 runs
GPN-Star (V)0.450.444 ± 0.018
NT-MS0.660.655 ± 0.002
RAGenome0.600.597 ± 0.002

The variance across seeds is small compared to the differences between models, confirming the findings of the paper. We thank the reviewer for this suggestion and will report the mean and standard deviation in the revised manuscript.

Q2: Ablation without retrieval

We trained the same RAGenome architecture (168M parameters) without retrieval, using the same training data, context lengths and pretraining stages. Due to the limited time of the rebuttal period, we trained the first two pretraining stages and evaluated the model at the end of each.

In the attached figure (https://anonymous.4open.science/r/iclr27_rebuttal-6127/no_retrieval.png), we observe that retrieval improves gene finding by around 0.2 MCC and ClinVar AUROC by around 0.3. Without retrieval, ClinVar performance is similar to sequence-only models such as NT-MS (0.57, Table 1). For gene finding, the results are consistent with Table 1, where sequence-only models reach strong performance only with far more parameters and compute (NT-MS, Evo2).

This ablation further illustrates that retrieval-based models provide a more efficient alternative to scaling sequence-only gLMs.

Q3: ClinVar evaluation setup

We use the ClinVar benchmark released with GPN-Star (Ye et al., 2026), available on Hugging Face in https://huggingface.co/datasets/songlab/clinvar_vs_benign. It contains 50,164 missense single-nucleotide variants labeled as pathogenic (22,254) or benign (27,910).

We refer the reviewer to Ye et al. (2026) for the construction details, which are based on ClinVar release 20220924. We used the full dataset without any additional filtering, and all models are evaluated on exactly the same set of variants. Each variant is scored zero-shot by the log-likelihood ratio between the alternate and the reference allele, and we report AUROC. Finally, all scores reported in Table 1 were recomputed by us with the described pipeline.

We hope that these clarifications, together with the new experiments, address the reviewer’s concerns.

Thank you for the clarifications and additional experiments. They address my main concerns, and I am raising my rating from 2 to 3.

We are glad that the our new experiments addressed the main concerns and we thank the reviewer for raising the rating.

Reviewer_JQQBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RAGenome is a retrieval-based genomic language model that retrieves homologous segments for human query sequences from a 100-vertebrate whole-genome alignment. Through gap removal, sliding-window attention with sparse cross-species cross-attention layers, and staged training, it extends MSA model context from 128 nt to 13,312 nt, improving gene-recognition MCC from 0.45 to 0.60 while remaining competitive on pathogenic-variant ranking at a fraction of the training cost of large gLMs.

Critical strengths
  • Observing that 73% of tokens in a 100-vertebrate whole-genome alignment are gaps, the authors remove gaps, use alignment column indices as RoPE positions to preserve cross-species correspondence, concatenate species with taxonomy embeddings, and cap retrieval at a fixed budget sampled across clades, so token count no longer scales with length × species.
  • The 168M-parameter model combines 12 within-species sliding-window attention layers with 3 cross-species cross-attention layers (human attending to other species) and is trained in four stages that jointly expand context to 13,312 nt and genome coverage.
Critical weaknesses
  • Only two tasks are evaluated: BEND gene finding and ClinVar. Other long-range tasks are missing, such as BEND enhancer annotation and histone modification, splicing variants, and long-range eQTL.
Questions for authors
  • Why are species in the same clade as humans excluded (leaving 89 species)? What would happen if primates were included, particularly for the pathogenic-variant task?
  • The paper states that the same positions are masked across species within a clade, yet only human tokens are predicted. Why are tokens from other species masked as well? Could the authors explain this mechanism more clearly?
1 comment

We thank the reviewer for the detailed review and we hope the following responses address each point thoroughly.

W1: Additional evaluation tasks

We evaluated RAGenome, GPN-Star and NT-MS on additional tasks from BEND. We followed the same protocol used for gene finding in the paper (train a CNN model on frozen embeddings) as also proposed in the BEND paper.

Due to the limited time of the rebuttal period, we used stratified subsets for some tasks (10% for histone modification and CpG methylation, 5% for chromatin accessibility). The subsets were sampled within each split and label, so that split sizes and label proportions are preserved and all models are evaluated on the same subsets. Our NT-MS results agree with those reported in BEND on the full datasets (in parentheses), except for chromatin accessibility, where the 5% subset costs about 0.04 AUROC.

ModelEnhancer annotationChromatin accessibility**Histone modification*CpG methylation*
GPN-Star (V)0.080.770.790.93
NT-MS0.07 (0.06)0.75 (0.79)0.77 (0.78)0.92 (0.92)
RAGenome0.080.800.810.95

* stratified 10% subset; ** stratified 5% subset.

As also noted in the BEND paper, all models perform poorly on enhancer annotation, probably due to the sparsity of the signal and the small size of the dataset. On the remaining BEND tasks, RAGenome outperforms both models, showing that it is competitive beyond gene finding and variant effect prediction.

We also evaluated the three models on the causal eQTL task of the Human Genomics Long-Range Benchmark (LRB) (https://huggingface.co/datasets/InstaDeepAI/genomics-long-range-benchmark), following the protocol of training an MLP head on frozen embeddings:

ModelCausal eQTL (AUROC)
GPN-Star (V)0.716 ± 0.003
NT-MS0.732 ± 0.001
RAGenome0.725 ± 0.001
Mean ± std over 5 seeds.

We observe that RAGenome achieves better performance than GPN-Star and slightly worse than NT-MS (which has 15× more parameters).

We note, however, that GPN-Star, despite its 128 bp context, remains competitive on all of these tasks, which suggests that these tasks do not fully test long-range capabilities, unlike gene finding. Together with gene finding and variant effect prediction, these results indicate that RAGenome successfully unifies explicit evolutionary modeling with long-range genomic understanding within a single, efficient architecture.

Q1: Why are species in the same clade as humans excluded?

We exclude the clade of humans from retrieval so that more of the retrieval budget is spent on distant species, where the conservation signal is more informative. However, our framework supports including the human clade, provided that these species are masked at exactly the same positions as the human sequence. This is important, because otherwise the masked human nucleotides could be trivially ‘unmasked’ from the nearly identical primate sequences.

For the pathogenic-variant task, we expect the inclusion of primates to add little evolutionary information. Primates have diverged only recently, so few substitutions are expected even at unconstrained sites, and the signal distinguishing pathogenic from benign variants comes mainly from conservation across more distant species.

Q2: Why are tokens from other species masked as well?

The reviewer is right that only the masked human tokens are predicted. We mask tokens of the retrieved species to encourage the model to combine evolutionary information across all clades and the surrounding sequence context, instead of relying on specific species. It acts as a form of structured input corruption that makes the MLM task more challenging, and it makes the model robust to the missing or unaligned positions that are common in WGAs. We will revise the manuscript to clarify this mechanism.

We hope these clarifications and the new experiments presented above address the reviewer’s concerns.

Reviewer_MygvOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tries to extend MSA-based genomic language models to long contexts with retrieval-based paradigm.

  1. removes gap tokens from the aligned species and keeps their alignment positions through RoPE.
  2. limits the number of retrieved tokens by sampling species across clades.
  3. uses local attention within each species and cross-attention from human to the other species.
  4. trained with a curriculum that gradually increases context length and genome coverage.

On BEND gene finding it improves over existing MSA-based models, and on ClinVar it stays close to them.

Critical strengths

Overall, the paper proposed a solution to a real limitation of MSA-based models with a simple idea.

  1. Removing gap tokens is a practical way to make alignment-based models scale.
  2. The evolutionary signal is not lost as context grows. Variant effect performance stays stable across training stages and remains above sequence-only models, so the long-context design keeps what MSA models are good at.

These two points make the work a reasonable step toward long-context alignment-based models.

Critical weaknesses
  1. The authors claims that context extension improves both on gene finding and VEP tasks, but from table 3, multiple variables changed simultaneously, like while context length increased, the Genome coverage or retrieval budget also increased.
  2. The authors claims that model’s "within-species longer-range interactions", but regarding table 2 and eq. 2, 256 per layer for 12 layers, the maximum length in species might be only around 3072 nucleotides.
Questions for authors
  1. Could you please add up some ablations only on context-length?
  2. Would the maximum length in species around 3072 nucleotides be bit limited?
5 comments

We thank the reviewer for the constructive feedback and we address each point separately below.

Q1: Ablation only on context length

The reviewer is right that in some of the pretraining stages we change several factors at once (Table 3). From stage 2 to stage 3 we increase both the context length L (1,024 → 4,096) and the retrieval budget B (24k → 80k). To disentangle these factors, we trained a new model starting from the last checkpoint of stage-2 with the same exact settings as stage 3, but keeping the context length at 1,024. The table below summarizes the ablation:

Stage 3 (paper)Stage 3 ablation (same context as stage 2)
Context length L4,0961,024
Genome coveragetop 40%top 40%
Retrieval budget B80,00080,000
Effective batch size512512

As shown in the attached figure (https://anonymous.4open.science/r/iclr27_rebuttal-6127/context_ablation.png), after 2,000 GPU hours (red line), gene finding performance does not improve beyond stage 2 when the context is not increased (0.480 MCC while in stage 2 it is 0.476).

Therefore, the longer context is the main driver of gene-finding improvement, not the larger retrieval budget or the additional training. As expected, variant effect prediction is unaffected by context length, further verifying that the retrieval mechanism preserves its evolutionary signal regardless of the context size.

Q2: Maximum intra-species length

As presented in Table 2, the local attention window (ww) is 512, i.e. each layer attends to ±256 positions. For 12 local attention layers, the intra-species receptive field of each query position becomes 512×12=6,144512 \times 12=6{,}144 tokens (3,072 positions on each side). At the same time, inter-species cross-attention layers are global (Figure 3): every query position attends to the retrieved homologous sequences across the entire input. So, interactions longer than that intra-species receptive field can be captured through evolutionary signals from the retrieved species.

We agree with the reviewer that the intra-species receptive field is an important parameter. When scaling RAGenome to even longer contexts, the local window size and the number of layers should be increased accordingly to keep a large enough receptive field.

We will clarify the effective receptive field in the revised manuscript.

Thanks for authors’ detailed replies. They address my previous concerns, but I still have some questions:

  1. How did the retrieved sequence get masked? Are the same alignment columns masked as in the human sequence, and at what rate?
  2. Could you please introduce bit more about PhastCons score in this paper so that it can be more comprehensive? Also adding up baselines of PhastCons and PhyloP might make it more straightforward to demonstrate its advantages compared to conservation signal that the alignment already contains.
  3. While the work claims less compute cost compared to large-scale gLM, compared to GPN-Star, it seems to require more GPU hour while performing worse on VEP. What is the insight here?

We thank the reviewer for the positive feedback and for engaging in discussion. We are very eager to improve our paper and are glad that our replies addressed the previous concerns. We address the new questions separately below:

Q1: Masking of the retrieved sequences

As presented in Section 4.4, for the human sequence we mask short contiguous spans of nucleotides (3 to 6 nucleotides) together, instead of individual tokens, covering on average 15% of the positions in each training sample. For the retrieved sequences, we use the same masking rate (15%), but we mask individual alignment columns, with an independent mask for each clade. As a result, all species of the same clade share exactly the same mask, while different clades are masked at different positions.

We apply this masking procedure to encourage the model to predict the masked human tokens using the aligned homologous nucleotides of the clades in which these positions remain unmasked. Our masking pipeline is highly influenced by the one in GPN-Star (Ye et al., 2026), but slightly modified to account for our retrieval mechanism. We will include an additional section in the Appendix that describes the whole masking procedure in more detail.

Q2: Conservation baselines

In this work, we use PhastCons (Siepel et al., 2005) only as a selection method for the conserved regions to use in the different pretraining stages (Appendix A.3). We will expand Appendix A.3 to describe PhastCons and how these scores are used in more detail.

Following the reviewer's suggestion, we evaluated PhastCons and PhyloP on exactly the same ClinVar variants, using the scores computed from the same 100-vertebrate alignment that RAGenome retrieves from:

MethodClinVar AUROC
PhastCons0.77
PhyloP0.85
GPN-Star (V)0.92
RAGenome0.87

We observe that RAGenome outperforms both conservation scores computed from the same alignment, indicating that our retrieval-based model captures richer information than the conservation signal that the alignment already contains. We will add these baselines to Table 1, and we thank the reviewer for the suggestion.

Q3: Compute compared with GPN-Star

First, we want to point out that RAGenome reaches almost its best performance in variant effect prediction already from the early stages of pretraining (Figure 4). For example, after training for 864 GPU hours (less than the ~1,000 GPU hours of GPN-Star), RAGenome already achieves ~0.87 AUROC on ClinVar. The rest of the GPU hours of the model training are devoted to scaling the input context with two goals: 1) improving the performance on long-range tasks, such as gene finding and 2) keeping the performance at a high level on conservation-based tasks, which depend mostly on the WGA and not on the long input context.

As for the gap of 0.05 AUROC with GPN-Star, we assume that it is connected to how often each model attends to the WGA. GPN-Star alternates between attention along the query sequence and attention across species in every layer, so the representation of each position is refined with the aligned sequences throughout the network. However, the cost of this design grows increasingly with both the sequence length and the number of species in every layer. In RAGenome, evolutionary information from the retrieved sequences is used only through the cross-attention layers in the middle part of the network (Figure 3), while the remaining layers use local attention within the human sequence. This design allows scaling the input to 13,312 nucleotides, at the cost of a less fine-grained use of the alignment, which likely explains the remaining gap in variant effect prediction. Since our retrieval mechanism does not depend on the number of cross-attention layers, we plan to explore adding more of them to test whether this closes the gap.

We thank the reviewer for the useful feedback and we will clarify all these points in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.