- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RAGenome, a retrieval-based genomic language model pretrained on the human-referenced whole-genome alignment of 100 vertebrates. The main motivation is that MSA-based models like GPN-Star work well on evolutionary tasks, but their context is limited to around 128 nt due to the large share of positions occupied by gap tokens. RAGenome removes these gaps and uses the original alignment column indices as RoPE positions to keep the alignment information. It also samples homologs based on phylogeny to control the input length. The model has 168M parameters and is trained with span MLM in four stages. On BEND gene finding, by using frozen embeddings, it improves MCC from 0.45 for GPN-Star to 0.60. For zero-shot ClinVar variant effect prediction, the AUROC is 0.87, which is still lower than that of GPN-Star, but better than several large sequence-only models. From the training-stage and inference-length ablations, longer context seems useful for gene finding, but it does not bring much improvement for variant effect prediction.
Critical strengths
The motivation is clear, and the paper addresses a real limitation of MSA-based genomic models. Removing WGA gaps while keeping the alignment column indices as RoPE positions is a reasonable idea. By integrating phylogeny-based homolog sampling and local/cross-species attention, RAGenome can use longer context beyond around 128 nt. The paper also reports the token savings from removing gaps.
The results on the two tasks show some benefits. Gene finding improves with longer context and performs better than GPN-Star (MCC 0.60 vs 0.45). ClinVar VEP is also better than some large sequence-only models, such as NT-MS (AUROC 0.87 vs 0.57). I think these results are promising.
Critical weaknesses
My main concern is that the evaluation is too limited for the broader claims made in the paper. Gene finding and ClinVar VEP are reasonable choices for testing long-range and evolutionary modeling, and the results are promising. However, gene finding is the only downstream task used to support the claim of long-range genomic understanding. It is still unclear whether the benefit extends to other tasks that require longer context. The paper also does not report variation across runs of the gene-finding probe or uncertainty estimates for the main results. I would like to see broader evaluation and a clearer assessment of how robust the improvements are. This limited evidence is the main reason for my recommendation.
Another concern is that the benefit of retrieval is not clearly established against a comparable model trained without it. Appendix A.5 does show that longer inference context improves gene finding at a fixed checkpoint, which is useful evidence. However, setting B=0 only at inference in A.6 tests a model that was never trained without retrieval, as the authors also acknowledge. This does not tell us how a model trained without retrieval would perform under comparable conditions. The comparison with GPN-Star also differs substantially in context length and reported training cost (16,032 vs about 1,000 GPU-hours in Table 1). A model trained without retrieval, with comparable context length, model size, training data, and compute, would help establish what retrieval adds beyond long-context sequence modeling. Without this control, the contribution of retrieval to the reported gains remains unclear.
Questions for authors
-
Can you provide results on one or two additional tasks that require longer context, from BEND or a similar benchmark? This would help assess whether the improvement extends beyond gene finding. For gene finding, can you also report the mean and standard deviation across multiple runs of the downstream probe?
-
Do you have results for a model trained without retrieval, using a similar model size, the same context length and training data, and comparable compute? The B=0 experiment in A.6 removes retrieval only at inference, while the model was trained with it. A training comparison would help clarify what retrieval adds beyond long-context sequence modeling.
-
Please clarify the ClinVar evaluation setup, including the dataset version, filtering criteria, and the numbers of pathogenic and benign variants after filtering. Were all models evaluated on the same filtered set of variants? Please also clarify whether the baseline VEP scores were recomputed or taken from prior work.
3 comments
Thank you for the clarifications and additional experiments. They address my main concerns, and I am raising my rating from 2 to 3.
We are glad that the our new experiments addressed the main concerns and we thank the reviewer for raising the rating.
We thank the reviewer for the constructive feedback and we address each point below.
Q1: Additional tasks and gene finding variation
For additional tasks, we refer the reviewer to our response to Reviewer JQQB and to the context length ablation in our response to Reviewer Mygv.
For gene finding, we repeated the downstream probe training with 5 different seeds and report the mean and standard deviation of the test MCC for the three models below:
The variance across seeds is small compared to the differences between models, confirming the findings of the paper. We thank the reviewer for this suggestion and will report the mean and standard deviation in the revised manuscript.
Q2: Ablation without retrieval
We trained the same RAGenome architecture (168M parameters) without retrieval, using the same training data, context lengths and pretraining stages. Due to the limited time of the rebuttal period, we trained the first two pretraining stages and evaluated the model at the end of each.
In the attached figure (https://anonymous.4open.science/r/iclr27_rebuttal-6127/no_retrieval.png), we observe that retrieval improves gene finding by around 0.2 MCC and ClinVar AUROC by around 0.3. Without retrieval, ClinVar performance is similar to sequence-only models such as NT-MS (0.57, Table 1). For gene finding, the results are consistent with Table 1, where sequence-only models reach strong performance only with far more parameters and compute (NT-MS, Evo2).
This ablation further illustrates that retrieval-based models provide a more efficient alternative to scaling sequence-only gLMs.
Q3: ClinVar evaluation setup
We use the ClinVar benchmark released with GPN-Star (Ye et al., 2026), available on Hugging Face in https://huggingface.co/datasets/songlab/clinvar_vs_benign. It contains 50,164 missense single-nucleotide variants labeled as pathogenic (22,254) or benign (27,910).
We refer the reviewer to Ye et al. (2026) for the construction details, which are based on ClinVar release 20220924. We used the full dataset without any additional filtering, and all models are evaluated on exactly the same set of variants. Each variant is scored zero-shot by the log-likelihood ratio between the alternate and the reference allele, and we report AUROC. Finally, all scores reported in Table 1 were recomputed by us with the described pipeline.
We hope that these clarifications, together with the new experiments, address the reviewer’s concerns.