- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper argues that current Biological Language Models (BLMs) are good at capturing deep evolutionary conservation, but not human mutational constraint. Across DNA, RNA, and protein models, the authors find that BLMs perform better on strong-effect, rare variants than on common functional variants. The authors show that incorporating human constraint alongside BLM variant effect scores improves their prediction performance on common human variants, with statistically significant improvements. They also show similar improvements with MSA-based and sequence-to-function models.
Critical strengths
- The paper evaluates models across different modalities, model architectures, and training methodologies. By testing DNA, RNA, protein, MSA-based, and sequence-to-function models, the authors show that their findings generalize across different models and architectures.
- The paper finds an important limitation of BLMs that might be overlooked when evaluating them on strong-effect, rare variants. By comparing their performance on both rare and common functional variants, the authors show that good performance on rare disease variants does not necessarily mean it translates to common variants.
Critical weaknesses
While the authors show statistically significant improvements in prediction performance, the performance on common functional variants is relatively low, especially for eQTLs. For example, Evo2's AUROC on eQTL increases from 0.512 to 0.575 after incorporating human constraint, which is still relatively close to random guessing. This makes me question how useful these improvements would be in practice.
Questions for authors
- Given that prediction performance is relatively low for common functional variants, even after incorporating human constraint, could the authors expand on the practical significance of these improvements?
- Is the logistic regression meta-model genuinely demonstrating representation complementarity, or just feature addition?
- Table E.2 shows substantial fluctuations depending on whether raw or absolute likelihood ratio scores are used (e.g., for GWAS, Evo2 jumps from 0.657 to 0.671 AUROC, while GPN-Star drops). In an unsupervised or zero-shot setting where ground-truth effect direction is unknown a priori, how should a practitioner decide between signed vs. unsigned scores without introducing label leakage?
- The authors explicitly filtered all datasets to variants located within annotated exons to enable cross-modal comparisons with RNA and protein models. Given that many DNA language models (Evo2, NTv3) and sequence-to-function models (AlphaGenome) are primarily designed to capture non-coding regulatory syntax (promoters, enhancers, chromatin marks), does evaluating them exclusively on exonic variants artificially suppress their baseline performance?
3 comments
Thank you for the detailed responses and additional analyses. Based on the additional evidence, I have revised my assessment from weak rejection to weak acceptance, subject to confirming the detailed results in the shared response.
Dear Reviewer FERz,
We're glad that our responses were able to address your questions. Thank you for your time and feedback, which have helped add additional depth to our work.
Response to reviewer FERz
We thank the reviewer for their helpful feedback and for pointing out areas where our presentation of our contributions is unclear. We also appreciate the reviewer’s assessment that the breadth of our work shows that our “findings generalize across different models and architectures” and that we identify an “important limitation of BLMs”.
Thank you for raising this point, as it’s critical that we make the practical takeaways of our work clear. There are two practical takeaways from the improvements to model performance after incorporating human constraint:
We also put forth that while absolute performance is one practical outcome of our work, there are also higher level practical takeaways. First, low absolute performance on the eQTL and GWAS datasets relative to ClinVar demonstrates the bias of strong effect rare variant evaluation. Second, we believe the general takeaway that human-specific variation should be incorporated into models used for variant effect predictions in humans is also helpful in practice due to its potential to guide future directions in the field.
In our updated manuscript, we have edited our discussion to better summarize the practical takeaways of our work, as described above.
We agree that in many cases, the logistic regression meta-model represents feature addition (i.e. a model that performs equivalently to the greater of its individual parts), but we do see that the logistic regression meta-models incorporating the strongest base models (AlphaGenome VI, GPN-Star) show actual complementarity (i.e. a model that outperforms any of its individual parts). We refer to P1 in the shared response for a more detailed discussion.
The question of whether to use signed or unsigned predictions in a zero-shot setting where it’s unknown which allele (if any) introduces a deleterious effect is both a very good question as well as a difficult one. In the absence of any prior knowledge of deleteriousness, we hypothesize that unsigned scores are more likely to capture “broad strokes” functionality of a genomic loci. However, due to the presence of functional labels in our setting, we switch to a cross-validated selection of signed or unsigned model scores for assessing the performance of model scores alone on the eQTL and GWAS datasets and obtain equivalent results. We describe this analysis in P2 of the shared response.
We thank the reviewer for raising this point, as we have now run our analysis on intergenic regions using DNA-only models and found that our main findings of complementarity also hold for DNA-only models applied to variants in intergenic regions, which we believe strengthens the overall takeaways of our work. We refer to P3 of the shared response for more details.
We appreciate the reviewer’s time and effort in providing thoughtful feedback. We thank them for spurring us to sharpen the discussion of the practical takeaways of our work. We hope that we have been able to answer your questions and if so, we kindly ask you to reconsider your assessment of our work. If any questions or concerns remain, we are happy to engage in further discussion. Thank you!