acceptodds
Under review as a conference paper at ICLR 2027

Human-specific Variation Complements Biological Language Models

Abstract

Modern biological language models are trained on vast sequence corpora representing organisms across the tree of life, enabling models to learn representations of biological sequences that generalize across a wide range of evolutionarily disparate organisms. However, for deepening understanding of human biology and disease, a human-centric evolutionary timescale is important. Human mutational constraint identifies genomic regions that have only recently been subject to selection within the human lineage and may have functional consequence for human-specific traits and diseases. We show that across modalities (DNA, RNA, and proteins), state-of-the-art biological language models capture deep evolutionary conservation but are weak proxies of human constraint. Furthermore, human constraint provides complementary information to the models tested, improving their ability to identify common functional variation with effects on both organismal and molecular phenotypes, despite the models' large scale pre-training. We additionally demonstrate that state-of-the-art MSA-based and sequence-to-function models can also be improved by explicit incorporation of metrics of human constraint. These results point to an unmet need to develop methods that model biological sequences not just across deep evolutionary divergence but also at a human-focused timescale.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Variant effect prediction, DNA language model, RNA language model, protein language model, biological language model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FERzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper argues that current Biological Language Models (BLMs) are good at capturing deep evolutionary conservation, but not human mutational constraint. Across DNA, RNA, and protein models, the authors find that BLMs perform better on strong-effect, rare variants than on common functional variants. The authors show that incorporating human constraint alongside BLM variant effect scores improves their prediction performance on common human variants, with statistically significant improvements. They also show similar improvements with MSA-based and sequence-to-function models.

Critical strengths
  1. The paper evaluates models across different modalities, model architectures, and training methodologies. By testing DNA, RNA, protein, MSA-based, and sequence-to-function models, the authors show that their findings generalize across different models and architectures.
  2. The paper finds an important limitation of BLMs that might be overlooked when evaluating them on strong-effect, rare variants. By comparing their performance on both rare and common functional variants, the authors show that good performance on rare disease variants does not necessarily mean it translates to common variants.
Critical weaknesses

While the authors show statistically significant improvements in prediction performance, the performance on common functional variants is relatively low, especially for eQTLs. For example, Evo2's AUROC on eQTL increases from 0.512 to 0.575 after incorporating human constraint, which is still relatively close to random guessing. This makes me question how useful these improvements would be in practice.

Questions for authors
  1. Given that prediction performance is relatively low for common functional variants, even after incorporating human constraint, could the authors expand on the practical significance of these improvements?
  2. Is the logistic regression meta-model genuinely demonstrating representation complementarity, or just feature addition?
  3. Table E.2 shows substantial fluctuations depending on whether raw or absolute likelihood ratio scores are used (e.g., for GWAS, Evo2 jumps from 0.657 to 0.671 AUROC, while GPN-Star drops). In an unsupervised or zero-shot setting where ground-truth effect direction is unknown a priori, how should a practitioner decide between signed vs. unsigned scores without introducing label leakage?
  4. The authors explicitly filtered all datasets to variants located within annotated exons to enable cross-modal comparisons with RNA and protein models. Given that many DNA language models (Evo2, NTv3) and sequence-to-function models (AlphaGenome) are primarily designed to capture non-coding regulatory syntax (promoters, enhancers, chromatin marks), does evaluating them exclusively on exonic variants artificially suppress their baseline performance?
3 comments
AuthorsOpenReview ↗

Response to reviewer FERz

We thank the reviewer for their helpful feedback and for pointing out areas where our presentation of our contributions is unclear. We also appreciate the reviewer’s assessment that the breadth of our work shows that our “findings generalize across different models and architectures” and that we identify an “important limitation of BLMs”.


Q1/W1: Given that prediction performance is relatively low for common functional variants, even after incorporating human constraint, could the authors expand on the practical significance of these improvements?

Thank you for raising this point, as it’s critical that we make the practical takeaways of our work clear. There are two practical takeaways from the improvements to model performance after incorporating human constraint:

  • First, to point out a problem: existing approaches for variant effect prediction in humans more closely capture deep conservation than human mutational constraint, and this hampers their ability to identify common functional variants. Assessing across a breadth of models establishes that this is a general trend. In this context, we focus less on the absolute performance numbers and more on the improvement across models when incorporating human constraint.
  • Second, to improve the best performing approach: for these results to be immediately helpful in practice, it’s less important for every model to reach a high absolute performance, but rather to improve the strongest approach. AlphaGenome VI, the strongest model on eQTLs, does have its performance improved from an AUROC of 0.597 to 0.620 when including human constraint, and further improved to 0.641 when including human constraint and additionally conservation.

We also put forth that while absolute performance is one practical outcome of our work, there are also higher level practical takeaways. First, low absolute performance on the eQTL and GWAS datasets relative to ClinVar demonstrates the bias of strong effect rare variant evaluation. Second, we believe the general takeaway that human-specific variation should be incorporated into models used for variant effect predictions in humans is also helpful in practice due to its potential to guide future directions in the field.

In our updated manuscript, we have edited our discussion to better summarize the practical takeaways of our work, as described above.

Q2: Is the logistic regression meta-model genuinely demonstrating representation complementarity, or just feature addition?

We agree that in many cases, the logistic regression meta-model represents feature addition (i.e. a model that performs equivalently to the greater of its individual parts), but we do see that the logistic regression meta-models incorporating the strongest base models (AlphaGenome VI, GPN-Star) show actual complementarity (i.e. a model that outperforms any of its individual parts). We refer to P1 in the shared response for a more detailed discussion.

Q3: how should a practitioner decide between signed vs. unsigned scores without introducing label leakage?

The question of whether to use signed or unsigned predictions in a zero-shot setting where it’s unknown which allele (if any) introduces a deleterious effect is both a very good question as well as a difficult one. In the absence of any prior knowledge of deleteriousness, we hypothesize that unsigned scores are more likely to capture “broad strokes” functionality of a genomic loci. However, due to the presence of functional labels in our setting, we switch to a cross-validated selection of signed or unsigned model scores for assessing the performance of model scores alone on the eQTL and GWAS datasets and obtain equivalent results. We describe this analysis in P2 of the shared response.

Q4: Given that many DNA language models and sequence-to-function models are primarily designed to capture non-coding regulatory syntax, does evaluating them exclusively on exonic variants artificially suppress their baseline performance?

We thank the reviewer for raising this point, as we have now run our analysis on intergenic regions using DNA-only models and found that our main findings of complementarity also hold for DNA-only models applied to variants in intergenic regions, which we believe strengthens the overall takeaways of our work. We refer to P3 of the shared response for more details.


We appreciate the reviewer’s time and effort in providing thoughtful feedback. We thank them for spurring us to sharpen the discussion of the practical takeaways of our work. We hope that we have been able to answer your questions and if so, we kindly ask you to reconsider your assessment of our work. If any questions or concerns remain, we are happy to engage in further discussion. Thank you!

Thank you for the detailed responses and additional analyses. Based on the additional evidence, I have revised my assessment from weak rejection to weak acceptance, subject to confirming the detailed results in the shared response.

Dear Reviewer FERz,

We're glad that our responses were able to address your questions. Thank you for your time and feedback, which have helped add additional depth to our work.

Reviewer_eQJQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether biological language models (BLMs) capture evolutionary signals across different timescales, focusing on deep evolutionary conservation and human mutational constraint. Using phyloP and Gnocchi scores alongside variant effect prediction benchmarks from ClinVar, GWAS, and GTEx, the authors systematically evaluate six representative DNA, RNA, and protein language models. The results reveal that existing BLMs correlate strongly with deep evolutionary conservation but weakly with human mutational constraint, while exhibiting limited performance on common functional variants. By incorporating human constraint through simple feature combinations, the authors demonstrate improved variant effect prediction across multiple models, including MSA-based and sequence-to-function approaches.

Critical strengths
  1. This paper evaluates a variety of biological language models across three modalities—DNA, RNA, and proteins—and further encompasses methods based on multiple sequence alignments (MSAs) and sequence-to-function prediction, thereby supporting the generalizability of the research findings across different modeling approaches.

  2. By distinguishing between deep evolutionary conservation and human-specific mutational constraints, the paper reveals significant limitations in existing models; furthermore, it demonstrates through multiple benchmark experiments that incorporating complementary evolutionary annotation data can improve performance in predicting the effects of genetic variants.

Critical weaknesses
  1. The paper relies primarily on zero-shot variant effect scoring, rather than directly analyzing the internal representations learned by the models.
  2. Cross-validation is performed by grouping variants solely based on their identifiers—rather than by gene or genomic region—meaning that neighboring variants, which share similar regional constraint scores and biological contexts, may appear in both the training and test sets.
  3. Although combining BLM scores with Gnocchi improves the performance of the original models, Gnocchi alone is sufficient to achieve comparable predictive results across several DNA and RNA models.
Questions for authors
  1. Can the authors demonstrate that information regarding human mutational constraints is truly absent from the model—rather than simply failing to be adequately captured by zero-shot variant effect scoring?
  2. Would the study’s conclusions change if the model’s intermediate-layer representations were probed, or if different types of genomic annotations were compared at a consistent genomic resolution?
  3. Do the reported performance gains remain statistically significant when employing cross-validation strategies—such as those based on individual genes, genomic regions, or the exclusion of entire chromosomes?
  4. when evaluated on independent genomic regions, what is the incremental predictive gain provided by each BLM compared to using Gnocchi alone?
  5. How sensitive are the study’s conclusions to the thresholds used to distinguish between functional and non-functional variants? Do these conclusions hold when DNA language models are applied to evaluate variants located in non-exonic regulatory regions?
5 comments
AuthorsOpenReview ↗

Response to reviewer eQJQ (2/2)

Q5: How sensitive are the study’s conclusions to the thresholds used to distinguish between functional and non-functional variants?

We assessed the robustness of our findings with regards to the thresholds for defining variant functionality by rerunning our analyses with more stringent PIP thresholds for the GWAS and eQTL dataset. We set the PIP thresholds to 0.9 and 0.01 for functional and non-functional respectively, resulting in a ~40% reduction in the number of functional variants in each dataset and ~25% and ~75% reduction in the number of non-functional variants for the GWAS and eQTL datasets, respectively:

LabelGWASGWAS (Stringent)eQTLeQTL (Stringent)
Non-functional119,96686,89667,07215,464
Functional1,69296010,9616,321

We found that evaluating on the more stringently thresholded variant datasets results in similar improvements relative to base models and relative to human constraint alone.

% AUROC improvement relative to base model (stringent PIP thresholding):

DatasetEvo2NTv3RiNALMoOrthrusESM-CSaProtAlphaGenome VIGPN-Star (V)GPN-Star (M)GPN-Star (P)
GWAS9.24 (6.27, 12.2)38.6 (33.0, 44.2)48.0 (42.2, 54.4)47.0 (41.2, 53.2)9.22 (5.27, 13.8)6.86 (3.41, 10.6)1.58 (0.65, 2.56)2.36 (0.82, 3.99)1.48 (0.39, 2.74)2.36 (1.07, 3.8)
GWAS (Stringent)7.48 (3.86, 11.0)41.3 (33.5, 49.6)51.1 (43.0, 60.1)51.7 (43.8, 59.8)10.5 (5.16, 16.0)8.68 (3.96, 13.7)1.39 (0.36, 2.42)3.09 (1.17, 5.13)0.56 (-0.54, 1.89)1.89 (0.51, 3.53)
eQTL14.7 (12.8, 16.7)13.6 (11.6, 15.5)14.5 (12.6, 16.6)14.5 (12.6, 16.6)23.7 (14.4, 34.3)23.1 (13.9, 34.3)7.54 (6.31, 8.79)8.47 (6.96, 10.1)8.53 (7.06, 9.99)9.0 (7.39, 10.5)
eQTL (Stringent)15.4 (12.4, 18.1)15.6 (12.6, 18.2)17.2 (14.1, 20.0)16.4 (13.4, 19.1)27.0 (10.7, 44.8)27.0 (10.4, 45.4)7.41 (5.89, 8.96)7.8 (5.6, 9.77)8.0 (6.04, 9.68)8.4 (6.43, 10.2)

% AUROC improvement relative to Gnocchi alone (stringent PIP thresholding):

DatasetEvo2NTv3RiNALMoOrthrusESM-CSaProtAlphaGenome VIGPN-Star (V)GPN-Star (M)GPN-Star (P)
GWAS14.8 (11.6, 18.1)0.49 (-0.76, 1.78)-0.40 (-0.98, 0.13)0.40 (-0.39, 1.11)16.9 (10.3, 24.0)19.0 (12.5, 25.9)27.8 (23.8, 32.2)23.8 (20.3, 27.5)25.1 (21.4, 29.2)23.5 (19.9, 27.4)
GWAS (Stringent)17.6 (13.7, 21.7)1.58 (-0.16, 3.37)-0.13 (-1.10, 0.82)-0.47 (-1.58, 0.49)13.8 (6.83, 20.8)14.2 (6.88, 21.9)30.7 (25.5, 36.1)24.8 (20.5, 29.3)27.8 (23.1, 32.5)25.2 (20.6, 30.1)
eQTL0.33 (-0.04, 0.67)1.06 (0.55, 1.55)1.03 (0.55, 1.51)1.78 (1.22, 2.27)-1.24 (-3.53, 1.26)-1.25 (-3.35, 0.85)8.40 (7.16, 9.61)2.35 (1.62, 3.12)2.12 (1.27, 2.95)1.49 (0.76, 2.19)
eQTL (Stringent)0.91 (0.41, 1.43)1.08 (0.45, 1.70)0.94 (0.36, 1.53)1.67 (0.98, 2.40)-3.13 (-5.74, -0.56)-3.45 (-6.26, -0.89)12.0 (10.3, 14.0)3.31 (2.31, 4.38)3.21 (2.08, 4.48)2.73 (1.70, 3.89)

We believe these results demonstrate that our core findings are robust to the selection of functionality thresholds.

Q5: Do these conclusions hold when DNA language models are applied to evaluate variants located in non-exonic regulatory regions?

We refer to P3 of the shared response for a detailed discussion and results, but thank the reviewer for raising this point that spurred us to increase the generality of our work. We find that our main results hold for DNA-only models applied to variants in non-transcribed regions.


We thank the reviewer for their insightful feedback, as their questions have led to a deeper understanding of what information of human constraint is present in biological language models. We look forward to engaging in further discussion, so please let us know if you have any additional questions or points of concern that would prevent you from increasing your score assessment. Thank you!

AuthorsOpenReview ↗

Response to reviewer eQJQ (1/2)

We thank the reviewer for the helpful feedback which has helped us both improve the rigor of our findings as well as deepen the takeaways in terms of how to think about developing future methods that incorporate human-specific variation. We're gratified that the reviewer found that our extensive evaluations "support the generalizability of the research findings" and that our work "reveals significant limitations in existing models".


Q1,2/W1: Can the authors demonstrate that information regarding human mutational constraints is truly absent from the model—rather than simply failing to be adequately captured by zero-shot variant effect scoring?

We agree that zero-shot variant effect scores do not establish that information of human constraint is absent from models and thus used a linear probe to see if information of human mutational constraint can be extracted from the models.

We defined a subset of our original dataset (for computational reasons) by sampling 10,000 genomic loci each from the high and low conservation/human constraint groups used in Table 2, for a total of 40,000 loci. For each model, we extracted final layer hidden representations of each loci and fit a logistic regression to classify highly vs poorly conserved or human-constrained sites. We used 5-fold cross-validation grouped by chromosome, with consistent fold assignments across all models, and quantified performance via AUROC:

AnnotationMethodEvo2NTv3RiNALMoOrthrusESM-CSaProtAverage
ConservationZero-shot0.920.720.590.570.860.860.70
ConservationLinear probe0.960.930.910.860.920.920.91
Human constraintZero-shot0.580.550.510.520.540.530.54
Human constraintLinear probe0.880.830.830.750.770.750.81

We see above that the linear probes are able to discriminate both conserved and human-constrained sites, though models still identify deep evolutionary conservation much more readily. This is exciting, as together with our finding that human constraint can improve variant effect predictions, these results suggest that lightweight training (e.g. supervised finetuning) could surface latent knowledge of human constraint and further improve models.

We have included these results in an additional section in the main text with a discussion of their implications for future development of human constraint-informed models. We have also adjusted wording throughout our manuscript to clarify that human constraint is not absent within the models, but rather not surfaced through standard variant effect scoring methodology.

Q3,4/W2,3: Do the reported performance gains remain statistically significant when employing cross-validation strategies? What is the incremental predictive gain provided by each BLM compared to using Gnocchi alone?

To test whether region-level leakage occurs with the current cross-validation scheme, we repeated the feature complementarity analyses using a 5-fold chromosome-based cross-validation (i.e. assigning entire chromosomes to train or test with roughly balanced fold sizes).

The results show that the two main readouts of our analyses are largely unchanged from the values reported in Tables E.4 and E.5 of our manuscript (we have only shown a subset of values for brevity here). We show results for the GWAS+eQTL dataset below as % AUROC changes with 95% confidence intervals from paired bootstraps.

Changes relative to base model:

FeaturesEvo2NTv3RiNALMoOrthrusESM-CSaProtAlphaGenome VIGPN-Star (V)GPN-Star (M)GPN-Star (P)
Conservation0.86 (-0.12, 1.8)4.8 (3.5, 6.2)5.5 (4.1, 6.9)5.7 (4.3, 7.1)5.1 (1.1, 9.3)3.1 (-0.55, 6.9)4.4 (3.5, 5.3)-0.21 (-0.52, 0.11)0.56 (0.16, 0.95)0.028 (-0.72, 0.68)
Human constraint12.0 (10.0, 14.0)12.0 (10.0, 14.0)12.0 (11.0, 14.0)13.0 (11.0, 15.0)5.1 (1.2, 8.8)4.6 (1.1, 8.1)2.5 (1.7, 3.2)7.0 (5.6, 8.3)5.7 (4.4, 6.9)6.9 (5.6, 8.1)
Both12.0 (11.0, 14.0)14.0 (12.0, 15.0)14.0 (12.0, 16.0)14.0 (12.0, 16.0)6.9 (2.7, 11.0)5.0 (1.1, 9.1)6.0 (5.0, 7.0)6.9 (5.5, 8.3)5.8 (4.5, 7.0)7.0 (5.6, 8.3)

Changes relative to Gnocchi alone:

DatasetEvo2NTv3RiNALMoOrthrusESM-CSaProtAlphaGenome VIGPN-Star (V)GPN-Star (M)GPN-Star (P)
GWAS15.0 (12.0, 18.0)0.49 (-0.76, 1.8)-0.4 (-0.98, 0.13)0.4 (-0.39, 1.1)17.0 (10.0, 24.0)19.0 (13.0, 26.0)28.0 (24.0, 32.0)24.0 (20.0, 27.0)25.0 (21.0, 29.0)24.0 (20.0, 27.0)
eQTL0.33 (-0.039, 0.67)1.1 (0.55, 1.5)1.0 (0.55, 1.5)1.8 (1.2, 2.3)-1.2 (-3.5, 1.3)-1.2 (-3.4, 0.85)8.4 (7.2, 9.6)2.4 (1.6, 3.1)2.1 (1.3, 2.9)1.5 (0.76, 2.2)

We see that improvements relative to base models remain statistically significant, as do improvements over human constraint alone for the strongest base models (AlphaGenome VI, GPN-Star models). We will update our manuscript to use values from the chromosome-based CV throughout.

Hello, we wanted to follow-up since the discussion period is ending in a few days. First, thank you for the time you've already spent engaging with our submission.

We hope to have time to respond to any additional questions or comments you may have in response to our additional analyses, so we would be grateful if you're able to review our response to your feedback and share any additional thoughts. If our responses have sufficiently addressed your concerns, we would also be thankful if you would consider updating your assessment accordingly. We understand that serving as a reviewer is a large time commitment, so thank you again for your time.

Reviewer_qwm2OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper asks which evolutionary timescale biological language models encode, contrasting 447-way mammalian phyloP against within-human constraint (Gnocchi z-scores from 76,156 gnomAD genomes, 1 kb windows). Six zero-shot BLMs across DNA, RNA and protein, plus GPN-Star and AlphaGenome VI, are scored on 324,073 exonic variants from ClinVar, fine-mapped UKBB/All of Us GWAS, and GTEx v11 eQTLs. Findings: all models correlate with phyloP (median r = 0.32) but not Gnocchi (median r = 0.059); all do well on rare ClinVar variants (median AUROC 0.892) and near chance on common GWAS/eQTL variants (0.579/0.513); adding Gnocchi via degree-2 polynomial logistic regression raises common-variant AUROC by 11.9% on average.

Critical strengths
  1. Six models with genuinely different inductive biases, scored on one harmonized variant set under one protocol. Conservation AUROC 0.578–0.865 against constraint AUROC 0.531–0.573 holds across all of them. Prior work made this point for single DNA models; the cross-modality invariance is what makes "property of the training paradigm" credible, and it is the main reason I do not recommend outright rejection.
  2. The appendix reports the analysis that undercuts the paper's own headline (Table E.5), the phyloP/Gnocchi resolution confound (Appendix D), and an AlphaGenome positive control showing the eQTL task is learnable. Code and checkpoints are specified.
Critical weaknesses
  1. The title claim is contradicted by Table E.6. For all six BLMs, model + Gnocchi is statistically indistinguishable from Gnocchi alone. Evo2 on eQTL: 0.575 [0.567, 0.583] combined vs 0.572 [0.564, 0.580] for Gnocchi by itself. Table E.5 shows four BLMs are 8–10% worse than Gnocchi alone, and adding them on top buys 0.4–1.4% AUROC; SaProt contributes −0.31%. The advertised +11.9%/+25.6% is improvement over a base at the class prior (GWAS base AUPRC 0.014–0.017), not complementary information. Real complementarity holds for AlphaGenome VI (+8.29% over Gnocchi alone on eQTL, CI excluding zero) and partly GPN-Star. Section E.9 states this correctly; the abstract, Takeaways 3 and 4, and Figure 3 generalize it to the BLMs, where it fails.
  2. Nothing separates human constraint from an allele-frequency or regional confound. Gnocchi is a function of local gnomAD variant density, and the scored variants are gnomAD variants contributing to their own window's observed count. Table 1 shows the GWAS functional and non-functional classes differ in median MAF by more than 2x (0.083 vs 0.19). None of the obvious controls are run: MAF alone, model + MAF, leave-one-variant-out Gnocchi, MAF-matched sampling, or a non-constraint 1 kb covariate (GC, mappability, local SNV density). Given the 0.4–1.4% headroom in W1, any regional feature could plausibly recover the gain. Gnocchi is also below chance on ClinVar (0.479), which argues it is not measuring per-variant constraint at the resolution the claims need.
  3. The resolution mismatch is dismissed on evidence that contradicts the dismissal. Window-aggregating both sides raises phyloP/Gnocchi Spearman from 0.029 to 0.172 and GPN-Star(V)/Gnocchi from 0.152 to 0.246, yet the text calls this not substantial and Figure E.3 is still captioned "no correlation." Separately, AlphaMissense and CADD v1.7 are cited but never run on the 47,752-variant missense subset, despite CADD already being an instance of the integration the paper advocates. Table 3 also reports the larger of signed and unsigned AUROC selected on the evaluation data.
Questions for authors
  1. Do you agree that base + Gnocchi falls within the bootstrap CI of Gnocchi alone for all six BLMs on GWAS+eQTL? If so, can the abstract and Takeaway 3 be restated in terms of Table E.5 rather than E.4, and which claims survive?
  2. What are AUROC/AUPRC for MAF alone, model + MAF, and model + a non-constraint 1 kb covariate under the same protocol?
  3. Is the scored variant included in its own Gnocchi window count? What happens under leave-one-out Gnocchi or MAF-matched sampling?
  4. Does "captures conservation but not constraint" survive with model scores and phyloP both averaged to 1 kb before correlating?
  5. AlphaMissense and CADD v1.7 on the missense subset, with and without Gnocchi?
  6. Why is Gnocchi below chance on ClinVar, and if it is anti-predictive on the gold-standard set, what justifies reading its eQTL gains as functional constraint?
5 comments
AuthorsOpenReview ↗

Response to reviewer qwm2 (1/3)

First, we thank the reviewer for the many great points of feedback that have helped significantly raise the rigor of our work. We're also glad that they found that the breadth of our analysis across models with distinct inductive biases to be insightful and that they appreciated the transparency of our appendix analyses.


Q1/W1: Do you agree that base + Gnocchi falls within the bootstrap CI of Gnocchi alone for all six BLMs on GWAS+eQTL?

We agree that the marginal bootstrap CIs of the base model and base model+Gnocchi overlap for many of the (model, dataset) pairs. However, for direct comparison between two configurations, the paired bootstrap CIs of performance differences in Table E.5 are more accurate as each bootstrap iteration evaluates both configurations on the same set of samples. A paired CI that does not overlap zero indicates a statistically significant difference.

Most gains over Gnocchi alone are modest, so the paired bootstrap is used for a more sensitive assessment: were the effects stronger, then the marginal bootstrap CIs would also not overlap (as seen for AlphaGenome VI). On the plus side, complementarity is most clear in exactly the cases where it's most useful in practice: with the strongest performing base models (AlphaGenome VI and the GPN-Star variants).

Regarding the models for which real complementarity holds and the models for which one feature dominates the combination, we refer to the discussion in P1 of the shared response.

Q2/W2: What are AUROC/AUPRC for MAF alone, model + MAF, and model + a non-constraint 1 kb covariate under the same protocol?

Below we show AUROC/AUPRC of MAF (from gnomAD genomes v4.1) on our three datasets, alongside phyloP and Gnocchi:

DatasetMAFphyloPGnocchi
ClinVar0.80/0.130.87/0.290.44/0.05
GWAS0.58/0.020.69/0.060.59/0.02
eQTL0.50/0.150.52/0.140.57/0.17

Note that ClinVar values are different from Table 3 in our main text as 31% (51,119 / 163,149) ClinVar variants don't have MAFs annotated in the Gnomad genomes dataset. We see that MAF performs well on ClinVar and has similar performance to Gnocchi on the GWAS variant set, but falls behind Gnocchi on the eQTLs, suggesting that region-level human mutational constraint brings distinct information from MAF of the variant alone.

We agree that exploring other region-level covariates can help strengthen the case for human constraint. As Gnocchi scores are based on the difference between an estimated expected number of variants and the observed number of variants in a 1kb window, local SNV density is likely quite related to Gnocchi score. Indeed, we found a strong negative correlation between Gnocchi z-scores and the number of observed variants in each window (Spearman =-0.44). We also note that Gnocchi uses GC content to perform regional adjustments to its trinucleotide mutation rates ("Collection of genomic features" in their Methods), leading us to explore mappability as a non-constraint 1kb covariate.

We averaged the UCSC Genome Browse Umap k=50 mappability track within each of Gnocchi’s 1kb windows and reran the polynomial logistic regression protocol on GWAS variants with mappability and MAF and found that:

  • Mappability brings no improvement over stronger base models, whereas weaker base models show some positive improvement that lags behind human constraint, suggesting some possible region-level confounding.
  • Similarly, MAF has predictive power that is redundant when combined with stronger base models but benefits weaker models.

Below we show the results as % AUROC change over the base model with paired bootstrap 95% CIs in parentheses:

FeaturesEvo2NTv3RiNALMoOrthrusESM-CSaProtAlphaGenome VIGPN-Star (V)GPN-Star (M)GPN-Star (P)
phyloP6.53 (3.89, 9.14)35.6 (30.0, 40.9)44.7 (38.5, 51.1)43.5 (38.2, 49.2)8.31 (4.42, 12.9)5.88 (2.56, 9.53)0.48 (0.04, 1.00)0.45 (-0.98, 1.84)0.05 (-0.76, 0.86)0.58 (-0.56, 1.80)
Gnocchi3.13 (0.95, 5.54)15.7 (11.5, 20.7)22.6 (17.7, 27.9)22.2 (17.4, 27.8)1.52 (-0.88, 3.72)1.23 (-1.04, 3.17)1.29 (0.50, 2.13)2.03 (0.52, 3.65)1.44 (0.35, 2.65)1.58 (0.50, 2.80)
Mappability (k=50)1.85 (0.71, 2.91)7.56 (4.67, 10.4)9.42 (6.05, 12.7)8.47 (5.57, 11.5)0.89 (-0.54, 2.38)0.68 (-0.41, 1.77)0.29 (-0.13, 0.64)1.05 (0.22, 1.79)0.73 (0.12, 1.28)0.61 (-0.04, 1.23)
MAF0.12 (-1.58, 1.85)16.4 (12.0, 21.4)23.9 (18.4, 29.3)23.4 (18.3, 29.0)10.8 (6.75, 15.4)8.84 (5.28, 12.9)0.31 (-0.26, 0.84)-0.29 (-1.49, 0.87)0.10 (-0.77, 0.99)-0.46 (-1.42, 0.52)

We believe these results suggest that while some region-level confounding may occur in weaker base models, the net effect of using human constraint for identifying functional common variants is still positive. We will include the above tables and a full discussion of the results in our updated appendix along with a brief discussion in the Results Section 4.3 of the main text.

AuthorsOpenReview ↗

Response to reviewer qwm2 (2/3)

Q3: Is the scored variant included in its own Gnocchi window count?

We refer to the gnomAD consortium’s original publication (PMID: 38057664), which details the Gnocchi calculation. The authors precomputed Gnocchi scores using an observed variant count based on gnomAD genomes v3.1.2 variants with AF 0.1%. Our datasets have the following percentages of variants with gnomAD genomes v4.1 AF 0.1%: ClinVar = 38.0% (62,028 / 163,149), GWAS = 4.6% (5426 / 118,725), eQTL = 0.001% (1 / 77,046). Additionally, each 1kb Gnocchi window contains an average of 170 observed variants under the author's definition above.

These counts suggest that individual variants in our dataset are unlikely to skew their assigned Gnocchi z-score.

Q4/W3: Does "captures conservation but not constraint" survive with model scores and phyloP both averaged to 1 kb before correlating?

We agree that the analysis in Appendix D shows that aggregating basepair-level phyloP and GPN-Star variant effect scores to the same 1kb resolution of Gnocchi's windows increases their correlation with Gnocchi. However, it remains true that the GPN-Star models’ correlations with evolutionary conservation are much higher (Spearman = 0.53, 0.53, 0.49 for GPN-Star V/M/P).

GPN-Star’s precomputed genome-wide scores enabled this analysis, but unfortunately due to compute limitations, we are unable to perform a similar region-level saturation mutagenesis for other models (even 1000 / 324,073 variants in our dataset would result in 6M evaluations (6*1000*1000)). However, we believe the GPN-Star results suggest that the same procedure would similarly increase other models’ correlations with human constraint by ~50%. In the revised manuscript, we have made the following updates:

  • Main text Section 4.1: state that aggregating conservation and model scores to a region level metric increases correlation with region-level human constraint, but that model scores still more strongly reflect evolutionary conservation.
  • Figure E.3 caption: explicitly describe the resolution difference and link to Appendix D.
  • Appendix D: update wording to reflect non-trivial increase in Gnocchi correlation upon region-level aggregation
  • Throughout main text: where present, update wording around models "not capturing constraint" to models "being poor proxies of constraint"
    • This is also encouraged by the linear probe analysis suggested by reviewer eQJQ below

Q5/W3: AlphaMissense and CADD v1.7 on the missense subset, with and without Gnocchi?

We applied AlphaMissense and CADD v1.7 to our missense cohort, shown below as AUROC values with percentage improvements from Gnocchi incorporation in parentheses:

DatasetAlphaGenome VIAlphaMissenseCADD v1.7
eQTL0.57 (+5.3%)0.49 (+19.2%)0.54 (+8.5%)
GWAS0.67 (+0.9%)0.66 (+0.7%)0.68 (+0.3%)

Missense cohort % AUROC change relative to Gnocchi alone (paired bootstrap CIs):

DatasetAlphaGenome VIAlphaMissenseCADD v1.7
eQTL3.93 (-0.94, 9.12)-0.17 (-1.71, 1.61)1.44 (-2.78, 5.95)
GWAS26.3 (19.5, 34.4)24.2 (17.5, 30.9)25.9 (19.2, 33.6)

We compare to AlphaGenome VI as it already incorporates AlphaMissense scores and find that AVI outperforms AlphaMissense, with AlphaMissense benefiting similarly or more so from Gnocchi inclusion. We see that improvements on eQTLs relative to Gnocchi alone are not statistically significant, consistent with our findings that human constraint shows less complementarity to model scores applied to missense mutations.

We also assess CADD v1.7 on the genome-wide cohort, where we similarly show absolute AUROC with percentage improvement from Gnocchi incorporation in parentheses:

DatasetAlphaGenome VICADD v1.7
eQTL0.59 (+3.9%)0.56 (+5.5%)
GWAS0.74 (+1.3%)0.73 (+1.1%)

and % AUROC change relative to Gnocchi alone:

DatasetAlphaGenome VICADD v1.7
eQTL8.40 (7.16, 9.61)3.79 (2.83, 4.69)
GWAS27.8 (23.8, 32.2)26.2 (22.0, 30.5)

While CADD v1.7 explored integration of human constraint metrics, the authors ultimately used Roulette local mutation rate estimates. While Roulette does use observed human variation to estimate local mutation rates, Gnocchi explicitly quantifies regional enrichment or depletion of observed variation relative to expectation. Consistent with this difference, we find above that Gnocchi scores are complementary to CADD scores genome-wide, similar to what was observed for AlphaGenome VI scores.

AuthorsOpenReview ↗

Response to reviewer qwm2 (3/3)

Q6/W2: Why is Gnocchi below chance on ClinVar,

The results in Table 3 show AUC for each score as a univariate predictor, in which Gnocchi performs slightly below random chance on ClinVar. In the polynomial logistic regression Gnocchi model we fit for comparison to the feature combination models, we see positive predictive power for ClinVar:

DatasetGnocchiPhyloP
eQTL0.570.50
GWAS0.590.69
ClinVar0.540.87

By including polynomial terms of Gnocchi (e.g. Gnocchi z-scores squared), the predictor can leverage both human mutational constraint (positive Gnocchi z-scores) as well as acceleration (negative Gnocchi z-scores). We believe this establishes that Gnocchi, and the human-specific variation it represents, is not purely anti-predictive for ClinVar.

W3: Table 3 also reports the larger of signed and unsigned AUROC selected on the evaluation data.

We agree that selecting whether to use signed or unsigned model scores for model variant effect prediction based on the full variant set represents data leakage. We updated the analysis to use cross-validated selection of signed or unsigned model scores, finding equivalent results. We refer to point P2 in the shared response for a full discussion.


We would like to thank the reviewer again for their time and effort in providing such detailed and careful feedback. We believe your review has resulted in a more complete and rigorous manuscript. We hope that our responses have answered all of your questions, and if so, we kindly ask you to consider raising your score. Please let us know if you have any additional feedback or remaining points of concern that would prevent you from increasing your assessment, as we look forward to engaging in a productive dialogue to further improve our work. Thank you again!

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global response (2/2)

P3: Effect of incorporating human constraint with DNA models on intergenic variants (eQJQ, FERz)

Thank you to the reviewers for raising this point, as investigating whether our findings hold in non-transcribed regions has helped increase the generality of our findings. To see if our findings held for DNA-only models applied to non-transcribed regions, we filtered the original ClinVar, GWAS, and eQTL datasets to autosomal SNVs located in intergenic regions. For computational reasons, we downsampled the large number of putatively non-functional variants (i.e. negatives) to 20% of the total, resulting in the following variant counts:

LabelGWASeQTLClinVar
Functional332118903
Non-functional459762472558

We excluded ClinVar from downstream analyses due to the very low number of pathogenic variants in intergenic regions.

We used this intergenic variant dataset to perform the same procedure of combining model variant effect scores and metrics of conservation and human constraint in a quadratic logistic regression model. The tables below show results analogous to those in our original Figure 3 and Table E.5. We report % AUROC change with 95% confidence intervals from a paired bootstrap shown in parentheses.

% AUROC change of combined models relative to base model on GWAS+eQTL variants in intergenic regions:

FeaturesNTv3Evo2AlphaGenome VIGPN-Star (V)GPN-Star (M)GPN-Star (P)
Conservation1.46 (0.74, 2.13)0.32 (-0.66, 1.27)3.92 (2.73, 5.11)0.54 (-0.17, 1.25)0.44 (-0.06, 1.0)0.23 (-0.18, 0.63)
Human constraint21.7 (19.8, 23.8)23.1 (21.0, 25.4)14.6 (13.1, 16.3)17.8 (16.0, 19.7)18.8 (16.8, 20.7)19.1 (17.2, 21.1)
Both21.8 (19.9, 23.9)23.3 (21.1, 25.6)15.7 (14.1, 17.5)17.8 (16.0, 19.7)18.7 (16.7, 20.6)19.0 (17.1, 21.1)

% AUROC change of combined models relative to Gnocchi on variants in intergenic regions:

DatasetEvo2NTv3AlphaGenome VIGPN-Star (V)GPN-Star (M)GPN-Star (P)
eQTL-0.09 (-0.21, 0.04)0.25 (-0.01, 0.51)4.22 (3.54, 4.88)0.95 (0.57, 1.36)0.67 (0.31, 1.02)0.71 (0.35, 1.06)
GWAS-0.93 (-5.09, 2.75)-3.26 (-7.97, 1.63)9.71 (2.78, 17.4)4.85 (-1.05, 10.9)8.35 (0.89, 16.5)6.65 (0.08, 13.7)

Overall, we find that our main findings also hold for variants in intergenic regions: adding metrics of constraint improves performance relative to base models alone, and for the best performing model (AlphaGenome VI), over human constraint alone. We believe these results further support the generality of our findings.

Additional analyses performed

In addition to the analyses described above, we also conducted 6 additional analyses to investigate feedback from individual reviewers:

  1. Comparison of MAF with conservation and human constraint. We find that MAF distinguishes strong effect variants from ClinVar, performs similarly to human constraint on GWAS variants, and falls behind human constraint on eQTL variants.
  2. Performance of mappability as an alternative region-level covariate to control for any possible region-level confounding in Gnocchi scores. While mappability can improve performance for weaker base models, it does not retain the complementarity benefits of human constraint for stronger base models.
  3. Incorporation of AlphaMissense and CADD as additional base models. Incorporating human constraint with these models similarly improves their performance on common functional variants.
  4. Linear probe analysis to classify highly conserved or human-constrained loci using models’ internal representations. We find that the linear probes from BLMs can identify highly human-constrained sites, but still below their ability to identify highly conserved sites, suggesting a promising future route for finetuning BLMs to better surface internal representations of human-specific variation.
  5. Chromosome-level cross-validation of feature combination experiments, showing that our results are robust to excluding entire chromosomes when fitting the logistic regression classifiers.
  6. Evaluation on dataset subsets with stringent functionality thresholds. We find that our results are robust to more stringent definitions of functional and non-functional variants.

We believe these additions, comprising 8 new analyses and refinements to our manuscript, have helped to significantly improve the rigor, clarity, and practical value of our work. We are committed to incorporating all feedback into the camera-ready version of the manuscript and look forward to further discussion.

AuthorsOpenReview ↗

Global response (1/2)

We genuinely appreciate all of the reviewers for their careful reading of our manuscript and detailed feedback.

We’re encouraged that the reviewers found our work to “reveal significant limitations in existing models” (eQJQ), “finds an important limitation of BLMs that might be overlooked” (FERz), and that our analysis across “six models with genuinely different inductive biases” results in the “‘property of the training paradigm’ credible” (qwm2). The reviewers’ feedback and suggestions have helped to improve the depth and rigor of our work.

Here, we provide a shared response with experiments applicable to points raised by multiple reviewers. We refer to these points as “P{number}” within the individual reviewer replies. We have also updated the PDF of our submission with textual edits shown in blue and will incorporate all of the new analyses within the camera-ready version of the manuscript.

Common points of feedback

P1: Improvements of base models with human constraint relative to human constraint alone (qwm2, eQJQ)

We appreciate the reviewers’ careful reading of our manuscript and agree that understanding the scenarios in which model scores and human constraint are truly complementary is key to our work. We first note two different definitions of complementarity: complementarity within a dataset (i.e. on a specific dataset, model scores+human constraint outperforms either alone) and complementarity across datasets (i.e. across all three datasets, model scores+human constraint outperforms either alone.

When considered across datasets, we see complementarity for all tested models, as the unified models using both a model score along with human constraint outperform either component alone. This is largely due to the model scores generally performing well on strong effect rare variants (ClinVar) and human constraint outperforming on common functional variants (GWAS and eQTL).

When considered within datasets, we see that one component often dominates the other: model scores carry the performance on ClinVar and human constraint carries the performance on GWAS and eQTL variants. However, true complementarity, where the combined model outperforms either component alone, can be seen exactly where it’s most useful in practice: with the strongest base models (AlphaGenome VI and the GPN-Star variants).

In the revised manuscript, we have clarified the distinction between complementarity across datasets and within datasets and highlighted the models for which strong complementarity holds and the ones for which one feature (model score or Gnocchi) dominates performance of the combination. We have incorporated the text from Section E.9 into Section 4.4 of the main text and restated Takeaway 3 to clarify that many models are outperformed by Gnocchi alone. We have also updated the abstract to more explicitly state that some models are outperformed by Gnocchi alone.

P2: Cross-validated selection of signed or unsigned model scores

Again, we thank the reviewers for helping raise the rigor of our work by pointing out where our methodology may not be applicable to a completely novel or unsupervised variant set. Our goal in selecting to use signed or unsigned model scores when evaluating model scores alone on GWAS and eQTL variants was to give the most optimistic comparison with evolutionary annotations. Selecting to use signed or unsigned model scores based on optimal performance would not be possible in practice due to a lack of ground truth labels.

To show how this process would look in a more general setting, we have repeated the analysis of Table 3 using chromosome-level cross-validation (5 folds), choosing between signed or unsigned scores for each (model, dataset) pair based on the training folds:

DatasetphyloPGnocchiEvo2NTv3RiNALMoOrthrusESM-CSaProtAlphaGenome VIGPN-Star (V)GPN-Star (M)GPN-Star (P)
ClinVar0.890.480.980.850.780.590.930.940.990.960.960.93
GWAS0.700.590.670.540.500.500.620.630.740.720.690.67
eQTL0.520.570.510.510.510.510.530.530.600.540.540.53

We find that performance is largely equivalent to the results shown in Table 3, indicating that selecting signed vs unsigned scores on the full dataset did not bias the takeaways. We will update Table 3 to use the cross-validated results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.