- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents a benchmark dataset for genetic variant epistasis from massively parallel reporter assay (MPRA) datasets. The motivation is to evaluate current genomic foundation models' ability to predict interactions between variants on the same haplotype, which is often overlooked by the current benchmarks. The proposed benchmark contains hierarchical predictive tasks: single-variant regression, interaction regression, non-additive pair detection and personal-genomic compositional gain. The authors test multiple pretrained foundation models on these tasks using zero-shot and linear probing. The results show a major performance gap between single-variant prediction and interaction prediction, and the weak interaction signals learned by the models offer little benefit to the gene expression prediction compared to additive reconstructions.
Critical strengths
- The proposed benchmark provides useful datasets and predictive tasks to evaluate the foundation models' capability in predicting complex and biologically relevant targets, namely epistatic variant interactions. The results establish a critical performance gap between single variant prediction and interaction prediction. Overall, the benchmark raises concerns about model capability and can facilitate the evaluation and tuning of future models.
- The hierarchical predictive tasks, especially T3 and T4, progressively test the model's capability and informativeness, which provides valuable insights into the models' predictive behavior.
Critical weaknesses
- As already stated in the discussion, the high-confidence interaction datasets are rather small. Though it is mainly caused by data availability, it makes statistical comparison less confident.
Questions for authors
- How reliable is the distinction between positives and negatives in T3? Since negatives seem to be defined as pairs not significantly called non-additive, they may not sufficiently represent the negative space.
- Since the baseline performance of T4 is already low, how much of the little performance gain compared to additive reconstruction is caused by the models' real failures to model epistatic interactions , and how much is by the task difficulty itself? Would metrics other than Spearman correlation show a different pattern?
5 comments
Response to Reviewer gkEy (Part 2/3)
- Robustness across negative definitions and thresholds: Probed S2F models significantly outperform
1/distancein every stratum, including against confirmed-additive negatives (AUROC vs. for the distance probe). Re-calling labels across 20 alternative threshold settings (; ), probed S2F models also consistently outrank both gLMs and1/distancein every setting.
Revisions: Added these analyses to Appendix E.6 (Table E4), and updated Section 5.3 to state explicitly that T3 negatives are pairs not called non-additive rather than pairs confirmed to be additive.
[Question 2a — T4 Low Baseline Accuracy vs. Model Failure]: "Since the baseline performance of T4 is already low, how much of the little performance gain compared to additive reconstruction is caused by the models' real failures to model epistatic interactions, and how much is by the task difficulty itself?"
We agree that low baseline expression-prediction accuracy makes it difficult to separate model limitations from task difficulty in T4 alone. As detailed in General Response, Section 2 (Tables R1–R2), two new analyses narrow the interpretation:
- Model-predicted non-additivity (Table R1): Most models produce substantial non-additive differences across individuals (median ), yet these adjustments do not improve expression prediction over additive reconstruction (median to ).
- Stratification by cis-genetic predictability (Table R2): Even among the top tertile of genes with stronger cis-genetic signal—where Enformer and AlphaGenome achieve clear baseline accuracy ()—joint haplotype scoring adds zero gain over summing single-variant effects (all high-minus-low CIs include zero).
Because the available population data still cannot quantitatively separate model error from residual task noise, we clarify in Section 5.4 that T4 evaluates the downstream predictive utility of modeled interactions, whereas our primary conclusions about interaction accuracy rest on the controlled MPRA evaluations (T2/T3).
[Question 2b — Alternative Evaluation Metrics]: "Would metrics other than Spearman correlation show a different pattern?"
No—alternative metrics across all four tasks confirm the same conclusions (Table gkEy-2). We added Pearson correlation (T1–T2, T4), chance-normalized NDCG (T1 NDCG@100 for large-effect prioritization; T4 two-tailed NDCG across all 421 individuals), and precision@50 and recall@200 (T3; AP was already reported), using 95% locus-group bootstrap CIs for T1–T3 and gene-level bootstrap CIs for T4.
Table gkEy-2. Evaluation across additional metrics (ZS = zero-shot, LP = linear probe):
| Task | Metric | Enformer | Borzoi | AlphaGenome | NT | Evo 2 | 1/distance |
|---|---|---|---|---|---|---|---|
| T1 (emVars) | Spearman ZS / LP (paper) | 0.36 / 0.68 | 0.41 / 0.66 | 0.46 / 0.73 | −0.04 / 0.45 | −0.03 / 0.61 | – |
| Pearson ZS / LP | 0.30 / 0.63 | 0.31 / 0.59 | 0.37 / 0.67 | −0.03 / 0.40 | 0.00 / 0.56 | – | |
| NDCG@100 ZS / LP | 0.22 / 0.43 | 0.20 / 0.35 | 0.33 / 0.50 | 0.03 / 0.18 | 0.01 / 0.31 | – | |
| T2 | Spearman ZS / LP (paper) | 0.19 / 0.43 | 0.27 / 0.46 | 0.38 / 0.39 | −0.18 / 0.16 | 0.18 / 0.35 | 0.03 / −0.16 |
| Pearson ZS / LP | 0.27 / 0.52 | 0.32 / 0.51 | 0.37 / 0.58 | −0.26 / 0.36 | 0.13 / 0.48 | −0.13 / −0.21 | |
| T3 | Precision@50 ZS / LP | 0.32 / 0.46 | 0.38 / 0.48 | 0.50 / 0.58 | 0.28 / 0.20 | 0.10 / 0.42 | 0.22 / 0.21 |
| Recall@200 ZS / LP | 0.29 / 0.31 | 0.31 / 0.35 | 0.35 / 0.41 | 0.24 / 0.25 | 0.13 / 0.28 | 0.26 / 0.26 | |
| T4 | Median (full − additive) | −0.001 | 0.003 | 0.001 | 0.000 | 0.000 | – |
| % genes | 48% | 55% | 54% | 48% | 50% | – | |
| Median (full − additive) | +0.002 | +0.006 | +0.004 | 0.000 | −0.001 | – |
NDCG is chance-normalized ( random, perfect). T1 NDCG@100 ranks variants by absolute effect; T4 NDCG is two-tailed (scoring both highest and lowest expressers) over all 421 individuals.
Response to Reviewer gkEy (Part 3/3)
- T1–T2: Pearson reproduces the Spearman results: the zero-shot ranking is unchanged (), probing improves every model on both tasks, and after probing, single-variant correlation still exceeds interaction correlation for all five models (T1 vs. T2 ). On T2, all probed models and zero-shot AlphaGenome and Borzoi significantly beat
1/distance. - T3: Precision and recall confirm improved top-of-list prioritization: zero-shot AlphaGenome reaches precision@50 versus for
1/distance( positive base rate), and recall@200 versus . - T4: Neither Pearson nor two-tailed NDCG shows a meaningful gain from full-haplotype over additive predictions (median to , all 95% CIs include zero; median to ), including on genes with stronger cis-genetic predictability (see General Response, Section 2).
Revisions: Added these results to Appendix Table A3 (T1–T3) and Table F4 (T4).
Response to Reviewer gkEy (Part 1/3)
We thank the reviewer for the positive assessment and thoughtful feedback. Please also see our General Response above, which details the harmonization of Siraj T2 interaction targets to
REF/REF(all T2 results below use the corrected targets) and shared personal-genome (T4) analyses. Below, we address each point in detail.We agree that the high-confidence interaction sets are small, reflecting the current availability of published MPRA datasets that assay all four haplotypes of naturally occurring variant pairs in the same sequence context—to our knowledge, Siraj et al. (2026) and Abell et al. (2022) are the largest such datasets. We therefore designed the benchmark so that new four-haplotype datasets can be added with the same task definitions and splits as they become available.
To assess statistical confidence directly, we report 95% locus-group bootstrap confidence intervals for every T1–T3 comparison. Several key comparisons remain clearly resolved despite the sample size: on the matched set (Appendix C.2, Table C2), single-variant ρ exceeds interaction ρ with CIs excluding zero for Enformer and AlphaGenome zero-shot and for four of five probed models (e.g., probed AlphaGenome 0.71 vs. 0.39, difference 0.31 [0.17,0.46]). In addition, as detailed under Question 1 below, our T3 conclusions hold across stratified negative definitions and 20 alternative labeling thresholds.
We thank the reviewer for raising this point. The 2,021 negatives represent all candidate pairs not classified as non-additive by Siraj et al., rather than pairs confirmed to be additive. Recognizing this, we audited the calling procedure and stratified the negative space to test the robustness of our conclusions:
Table gkEy-1. Siraj T3 AUROC across alternative negative sets (each vs. the 143 positives; zero-shot / linear probe):
*Paired locus-group bootstrap 95% CI for (model − 1/distance) excludes zero. The distance probe uses one-feature logistic regression on the same folds.