acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking the Capabilities Needed to Design DNA

Abstract

Genomic foundation models are increasingly used to propose DNA sequences and edits. But successful design requires capabilities that current benchmarks do not measure. Here we introduce a benchmark that assesses DNA Design Readiness Across Functional Tasks (DRAFT), which includes sequence priors, functional prediction, biological context, genetic interactions, and selection for a desired response. Across eight tasks built on five experimental datasets, we find that model choice and how model representations are accessed both affect performance. Frozen-representation outputs can match or exceed models' native outputs, and functional post-training substantially improves their predictive capabilities. These outputs can also transfer across experimental assays and genes, suggesting that models capture functional patterns beyond the data used to fit individual predictors. At the same time, strong prediction of average effects can conceal weaker prediction of context-specific effects, and accurate prediction of combined mutation effects does not imply accurate prediction of their interactions. In design tasks, model predictions can reduce the number of candidates needed to find a desired response, but experimental selection reveals substantial room for better choices, particularly for context-specific and multi-cell objectives. Overall, DRAFT shows that genomic models contain useful information for DNA design, while highlighting biological context and genetic interactions as important remaining limitations.

Primary area:
datasets and benchmarks
Keywords:
DNA language models, genome editing, AI for Science, synthetic biology, genomic foundation models, benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aCG9OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DRAFT, a benchmark for evaluating DNA models across five capability categories: sequence priors, functional effects, context specificity, interactions, and design-oriented candidate selection. Using existing experimental datasets, it compares native model outputs and frozen representations with linear readouts against simpler baselines. The evaluations distinguish average effects from cell-specific differences and additive effects from mutational interactions, and assess candidate selection under target-matching and cell-selectivity objectives. The results suggest that strong aggregate predictive performance does not necessarily imply accurate interaction prediction or successful constrained candidate selection.

Critical strengths

The paper systematically evaluates multiple DNA models across diverse experimental datasets and includes simpler baselines alongside pretrained models. This provides a useful empirical comparison of when pretrained representations improve functional prediction and candidate selection.

Critical weaknesses

The manuscript does not sufficiently establish its novelty and added value relative to existing benchmarks such as BEND and DART-Eval. Although the context and interaction decompositions and candidate-selection evaluations provide additional diagnostic perspectives, it remains unclear whether they yield substantially new conclusions beyond established functional prediction evaluations. As presented, the contribution appears largely incremental, relying on existing datasets and standard evaluation procedures. A stronger contribution would require clearer evidence that the proposed evaluations reveal previously overlooked model limitations, materially improve model selection, or provide substantial biological or mechanistic insight. The figure 1 on the other hand directly shows the results of various testing ways without graphical illustration, making the manuscript hard to read and it is in a bad presentation form.

Questions for authors

What specific evaluation gaps in existing benchmarks, particularly BEND and DART-Eval, does DRAFT address? Which findings demonstrate that its proposed diagnostics provide additional information that materially changes model selection?

4 comments

Weakness 1. The manuscript does not sufficiently establish its novelty and added value relative to existing benchmarks such as BEND and DART-Eval.

DRAFT fills a needed DNA design gap

Our benchmark is uniquely built to measure knowledge which is critical to DNA selection and design, which other papers mention is a key underexplored application. DART-Eval directly mentions: ”the most biologically relevant applications of regulatory sequence models include […] designing sequences with desired properties”, further stating there is a need for more evaluation. Our benchmark fills this gap among the current works.

We have a novel framework with tasks satisfying mathematical categories. This unifies tasks across benchmarks and adds familiar low-level tasks to our novel higher-level design tasks. Atomic tasks are incredibly useful for diagnosing models (like they helped fix NTv3). With high-level tasks, it is incredibly hard (in some cases impossible) to diagnose bad results.

New conclusions. We directly quantify that strong mean-effect predictions can hide weak context-specific predictions. For example, strong predictions of double mutation effects hide a nearly absent interaction knowledge, as NTv3 reaches 0.500.50 on double mutations but only 0.050.05 on the non-additive term. Additionally, using aggregate predictions (such as Tang et al.) risks missing critical weaknesses our benchmark has brought to light. In the next part of the reply, we list additional insights!

On https://anonymous.4open.science/r/DRAFT-123 we have updated easy-to-use code. Inspired by NLP benchmarks like MTEB [1], we added a mechanism to let other researchers submit their own scores. We hope this can bring additional value to the community around genomic modelling and that other benchmarks will follow this initiative.

We have made a new extensive comparison with your suggestions and numerous others [2-8] in a new 'DRAFT in relation to other benchmarks' Appendix B (page 16). We have additionally improved upon 'Related works' (page 2). We hope this makes our positioning more clear.

Weakness 2. A stronger contribution would require clearer evidence that the proposed evaluations reveal previously overlooked model limitations, materially improve model selection, or provide substantial biological or mechanistic insight.

We additionally highlight these immediate consequences from our benchmark:

  • Design tasks compare the quality of models’ choices. Our selection tasks (e.g., taking multiple alleles or cell types into account) allow us to compare models by the quality of their solutions rather than a prediction metric alone, allowing clear discriminations even when shared effects completely dominate accuracy.

  • Accessing genomic knowledge. In the manuscript, we showed that NTv3 oddly performs better masking more bases -- and that learning a shift to the activations can do the same trick. Users are often interested in settings which differ slightly from the exact pretraining task, so model developers need to ensure their models do not become useless under such a minor difference.

  • Functional training greatly boosts utility, but native heads are often poor. Models trained with experimental measurements perform best across tasks. These gains are accessible through linear heads fitted to frozen embeddings, which also transfer across experiments and genes. This gives a practical direction for model development: functional supervision can contribute more than increasing model size (e.g., post-trained NTv3 100M outperforms the much larger sequence-trained model), and the benefits extend beyond the model’s original prediction heads.

  • Natural haplotypes. The models are considerably worse at naturally variable positions. On our panel of 512 SNP sites, no model beats uniform prediction in log loss. Even averaging eight masked queries leaves NTv3 650M post at 2.2 bits compared with the uniform baseline of 2 bits. Good performance on reference DNA alone would not reveal this weakness.

Weakness 3. The figure 1 on the other hand directly shows the results of various testing ways without graphical illustration, making the manuscript hard to read and it is in a bad presentation form.

We agree that nice visualizations can promote the readers’ understanding of our paper. We have added compact figures to Table 1 to illustrate the core idea of each category. We believe this also makes the purpose of the evaluations and the paper organization more clear.

We are grateful for the review and the input! We therefore kindly ask you to consider clearly accepting. Our camera-ready paper will naturally work to include the new diagrams better in the narrative.

References

[1] Muennighoff et al. MTEB: Massive Text Embedding Benchmark. EACL, 2023.

[2] Marin et al. BEND: Benchmarking DNA Language Models on Biologically Meaningful Tasks. ICLR, 2024.

[3] Liu et al. GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models. arXiv:2406.01627, 2024.

[4] Yang et al. OmniGenBench: A Modular Platform for Reproducible Genomic Foundation Models Benchmarking. arXiv:2505.14402, 2025.

[5] Patel et al. DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA. NeurIPS, 2024.

[6] Tang et al. Evaluating the representational power of pre-trained DNA language models for regulatory genomics. Genome Biology, 26:203, 2025.

[7] Trabucco et al. Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization. ICML, 2022.

[8] Shor et al. NucleoBench: A Large-Scale Benchmark of Neural Nucleic Acid Design Algorithms. bioRxiv, doi:10.1101/2025.06.20.660785, 2025.

(note to remaining reviewers: we are currently preparing answers)

Thanks you for your detailed reply and the revisions. Most of my comments have been addressed, and my questions have been clarified, so I’ve decided to raise my rating to 2.

The one thing that is still open is that: does DRAFT tell us something that would actually change which model someone picks, or guide how models are built, beyond what existing benchmarks already show? I would say this is exploration would be much more insightful and inform the others on the potential direction of the model design.

The benchmark tests a much more fine-grained understanding of function. This matters for how models rank individual sequences! The next Evo2, NTv3 or AlphaGenome needs to learn when cell context or genetic background changes outcomes. We would actually make that an explicit training objective.

Currently, the models tend to 'regress to the mean' in many cases. Those models would fare badly in our benchmark. In our reply to reviewer 8Ruu, we also added a benchmark comparison:

https://anonymous.4open.science/r/DRAFT-123/rebuttal-figures/benchmark_coverage.png

Thank you for replying. While discussions are ending now, we hope you enjoyed our work.

Reviewer_ccASOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents a benchmark suite, DRAFT, for evaluating various genomic-sequence models, focusing on large sequence models. In particular, the benchmarks center around functional predictions rather than sequence likelihoods, as functional predictions are the main targeted/advertised use-case for these models. These benchmarks try to capture mutation-effect prediction, contextual changes, and interactive effects. The paper also draws several conclusions from the benchmark results. Some of these conclusions are more well-known, such as the observation that functional training leads to better functional predictions. Other findings deserve more attention from the community, for example: although these large sequence models attain accurate predictions on each cell type, they do not appreciate contextual (i.e. cell-type) differences or interactive effects between mutations.

Critical strengths

Focus on real functional data

Many current works focus their evaluations of sequence models largely on sequence likelihoods, but neglect to evaluate a models' readiness for biological discovery via functional prediction. DRAFT focuses on functional prediction from real assays (e.g. OpenSplice, GTEx, etc.). Many benchmark tasks also use the assay's replicability as a way to contextualize the performance of these models.

Explicit decomposition of additive vs interactive/non-additive effects

One of the more unique things done by this paper is to evaluate additive and non-additive effects of mutations in an explicitly decoupled way, across many models. The authors were able to show that these models often predict double-mutant combinations almost entirely identically to the summation of the single-mutant components. Similarly, these models often predict that supposed cell-type-specific predictions tend to regress to the mean. That is, this paper quantitatively shows that models' high aggregate accuracy in predicting double-mutant combinations or cell-type-specific functions is hiding the fact that they fail to predict interactive effects between mutations, or true differences between cell types.

Good high-level organization of writing

The paper is cleanly structured around five named capabilities (prior, function, context, interaction, and design), and this structure is consistent throughout the paper.

Critical weaknesses

Near-zero interaction prediction may be explainable with weak/noisy epistasis

The paper finds that models do not produce accurate predictions on interaction terms, and this is framed as a failure of the model. This observation, however, could also be explained by the assays simply not having significant epistasis between mutations. There are a few lines of evidnece that already point to the underlying data being largely additive (and not epistatic). For example, the interaction replicate agreement for COMPASS was already only 0.171 (i.e. a relatively small amount of measured interaction is reproducible). The abltion performed (accuracy barely drops when predicted interaction is zeroed) does show that the model's double-mutant accuracy does not rely heavily on interaction terms. But it doesn't really show that there was a real and predictable epistatic effect that was missed. This issue also applies to the cell-type-deviation experiment, not just the interaction-effect experiment. Macro-averaging the correlations might be a factor in this, as well (it might hide some strongly epistatic mutations which are rarer).

GTEx evaluation is more simplistic

The contrast test for GTEx picks (per gene) the tissue with the largest-measured difference (i.e. the easiest possible contrast). It would be more informative to also test harder settings, such as smaller-difference tissues. On top of this, the evaluation only ever scores differences, so the model's ability to predict absolute/calibrated expression levels is never assessed.

Functional measurements are limited to a specific set of assays

The functional assays used in the benchmark skew heavily toward splicing and reporter activity. Several important functional modalities are never really evaluated as prediction targets, such as chromatin accessibility, TF binding, chromatin modifications, 3D contacts, etc. Accessibility is used as a proxy for expression, but is never treated as a prediction target. In reality, these other functional tasks are some of the most important tasks for computational biologists interested in these models.

Main text leaves too much detail out

Several key results are stated with very limited explanation on how the result was computed, which forces the reader to dive into the appendix. For example, the interaction-removal ablation and context-specific predictions constitute a major point of the paper, but the setup for how those experiments were conducted is not clear from the main text. To create more room, I suggest cutting down the Prior section, as this section is less meaningful from a functional standpoint.

Minor points

  • "Design" is misleading as a task, because it is not a generative task; I suggest renaming to "Selection" or something similar, since the model is used to prioritize a list of candidates, rather than generate a novel sequence
  • Genome-uniform autosomal sampling (Prior) heavily biases toward intergenic regions rather than genes and regulatory sequences that design/functional tasks actual target
  • Some sentences are hard to parse and read as run-ons, such as "COMPASS tests nonadditive interactions splicing effects"
Questions for authors
  • How is uniformity evaluated in missing-base prediction (i.e. what is the baseline here) (Prior)
  • Most results use ridge regression on a frozen backbone. Linear probes only produce outputs which are linearly decodable, which is reasonable but not tested with a non-linear head. It would be helpful to have a single experiment to check if a more expressive architecture on top of the embeddings could produce better predictions (e.g. a small MLP). Otherwise, it is difficult to tell if needed functional information is actually absent from the base model, or if it is simply non-linearly encoded.
  • Several models appear to do better on COMPASS (the transfer task) than on in-distribution OpenSplice held-out genes. What are some potential likely reasons for this?
4 comments

Thank you for such a detailed review. We start by addressing your questions.

Question 1. How is uniformity evaluated in missing-base prediction (i.e. what is the baseline here).

The uniform baseline assigns probability 0.25 to each base. We score the probability assigned to the correct hidden base using its negative log-probability, giving −log⁡2(0.25)=2-\log_2(0.25)=2 bits per base.

We also have the base-composition baseline, which estimates how common each nucleotide is in separate genomic windows and uses those frequencies as its prediction probabilities.

Question 2. It would be helpful to have a single experiment to check if a more expressive architecture on top of the embeddings could produce better predictions (e.g. a small MLP).

We have added this analysis, comparing MLP vs ridge on the full main table. See results here (also located at the end of the updated pdf):

http://anonymous.4open.science/r/DRAFT-123/rebuttal-figures/nonlinear-readouts-and-aggregate-scores.pdf

We normalize task scores relative to a naive baseline (0) and NTv3 (100), allowing us to combine different metrics. With MLPs, the combined model ranking is unchanged, and the readout mainly improves splicing predictions. Ridge actually performs better in several settings, likely due to its more regularized nature. We experienced XGBoost to perform similarly to MLPs.

This suggests that more expressive readouts may need more data or stronger regularization to generalize well. Linear probing measures what users can readily obtain from frozen embeddings with a simple readout, reflecting practical downstream settings where full fine-tuning may be infeasible; see [1].

Question 3. Several models appear to do better on COMPASS (the transfer task) than on in-distribution OpenSplice held-out genes. What are some potential likely reasons for this?

We did some testing to figure this out. (Note however that this occurs only for the small CNN and native NTv3 100M; other models have a lower transfer performance.)

COMPASS focuses on short exons and averages effects across cell lines, potentially favouring predictors of shared splicing signals. These differences can make an unseen assay easier for particular models.


Weakness 1. Near-zero interaction prediction may be explainable with weak/noisy epistasis.

Interesting direction! We analyse this using the five experimental replicates in the Siraj data. Three replicates identify strong, consistently signed interactions, and the remaining two are used for evaluation. This selects 6,136 interactions across 89 regulatory elements. We evaluate prediction accuracy using either all edits or only the strong interactions:

EditsRepeat agreementAlphaGenomeNTv3 preNTv3 post
All0.280.180.000.14
Strong interactions0.640.460.050.41
Values are Spearman correlations.

This shows that there are highly reproducible non-additive effects. Models recover these interactions to different degrees, consistent with their score on background-dependent effects. So interaction prediction is a capability that models can acquire, but overall mutation-effect accuracy does not measure it adequately, and the effect is larger for experimentally consistent interactions. We now present these analyses together in Figure 5 and have moved the COMPASS interactions to Appendix.

We similarly find the cell-specific effects to have strongly reproducible signal. For the difference between HepG2 and K562, correlation across experimental replicates reach 0.594, compared with 0.398 for AlphaGenome.

Due to this investigation, we have updated the main interaction score of Figure 5A / Table 1. We now score model predictions against two independent sets of experimental measurements. We also measure how well these two sets agree, replacing the previous naive reference (“transfer across cell types”). This makes a better goal for models’ performance.

(continued)

Weakness 2. GTEx evaluation is more simplistic. [...] It would be more informative to also test harder settings, such as smaller-difference tissues. On top of this, the evaluation only ever scores differences, so the model's ability to predict absolute/calibrated expression levels is never assessed.

We actually left this out of the paper, choosing the easier setting with more obvious signal. Absolute expression is however also a strong task, while smaller tissue differences seem to contribute some more noise:

MethodSmall differencesLarge differencesExpression RMSE
AlphaGenome59.2%83.6%1.13
NTv3 650M post59.0%83.0%1.18
Tissue-mean baseline57.9%75.1%1.53
Agreement between independent measurements82.9%97.0%—

Percentages show how often the higher-expressing tissue is correctly identified. Expression RMSE uses log(1 + TPM).

The models predict expression levels better than the baseline. For small tissue differences, however, their predictions remain close to the naive baseline. Added in Appendix E.1.

Weakness 3. Functional measurements are limited to a specific set of assays. [...] In reality, these other functional tasks are some of the most important tasks for computational biologists interested in these models.

Thanks for the ideas! We intentionally focused on using few resources for more coherence and less mental load, but you are right that other modalities will be needed for sequence selection in the real world.

It is actually very easy to include/exclude data and individual tasks using our categories. While out of scope for this rebuttal, we have already added a mechanism to let other researchers submit their own models to our website.

Maintaining a living leaderboard is thus highly possible. See our user-facing code, except the actual leaderboard website, on https://anonymous.4open.science/r/DRAFT-123

We have found two very great fits with TF-binding: SNP-SELEX for functional prediction [2] and SNP Bind-n-Seq for selection [3]. There are also great datasets available for your other suggestions, e.g., functional-prediction-style accessibility. Such could be integrated at a later stage outside the paper.

Weakness 4. Main text leaves too much detail out. Several key results are stated with very limited explanation on how the result was computed, which forces the reader to dive into the appendix.

We have revised several results sections to address this, while also improving the flow of presenting results and figures!


Minor points

"Design" is misleading as a task

We agree that the distinction should be explicit, but also believe design could be done without an explicit generative model (e.g. through search/optimization). We have added the following clarification in Section 2:

Here, we evaluate design through selection from experimentally assayed libraries, rather than de novo sequence generation. For the single-edit task, the data cover nearly all permitted substitutions, allowing models to search this design space.

Genome-uniform autosomal sampling (Prior) heavily biases toward intergenic regions

Genome-wide sampling is appropriate for assessing general sequence priors, although it indeed does not concentrate on functional regions. To check how sequence prior performance varies across the genome, we evaluated 2,048 additional sites (equally distributed across distinct genomic regions):

https://anonymous.4open.science/r/DRAFT-123/rebuttal-figures/regional-sequence-priors.png

A balanced mean across regions differs from the original genome-wide score by at most 0.067 bits per base. We decide to keep the uniform sampling for the main prior task as the other categories focus more on function-related regions.

We now also emphasize a related prior-result that was previously hidden away in appendices: that foundation models are currently incredibly poor/untrained at handling natural alleles!

Some sentences are hard to parse and read as run-ons

We have improved text throughout and fixed such ‘over-edited’ wordings.

We are sincerely thankful for your effort in reviewing. We believe the paper has improved throughout, so we kindly ask if you may now clearly accept our work, or if there are remaining comments!


References

[1] Oquab et al. “DINOv2: Learning Robust Visual Features without Supervision.” Transactions on Machine Learning Research (2024), §7.1, pp. 11–12.

[2] Yan et al. “Systematic analysis of binding of transcription factors to noncoding variants.” Nature (2021).

[3] Peña-Martínez et al. “Global Evaluation of Congenital Heart Disease-Associated Non-Coding Variants.” Research Square preprint (2026).

Reviewer_ccASOpenReview ↗

Response to authors

Thank you for the thorough response and the new analyses. A few final comments:

Epistasis The replicate-split analysis on the Siraj data addresses my main concern here. It shows that reproducible non-additive effects exist, and that different models recover them to different degrees. Some models do seem to approach replicate agreement. I think that this does change the takeaway of this analysis somewhat: instead of claiming that models fail to capture interactions, it may be more accurate to say that models' abilities to predict interactions can vary widely, and this is hidden by aggregate accuracy.

GTEx The small-difference result quite informative. Models are barely above the tissue-mean baseline, while agreement between measurements is high. This supports the main point about context, and I suggest mentioning it (even briefly) in the main text rather than isolating it to the appendix.

Overall, I appreciate the effort. The benchmark is a useful contribution, and its central findings are now better supported. That said, functional coverage is still mainly splicing and reporter assays, and that remains a fundamental limitation of this work. As I was already supportive of this paper, I will keep my original score. Best of luck to the authors.

Thanks for the engagement and positive score! After some highly encouraging results, we aim to include accessibility (possibly TF-binding).

This will naturally take longer to meaningfully integrate in a narrative. We will pursue it further as we prepare for a potential camera-ready version, as we agree that broad coverage is definitely nice to have. Thank you for all your inputs. We believe the paper has matured during this time!

Accessibility & TF-binding. https://anonymous.4open.science/r/DRAFT-123/rebuttal-figures/accessibility-and-binding.pdf

Reviewer_8RuuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DRAFT, a benchmark that asks whether genomic models can predict the functional information relevant to choosing DNA sequences or edits. The authors evaluate sequence priors, functional prediction, biological context, genetic interactions and selection from experimentally measured libraries. They compare native outputs with readouts fitted to frozen representations, including NTv3 before and after functional post-training. The experiments separately evaluate contextual differences and interaction components alongside overall effects, then assess the measured outcomes of model-selected candidates. The paper also includes an OpenSplice-to-COMPASS transfer test that excludes shared genes and overlapping target sequences, and an activation-shift experiment examining access to sequence priors.

Critical strengths
  • The selection experiments evaluate model choices using held-out candidate measurements, with controls matched on predicted target strength in Siraj and an ON-only control in Atlas. All four main Atlas pipelines benefit from OFF-cell information, but leading model-based selections retain negative mean utility. The positive mean utility of the cross-repeat experimental reference shows that better choices exist in the same library.

  • The interaction analysis shows what overall scores can conceal. On COMPASS, NTv3 650M post achieves correlations of 0.500 for double-mutant effects and 0.046 for measured interactions; removing predicted interaction leaves the former at 0.493. Siraj’s paired-background analysis supports this distinction. Strict OpenSplice-to-COMPASS transfer and matched NTv3 comparisons further strengthen the empirical contribution.

Critical weaknesses
  • I find the framework’s conceptual novelty limited. DART-Eval [1] evaluates cell-specific regulatory function and variant effects; Tang et al. [2] compare frozen genomic and functionally supervised representations and examine cross-assay transfer; MTSplice [3] separates shared and tissue-specific variant effects. DRAFT’s interaction ablations and measured-selection evaluations are valuable additions, but I see its contribution as primarily empirical. Discussing these omitted antecedents would clarify that contribution.

  • The design evaluations are retrospective and use fixed assayed libraries. Atlas calibration covers all twelve cell types, leaving performance in new cell types or without similar calibration data untested. The authors acknowledge this scope and provide valuable locus and family holdouts, but prospective selection outside the libraries remains untested. Cross-assay splicing transfer concerns effect prediction rather than the selection policy, limiting the generality of the design results.

[1] Patel, A., Singhal, A., Wang, A., Pampari, A., Kasowski, M., and Kundaje, A. (2024). DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA. Advances in Neural Information Processing Systems (NeurIPS), 37, Datasets and Benchmarks Track. arXiv:2412.05430v3. https://arxiv.org/abs/2412.05430v3.

[2] Tang, Z., Somia, N., Yu, Y., and Koo, P. K. (2025). Evaluating the representational power of pre-trained DNA language models for regulatory genomics. Genome Biology, 26, 203. https://doi.org/10.1186/s13059-025-03674-8.

[3] Cheng, J., Çelik, M. H., Kundaje, A., and Gagneur, J. (2021). MTSplice predicts effects of genetic variants on tissue-specific splicing. Genome Biology, 22, 94. https://doi.org/10.1186/s13059-021-02273-7.

Questions for authors

See Critical Weaknesses.

3 comments

Thank you for this review. We here treat weaknesses point-by-point:

Weakness 1 Discussing these omitted antecedents would clarify that contribution.

Thanks for highlighting these connections! We have expanded our discussion of DART-Eval and Tang et al., and now also acknowledge MTSplice’s separation of shared and tissue-specific splicing effects.

DRAFT (uniquely) builds on these foundations to evaluate whether models can make useful DNA choices. We compare models by how well chosen edits achieve the desired effect. This can expose differences when prediction scores are largely dominated by shared effects. We also compare choices made with/without context. This can establish when context directly improves design decisions, rather than trying to infer 'usefulness' from prediction accuracy alone.

We combine novel design tasks with simpler low-level tasks that can diagnose models more directly/atomically. We have prepared a new comparison against existing benchmarks in Appendix B; also available here:

https://anonymous.4open.science/r/DRAFT-123/rebuttal-figures/benchmark_coverage.png

Weakness 2A. The design evaluations are retrospective and use fixed assayed libraries.

A benchmark for DNA design needs a ground truth. For single-edit design, we have experimental measurements for nearly all possible substitutions, thus basically covering the entire design space. Evaluating model-selected sequences against experimental measurements is also well-established in DNA optimization and ML-assisted protein evolution [1,2].

Using a learned predictor for evaluation would introduce another source of error: designs could receive high scores by exploiting the predictor’s inaccuracies without achieving the desired biological response. (Fannjiang and Listgarten examine this problem directly in [3].) Experimental measurements let us judge the models’ choices independently of such a predictor. Added this reference to Section 2.

We believe a ‘data-near’ benchmark is more likely to stay relevant for a longer time!

Weakness 2B. Atlas calibration covers all twelve cell types, leaving performance in new cell types or without similar calibration data untested.

We now test whether calibration requires measurements from the cell type being predicted. For each of six cell types, we learned how to convert DNase predictions into reporter activity using measurements from the other five. We compared this with calibration using measurements from the target cell type itself:

Calibration dataAlphaGenomeNTv3 650M post
Target cell type0.670.33
Other five cell types0.670.42

Mean utility gain over random selection among sequences intended for the same cell types; higher is better.

The takeaway is that both models give selection gains without calibration measurements from the target cell type! Since the models already have native outputs for these cell types, this specifically tests whether calibration learned in other cell types can be reused.

Weakness 2C. Cross-assay splicing transfer concerns effect prediction rather than the selection policy, limiting the generality of the design results.

We have now tested whether our target response task also works across assays. Intuitively: We train linear readouts only on OpenSplice, then ask models to find another mutation with the same effect as a given COMPASS mutation. Only afterward do we use the COMPASS measurements to check how closely their choices match the desired effect:

RandomAlphaGenomeSpliceAINTv3 650MNTv3 100MGENERATORv2k-mer
Percentile50.039.439.541.044.147.247.2

Average percentage of sequences that are closer to the target response than the model's top-ranked choice; lower is better.

These results show that the models’ first choices match the target response more closely than random alternatives, even when no COMPASS measurements were available at any point. Added to Appendix G.1.

Thus, the benefit of using the foundation models extends to selecting mutations across assays and in held-out genes!


We hope this alleviated your concerns. Do not hold back if you have other comments! Thank you for reviewing our paper.


References

[1] Trabucco et al. Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization. ICML 2022.

[2] Wu et al. Machine learning-assisted directed protein evolution with combinatorial libraries. PNAS 2019.

[3] Fannjiang and Listgarten. Autofocused oracles for model-based design. NeurIPS 2020.

Thank you for the response and the additional experiments. The results on transferring calibration across cell types and selecting mutations across assays are particularly helpful and address my main concerns. The expanded discussion also clarifies DRAFT’s contribution. I remain positive about the paper and will keep my recommendation unchanged. Best of luck with the paper!

Thank you for the engagement and your positivity!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.