- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces DRAFT, a benchmark for evaluating DNA models across five capability categories: sequence priors, functional effects, context specificity, interactions, and design-oriented candidate selection. Using existing experimental datasets, it compares native model outputs and frozen representations with linear readouts against simpler baselines. The evaluations distinguish average effects from cell-specific differences and additive effects from mutational interactions, and assess candidate selection under target-matching and cell-selectivity objectives. The results suggest that strong aggregate predictive performance does not necessarily imply accurate interaction prediction or successful constrained candidate selection.
Critical strengths
The paper systematically evaluates multiple DNA models across diverse experimental datasets and includes simpler baselines alongside pretrained models. This provides a useful empirical comparison of when pretrained representations improve functional prediction and candidate selection.
Critical weaknesses
The manuscript does not sufficiently establish its novelty and added value relative to existing benchmarks such as BEND and DART-Eval. Although the context and interaction decompositions and candidate-selection evaluations provide additional diagnostic perspectives, it remains unclear whether they yield substantially new conclusions beyond established functional prediction evaluations. As presented, the contribution appears largely incremental, relying on existing datasets and standard evaluation procedures. A stronger contribution would require clearer evidence that the proposed evaluations reveal previously overlooked model limitations, materially improve model selection, or provide substantial biological or mechanistic insight. The figure 1 on the other hand directly shows the results of various testing ways without graphical illustration, making the manuscript hard to read and it is in a bad presentation form.
Questions for authors
What specific evaluation gaps in existing benchmarks, particularly BEND and DART-Eval, does DRAFT address? Which findings demonstrate that its proposed diagnostics provide additional information that materially changes model selection?
4 comments
References
[1] Muennighoff et al. MTEB: Massive Text Embedding Benchmark. EACL, 2023.
[2] Marin et al. BEND: Benchmarking DNA Language Models on Biologically Meaningful Tasks. ICLR, 2024.
[3] Liu et al. GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models. arXiv:2406.01627, 2024.
[4] Yang et al. OmniGenBench: A Modular Platform for Reproducible Genomic Foundation Models Benchmarking. arXiv:2505.14402, 2025.
[5] Patel et al. DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA. NeurIPS, 2024.
[6] Tang et al. Evaluating the representational power of pre-trained DNA language models for regulatory genomics. Genome Biology, 26:203, 2025.
[7] Trabucco et al. Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization. ICML, 2022.
[8] Shor et al. NucleoBench: A Large-Scale Benchmark of Neural Nucleic Acid Design Algorithms. bioRxiv, doi:10.1101/2025.06.20.660785, 2025.
(note to remaining reviewers: we are currently preparing answers)
Thanks you for your detailed reply and the revisions. Most of my comments have been addressed, and my questions have been clarified, so I’ve decided to raise my rating to 2.
The one thing that is still open is that: does DRAFT tell us something that would actually change which model someone picks, or guide how models are built, beyond what existing benchmarks already show? I would say this is exploration would be much more insightful and inform the others on the potential direction of the model design.
The benchmark tests a much more fine-grained understanding of function. This matters for how models rank individual sequences! The next Evo2, NTv3 or AlphaGenome needs to learn when cell context or genetic background changes outcomes. We would actually make that an explicit training objective.
Currently, the models tend to 'regress to the mean' in many cases. Those models would fare badly in our benchmark. In our reply to reviewer 8Ruu, we also added a benchmark comparison:
https://anonymous.4open.science/r/DRAFT-123/rebuttal-figures/benchmark_coverage.png
Thank you for replying. While discussions are ending now, we hope you enjoyed our work.
DRAFT fills a needed DNA design gap
Our benchmark is uniquely built to measure knowledge which is critical to DNA selection and design, which other papers mention is a key underexplored application. DART-Eval directly mentions: ”the most biologically relevant applications of regulatory sequence models include […] designing sequences with desired properties”, further stating there is a need for more evaluation. Our benchmark fills this gap among the current works.
We have a novel framework with tasks satisfying mathematical categories. This unifies tasks across benchmarks and adds familiar low-level tasks to our novel higher-level design tasks. Atomic tasks are incredibly useful for diagnosing models (like they helped fix NTv3). With high-level tasks, it is incredibly hard (in some cases impossible) to diagnose bad results.
New conclusions. We directly quantify that strong mean-effect predictions can hide weak context-specific predictions. For example, strong predictions of double mutation effects hide a nearly absent interaction knowledge, as NTv3 reaches 0.50 on double mutations but only 0.05 on the non-additive term. Additionally, using aggregate predictions (such as Tang et al.) risks missing critical weaknesses our benchmark has brought to light. In the next part of the reply, we list additional insights!
On https://anonymous.4open.science/r/DRAFT-123 we have updated easy-to-use code. Inspired by NLP benchmarks like MTEB [1], we added a mechanism to let other researchers submit their own scores. We hope this can bring additional value to the community around genomic modelling and that other benchmarks will follow this initiative.
We have made a new extensive comparison with your suggestions and numerous others [2-8] in a new 'DRAFT in relation to other benchmarks' Appendix B (page 16). We have additionally improved upon 'Related works' (page 2). We hope this makes our positioning more clear.
We additionally highlight these immediate consequences from our benchmark:
Design tasks compare the quality of models’ choices. Our selection tasks (e.g., taking multiple alleles or cell types into account) allow us to compare models by the quality of their solutions rather than a prediction metric alone, allowing clear discriminations even when shared effects completely dominate accuracy.
Accessing genomic knowledge. In the manuscript, we showed that NTv3 oddly performs better masking more bases -- and that learning a shift to the activations can do the same trick. Users are often interested in settings which differ slightly from the exact pretraining task, so model developers need to ensure their models do not become useless under such a minor difference.
Functional training greatly boosts utility, but native heads are often poor. Models trained with experimental measurements perform best across tasks. These gains are accessible through linear heads fitted to frozen embeddings, which also transfer across experiments and genes. This gives a practical direction for model development: functional supervision can contribute more than increasing model size (e.g., post-trained NTv3 100M outperforms the much larger sequence-trained model), and the benefits extend beyond the model’s original prediction heads.
Natural haplotypes. The models are considerably worse at naturally variable positions. On our panel of 512 SNP sites, no model beats uniform prediction in log loss. Even averaging eight masked queries leaves NTv3 650M post at 2.2 bits compared with the uniform baseline of 2 bits. Good performance on reference DNA alone would not reveal this weakness.
We agree that nice visualizations can promote the readers’ understanding of our paper. We have added compact figures to Table 1 to illustrate the core idea of each category. We believe this also makes the purpose of the evaluations and the paper organization more clear.
We are grateful for the review and the input! We therefore kindly ask you to consider clearly accepting. Our camera-ready paper will naturally work to include the new diagrams better in the narrative.