- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces a novel benchmark called Reporter to evaluation pretrained DNA models. They also do an extensive evaluation with 25 pretrained models, focusing on lora-based fine-tuning performance, as opposed to ZS performance.
Critical strengths
Nicely written paper, focused on an important topic, and with a clear evaluation. I also appreciate that the authors include the following sentence: "Our benchmark measures performance under this shared protocol, which may not reflect each backbone’s best attainable performance" when describing model performance, though I would argue that more effort could be considered for the baseline specifically (see below).
Critical weaknesses
Strong baselines are crucial for a benchmark (e.g., see this paper: https://arxiv.org/abs/2411.02796). I would encourage the authors to either further motivate the strength of their baselines or to put more effort to ensuring that their baselines are strong enough.
nit: line 51: "MLM" used in the intro before it's defined
nit: "We apply REPORTER to benchmark 25 gLM backbones, including six species-specific MLM base- lines that we pretrained, alongside AlphaGenome, a leading human functional-genomics model re- cently repurposed as a pretrained backbone": This sentence ws very hard for me to parse. It's quite important as it summarizes the evaluation done in the paper, so I'd suggest improving it.
Questions for authors
Is there any concern about data contamination? i.e., have any of the pretrainined models under consideration potentially seen any of this benchmarking data, given that the benchmark was derived from data from published studies?
2 comments
Thanks for the response. I will leave my positive score as is.
Response to Reviewer dWZb
Thank you for the review. As you note, our results reflect a shared protocol rather than each backbone's best attainable performance, which makes a strong baseline all the more important.
The revision motivates the From-Scratch Baseline and tests it in three ways.
We fixed these issues in the revision—thanks!
None of the models—including the post-trained version of Nucleotide Transformer and AlphaGenome, which are the two supervised backbones we evaluated—were trained on any MPRA data.