- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CrossRefEval, a benchmark addressing a pressing, unresolved question about the behavior of Genomic Language Models (gLMs) on real biological tasks. Most existing downstream evaluations use task sequences obtained through mapping to a single reference genome, leaving potential performance shifts and prediction inconsistencies unexplored when the same gLM is evaluated on the same task across different references. CrossRefEval pairs the T2T-YAO assembly with GRCh38, constructs five downstream tasks, and evaluates five gLM backbones under different readout and output-matching strategies. The experiments systematically demonstrate that positional and other differences arising from reference-genome mapping can substantially alter gLM predictions, and that gains in task metrics should be assessed separately from cross-reference disagreement.
Critical strengths
- The paper carefully distinguishes among the main factors under study, including the reference genomes and their associated downstream tasks, and additionally examines differences in readout CNN design and mapping strategies.
- As a benchmark of cross-reference prediction shifts, the paper pays careful attention to the robustness of small numerical differences, consistently using multiple seeds and fine-grained analyses to validate and attribute the observed effects.
Critical weaknesses
Although I am not deeply familiar with the paper's biological background, I see several potentially issues:
- The paper's central question is how reference choice affects downstream gLM predictions, including through differences in matched positions. Yet the authors devote considerable space to prediction shifts caused by varying the readout CNN's dilation, which seems to distract from that question. Most gLMs can perform base-level and sequence-level classification effectively with simple mean pooling, a position-independent projection, or a readout from specific tokens, without adding local convolution as another variable. Although BPE-based gLMs may require an additional CNN to approximate base-level capability, this choice still seems to introduce an artificial source of variation that complicates the analysis.
- The authors train only task-specific readout heads for evaluation. However, a backbone's pretraining corpus may also affect its ability to generalize across references in downstream tasks: training exclusively on HG38 may differ from training on multiple reference genomes and species. Architectural priors and tokenization granularity may likewise influence the results. The paper does not adequately summarize these backbone-specific differences or control for their effects. A more systematic discussion of these factors could make the benchmark more useful in practice.
- The manuscript shows clear signs of AI-generated prose in its redundancy contents. These issues appear particularly pronounced in the appendices, the authors could review and refine the writing.
Questions for authors
- In my view, T2's base-level classification is better suited than sequence-level classification to revealing prediction shifts caused by reference changes and differences in mapped positions. However, its design raises concerns: positive examples account for only 0.3143% of the data. Although introducing class-imbalance methods can improve training, this class distribution may weaken the reported Disagreement difference.