acceptodds
Withdrawn from ICLR 2027

CrossRefEval: Paired Evaluation of Genomic Models Across Reference Assemblies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Genomic models are usually evaluated on a single reference genome, leaving open how the same predictor responds when corresponding regions are represented by another assembly. This comparison requires explicit correspondence: for position-resolved prediction, corresponding bases can occur at different output indices, so the rule used to pair predictions can itself affect the measured comparison. We introduce CrossRefEval, a paired framework that holds the fitted predictor and decision rule fixed across reference representations and evaluates task performance, cross-reference consistency, and paired correctness at corresponding outputs. We instantiate CrossRefEval on the maternal haplotype assembly of T2T-YAO v2.0 and GRCh38, constructing 254,366 paired windows across five genomic prediction tasks. We evaluate five frozen DNA encoders with readouts trained separately on YAO and GRCh38. Holding prediction arrays fixed, changing only the output correspondence alters the comparison between readout variants: across ten encoder–training-reference combinations, the absolute correspondence-induced shift exceeds the magnitude of the readout difference measured under mapped correspondence in nine cases and reverses the direction of the three-seed mean comparison in six. Better task performance can also accompany greater cross-reference disagreement, while increased agreement can include shared errors. These results show that per-reference task scores alone do not characterize how predictions change across reference representations. CrossRefEval makes reference representation and output correspondence explicit parts of genomic model evaluation rather than fixed background assumptions.

Primary area:
datasets and benchmarks
Keywords:
genomic model evaluation, reference assemblies, paired benchmark, output correspondence, cross-reference consistency

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_yCUzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CrossRefEval, a benchmark addressing a pressing, unresolved question about the behavior of Genomic Language Models (gLMs) on real biological tasks. Most existing downstream evaluations use task sequences obtained through mapping to a single reference genome, leaving potential performance shifts and prediction inconsistencies unexplored when the same gLM is evaluated on the same task across different references. CrossRefEval pairs the T2T-YAO assembly with GRCh38, constructs five downstream tasks, and evaluates five gLM backbones under different readout and output-matching strategies. The experiments systematically demonstrate that positional and other differences arising from reference-genome mapping can substantially alter gLM predictions, and that gains in task metrics should be assessed separately from cross-reference disagreement.

Critical strengths
  1. The paper carefully distinguishes among the main factors under study, including the reference genomes and their associated downstream tasks, and additionally examines differences in readout CNN design and mapping strategies.
  2. As a benchmark of cross-reference prediction shifts, the paper pays careful attention to the robustness of small numerical differences, consistently using multiple seeds and fine-grained analyses to validate and attribute the observed effects.
Critical weaknesses

Although I am not deeply familiar with the paper's biological background, I see several potentially issues:

  1. The paper's central question is how reference choice affects downstream gLM predictions, including through differences in matched positions. Yet the authors devote considerable space to prediction shifts caused by varying the readout CNN's dilation, which seems to distract from that question. Most gLMs can perform base-level and sequence-level classification effectively with simple mean pooling, a position-independent projection, or a readout from specific tokens, without adding local convolution as another variable. Although BPE-based gLMs may require an additional CNN to approximate base-level capability, this choice still seems to introduce an artificial source of variation that complicates the analysis.
  2. The authors train only task-specific readout heads for evaluation. However, a backbone's pretraining corpus may also affect its ability to generalize across references in downstream tasks: training exclusively on HG38 may differ from training on multiple reference genomes and species. Architectural priors and tokenization granularity may likewise influence the results. The paper does not adequately summarize these backbone-specific differences or control for their effects. A more systematic discussion of these factors could make the benchmark more useful in practice.
  3. The manuscript shows clear signs of AI-generated prose in its redundancy contents. These issues appear particularly pronounced in the appendices, the authors could review and refine the writing.
Questions for authors
  1. In my view, T2's base-level classification is better suited than sequence-level classification to revealing prediction shifts caused by reference changes and differences in mapped positions. However, its design raises concerns: positive examples account for only 0.3143% of the data. Although introducing class-imbalance methods can improve training, this class distribution may weaken the reported Disagreement difference.
Reviewer_FnugOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

CrossRefEval asks whether a genomic model makes consistent and correct predictions when corresponding genomic regions are represented by different reference assemblies. They instantiate CrossRefEval on the maternal haplotype assembly of T2T-YAO v2.0 and GRCh38, constructing 254,366 paired windows across five genomic prediction tasks. They evaluate five genomic foundation models.

Critical strengths
  1. Novelty: The evaluation task is novel, i.e., whether a genomic model makes consistent and correct predictions when corresponding genomic regions are represented by different reference assemblies.

  2. Controlled experiments and careful uncertainty analysis: They evaluate five models and also provide uncertainty values.

Critical weaknesses
  1. Limited genomic model: There are more SOTA models, like Evo 2, GenomeOcean, DNABERT-S, ..., should also evaluate them.

  2. The biological interpretation of cross-reference differences remains underdetermined: The labels employed are derived from annotation transfer rather than from functionally validated targets verified independently on each genome. Consequently, "correctness of pairing" merely denotes consistency with the benchmark labels. In cases of inconsistency, it is generally impossible to definitively determine whether this reflects biological sensitivity or undesirable instability.

Questions for authors

1, Could you add more SOTA models, like Evo 2, GenomeOcean, and DNABERT-S?

  1. Could you give some numerical or verbal justifications for "labels employed are derived from annotation transfer rather than from functionally validated targets verified independently on each genome" is also reasonable in this benchmark?
Reviewer_pn3yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that the reference assembly, usually a fixed assumption in genomic model evaluation, should be an explicit evaluation variable. CrossRefEval applies one fitted predictor unchanged to corresponding regions of two assemblies and reports per-reference performance, cross-reference disagreement, and paired correctness under shared labels, using position maps to pair base-level outputs whose indices shift under indels. The benchmark pairs the maternal T2T-YAO v2.0 haplotype with GRCh38 across 254,366 windows and five splicing and regulatory tasks. With five frozen DNA encoders and light convolutional readouts, the authors show that replacing mapped correspondence with equal-index matching changes the comparison between two readout variants on position-resolved splicing, with the induced shift exceeding the mapped difference in nine of ten settings and the sign of the comparison reversing in six.

Critical strengths
  • (S1) A timely question with a clear framing. As T2T, diploid, and pangenome references mature, the cross-reference consistency of genomic models becomes a practical concern. The paper rightly interprets consistency jointly with performance and paired correctness rather than as an objective in itself, and Figure 1b explains the index-shift problem clearly.

  • (S2) Careful statistics and honest reporting. Chromosome-held-out splits, a chromosome-stratified block bootstrap, and BH correction required at two block widths exceed the usual rigor of genomic benchmarks. The sensitivity analyses are thorough, and null results are reported openly.

Critical weaknesses
  • (W1) The central correspondence finding is close to tautological and practically small. Firstly, equal-index matching pairs non-corresponding bases wherever indices shift, so different disagreement rates are expected, and the paper gives no evidence that any benchmark actually uses it. The conclusion in Section 5 is therefore a sensible requirement rather than a new finding. Secondly, in Table B2, the mapped readout difference spans −0.023 to +0.127 percentage points and the induced shift −0.264 to +0.252, against T2 flip rates of only 0.23% to 0.80% (Appendix H.3.3). The nearly 3-point shift of the one-hot control mainly reflects its 5 to 17 bp raw-input span (Appendix H.2).

  • (W2) The empirical basis is too narrow for general conclusions about genomic models. Firstly, the T2 predictors barely solve the task, with frozen-encoder MCC of 0.13 to 0.20 and a supervised CNN at 0.26 to 0.28 with donor precision of 8.4% to 9.9%. Yet the correspondence analysis uses only T2, where positions with a margin below 0.01 are 0.24% to 0.38% of units but 12.8% to 27.0% of flips (Appendix B.5.4). Secondly, only two readout variants are compared, and the T1 result is predictable, since global pooling discards the positional cue of motif-centered windows that pool16 restores. Moreover, domain-standard models such as SpliceAI [1], Enformer [2], and Borzoi [3] can run on both assemblies without retraining, which matches the fixed-predictor setting exactly, yet neither they nor fine-tuned or larger foundation models are evaluated.

  • (W3) The secondary findings are largely definitional, and the paper does not explain why predictions differ. Under a shared binary label, a resolved disagreement must become either joint correctness or a shared error (Section 3.2, Appendix D.2), so shared errors in all 30 settings are expected, and the GROVER T5 fractions of 37.5% and 44.0% vary with the threshold (Appendix G.4). The claim that performance and disagreement can rise together rests mainly on HyenaDNA on T2, where MCC gains of about 0.011 to 0.014 come with flip increases of about 0.08 points. Meanwhile, YY and YH scores typically differ by less than 0.003, and 57% to 73% of T3 to T5 test windows are identical across assemblies and cannot flip (Appendix A.6), which dilutes every aggregate. Given the null regional analysis, readers cannot tell model instability from legitimate responses to sequence differences, which is the key question for such a benchmark.

References

[1] Kishore Jaganathan, et al. Predicting Splicing from Primary Sequence with Deep Learning. Cell, 176(3):535–548.e24, 2019.

[2] Žiga Avsec, et al. Effective gene expression prediction from sequence by integrating long-range interactions. Nature Methods, 18:1196–1203, 2021.

[3] Johannes Linder, et al. Predicting RNA-seq coverage from DNA sequence as a unifying model of gene regulation. Nature Genetics, 57(4):949–961, 2025.

Questions for authors
  • (Q1) Does the ranking of the five encoders change across test references or correspondence rules? Showing that a realistic model-selection decision flips would largely address my concern about significance and raise that score.

  • (Q2) Is the correspondence effect an artifact of weak predictors or of the class-weighted argmax rule? Please relate the per-predictor quantity F_equal − F_map to T2 MCC or class-wise AP across the frozen settings, the one-hot control, and the residual CNN, and check ΔF and I under a top-k or calibrated rule. A zero-shot splice model would also help, but SpliceAI was trained on chromosomes that include your test chromosomes 4, 13, 19, and 21, so it suits consistency rather than accuracy comparisons. A robust effect would raise my soundness score.

  • (Q3) Can flips be stratified by distance to the nearest sequence difference and by substitution versus indel? This would separate genuine sequence effects from model instability and turn CrossRefEval into a diagnostic tool, and a clear result would raise my significance score.

Reviewer_6iTnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces CrossRefEval, a framework for evaluating genomic models across different reference assemblies. The authors build a benchmark from T2T-YAO v2.0 and GRCh38 across five splice and regulatory tasks. They evaluate five frozen DNA encoders with small readouts and report better task performance due to cross-reference disagreement.

Critical strengths
  1. The paper includes paired YAO–GRCh38 windows with shared labels, which might potentially be useful
Critical weaknesses
  1. The main finding is expected given the fact that at the same index, base pairs would change between two different reference genomes.
  2. DNA models use only sequence, so prediction changes come from sequence differences between paired windows, which prior work has already studied (Sasse et al., 2023; Huang et al., 2023).
  3. The labels have no independent validation. All labels are transferred from GRCh38-based annotations without independent functional data.
  4. The study uses one reference pair and only the maternal haplotype. This is insufficient to demonstrate that the protocol and findings are generalizable.
  5. The observations that performance can increase with disagreement, and that new agreement can include shared errors, are true for any pair of classifiers. They do not require this benchmark and provide no biological insight.
Questions for authors
  1. Is independent functional data from the YAO individual available, for example, RNA-seq splice junctions or chromatin accessibility?
  2. Does any current practice use equal-index matching for cross-assembly comparison? If not, please explain why equal-index matching is the appropriate baseline for this analysis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.