acceptodds
Under review as a conference paper at ICLR 2027

REPORTER Benchmarks Genomic Representations against Regulatory Activity Experiments

Abstract

Pretraining is increasingly being used to improve the performance of genomic sequence models, but assessing whether pretrained representations broadly capture sequences' gene-regulatory functions remains a challenge. Here, we introduce REPORTER, a set of 22 transfer-learning tasks that we use to evaluate several genomic language models (gLMs) and AlphaGenome—a leading functional-genomics model—as pretrained backbones. Our task data are curated from 14 publicly available massively parallel reporter assay experiments that altogether screened over 2.5 million DNA sequences from across the tree of life for transcriptional, splicing, or UTR-mediated regulatory activity. We find that the most capable gLMs can outperform models trained from scratch on nearly all of our tasks, but fall short of AlphaGenome on most of our animal tasks. Our results provide empirical insight into pretraining, backbone selection, and transfer learning. Looking forward, REPORTER can serve model developers as a standard benchmark and design target.

Primary area:
datasets and benchmarks
Keywords:
genomic language models, genomic foundation models, transfer learning, benchmarking, massively parallel reporter assays, sequence-to-function prediction, variant-effect prediction, regulatory genomics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_dWZbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a novel benchmark called Reporter to evaluation pretrained DNA models. They also do an extensive evaluation with 25 pretrained models, focusing on lora-based fine-tuning performance, as opposed to ZS performance.

Critical strengths

Nicely written paper, focused on an important topic, and with a clear evaluation. I also appreciate that the authors include the following sentence: "Our benchmark measures performance under this shared protocol, which may not reflect each backbone’s best attainable performance" when describing model performance, though I would argue that more effort could be considered for the baseline specifically (see below).

Critical weaknesses

Strong baselines are crucial for a benchmark (e.g., see this paper: https://arxiv.org/abs/2411.02796). I would encourage the authors to either further motivate the strength of their baselines or to put more effort to ensuring that their baselines are strong enough.

nit: line 51: "MLM" used in the intro before it's defined

nit: "We apply REPORTER to benchmark 25 gLM backbones, including six species-specific MLM base- lines that we pretrained, alongside AlphaGenome, a leading human functional-genomics model re- cently repurposed as a pretrained backbone": This sentence ws very hard for me to parse. It's quite important as it summarizes the evaluation done in the paper, so I'd suggest improving it.

Questions for authors

Is there any concern about data contamination? i.e., have any of the pretrainined models under consideration potentially seen any of this benchmarking data, given that the benchmark was derived from data from published studies?

2 comments
AuthorsOpenReview ↗

Response to Reviewer dWZb

Thank you for the review. As you note, our results reflect a shared protocol rather than each backbone's best attainable performance, which makes a strong baseline all the more important.

Strong baselines are crucial for a benchmark (e.g., see this paper: https://arxiv.org/abs/2411.02796). I would encourage the authors to either further motivate the strength of their baselines or to put more effort to ensuring that their baselines are strong enough.

The revision motivates the From-Scratch Baseline and tests it in three ways.

  1. Its CNN architecture follows established practice in MPRA modeling: all seven of our 14 source studies that trained a deep model on their data used a CNN (Section 3.4).
  2. A second training seed gives closely matching results (Figure S6).
  3. Neither added genomic context nor sequence shared between test and training windows explains the pretrained models' gains (Figure S11, Table S4).

nit: line 51: "MLM" used in the intro before it's defined

nit: "We apply REPORTER to benchmark 25 gLM backbones, including six species-specific MLM base- lines that we pretrained, alongside AlphaGenome, a leading human functional-genomics model recently repurposed as a pretrained backbone": This sentence ws very hard for me to parse. It's quite important as it summarizes the evaluation done in the paper, so I'd suggest improving it.

We fixed these issues in the revision—thanks!

Is there any concern about data contamination? i.e., have any of the pretrainined models under consideration potentially seen any of this benchmarking data, given that the benchmark was derived from data from published studies?

None of the models—including the post-trained version of Nucleotide Transformer and AlphaGenome, which are the two supervised backbones we evaluated—were trained on any MPRA data.

Thanks for the response. I will leave my positive score as is.

Reviewer_YcM7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

REPORTER curates 22 transfer-learning tasks from 14 massively parallel reporter assay studies, spanning animals, plants, yeast, and prokaryotes and covering transcription, splicing, and UTR-mediated regulation. It standardizes assay readouts, groups related sequences and parent constructs for partitioning, and evaluates pretrained genomic backbones using a shared prediction head and model-specific LoRA adapters. The benchmark includes 25 genomic language-model backbones, including six author-trained species-specific baselines, and the AlphaGenome convolutional encoder. Its analyses distinguish overall activity, within-element or paired variant effects, cell-line differences, and transfer between assay libraries. Strong pretrained models usually outperform the from-scratch reference under this protocol, but their advantages vary by regulatory function and species; mutation-specific cell-line effects remain challenging.

Critical strengths
  1. This is a substantial and useful benchmark contribution with careful attention to biological and experimental structure. The 22 tasks cover multiple taxa, three broad regulatory mechanisms, different assay readouts, and multiple cellular contexts. Sequence-identity clusters, common-parent groupings, and joint partitioning across related libraries address important sources of train/test contamination; cross-library transfer explicitly excludes source-training parent elements from target evaluation (Section 3.1; Appendix A.1–A.2, pp. 14–19). The detailed source-specific reconstruction work makes the collection much more valuable than a simple aggregation of published activity scores.

  2. The analyses go beyond a single leaderboard and identify meaningful limits of current representations. The paper distinguishes global activity from variant, within-window, and cell-line effects, and examines both cross-library transfer and the relationship with zero-shot VEP. AlphaGenome and Evo 2-7B each exceed the from-scratch reference on 19 of 22 tasks, while the designated species-specific MLM references do so on only 10; functional post-training improves NTv3-650M on all 13 jointly evaluated tasks (pp. 6–8). The contrast between good overall predictions and much weaker mutation-specific cell-line prediction is especially informative: AlphaGenome reaches 0.37 on the latter Kendall metric, whereas the other evaluated models are at or below 0.1 (p. 9; Fig. S9). The frozen-head comparison and AlphaGenome context sweep provide useful supporting diagnostics rather than relying solely on final rankings (Figs. S5C and S10).

Critical weaknesses

The exact benchmark is not yet sufficiently reproducible from the provided submission. The availability statement says that implementation code and author-trained backbones are not included, while processed data and exact splits are promised for later release (p. 10). This matters for a proposed standard benchmark because several consequential choices cannot be recovered exactly from prose alone. For example, five source studies use input pseudocounts inversely proportional to cluster size, but the precise normalization scale is not specified (Appendix A.1, p. 14). The paper correctly notes that these counts do not recover experimental uncertainty and that adding one before scoring makes the activity sensitive to common count rescaling (Appendix A.4, p. 21). Consequently, this is not just a missing convenience: the precise reconstruction can affect training, scoring weights, and reported correlations. An executable scorer, processing specifications, and the actual partition manifests would substantially strengthen the benchmark contribution.

Questions for authors
  1. Reproducibility clarification: Can you provide the exact partition manifests and executable scoring/processing specifications, including the pseudocount normalization constants and grouping/subsampling rules? Please also reconcile the strict split-cap description with the larger entries in Table S1. These details would determine whether independent users can reproduce the proposed benchmark scores.

  2. Expanded-context clarification: Appendix A.1 states that split checks exclude extended genomic flanks, while backbones receive different context lengths and the from-scratch model receives only the assayed oligo. Do the existing partition manifests or diagnostics characterize cross-split context overlap, and how should that inform the interpretation of comparisons across context lengths? Fig. S10 usefully tests AlphaGenome's sensitivity to context length, but it does not directly describe such overlap. I recognize that the benchmark compares adaptation protocols rather than isolating pretraining alone, and that context overlap is not by itself evidence of label leakage.

1 comment
AuthorsOpenReview ↗

Response to Reviewer YcM7

Thank you for the review. As you note, the source-specific reconstruction is much of what makes REPORTER useful, so the gaps you found in its documentation mattered. The pseudocount scale was unspecified, and Table S1 (now Table S2) did not match our description of the split caps. Both are fixed. We are especially grateful for your question on context overlap, which led to new Table S4 and Figure S11.

The exact benchmark is not yet sufficiently reproducible from the provided submission.

We want to make our work as easy as possible to reproduce and extend. We will release a Zenodo repository containing our datasets, as well as a GitHub repository including partition manifests, evaluation scripts, and training code. The GitHub repository will serve as a versioned implementation of REPORTER and a maintained resource for evaluating additional models.

Our pipeline for processing the raw data from our source studies is tangled up with our compute environment, so we do not plan to release it. Instead, the revised manuscript describes it in much more detail.

An anonymized preview of the repository is included in the revised supplementary material. It documents benchmark construction, study-specific processing, training, and evaluation. It includes partition statistics, code, configurations, processed-data samples for all 22 tasks, reference results, and a CPU-runnable scoring example.

For example, five source studies use input pseudocounts inversely proportional to cluster size, but the precise normalization scale is not specified

We included these details in the revised manuscript; their omission was an oversight. For each processed library, let ngn_g denote the number of records in sequence cluster gg measured in at least one condition, and let nmax⁡=max⁡gngn_{\max} = \max_g n_g. For oligo ii in cluster g(i)g(i) and an observed condition cc, we set

xic=nmax⁡ng(i),yic=xicaic,x_{ic}=\frac{n_{\max}}{n_{g(i)}},\qquad y_{ic}=x_{ic}a_{ic},

where aica_{ic} is the published linear activity ratio. Thus, members of the largest cluster receive an input pseudocount of 1. If the released activity is on the log2 scale, we use aic=2zica_{ic}=2^{z_{ic}}.

Please also reconcile the strict split-cap description with the larger entries in Table S1.

Done—thanks for pointing this out. In the revision, we assign related records to the same partition, targeting 2,000, 5,000, or 20,000 records per validation and test split, depending on the task. All remaining eligible records are assigned to training.

Do the existing partition manifests or diagnostics characterize cross-split context overlap, and how should that inform the interpretation of comparisons across context lengths?

Because our split checks consider only the assayed oligos, we had not measured how much flanking sequence test windows share with training windows. That overlap grows with context size and varies by assay design. At 8,192 bp, it ranges from under 4% in sparse gene-level libraries to nearly 100% in densely tiled ones. We now report it for every task in new Table S4, summarized below.

Test-set genomic positions that overlap training windows (%, new Table S4)

Task512 bp2,048 bp8,192 bp
ecoli_promoter_tss50.489.9100.0
drosophila_enhancer76.894.299.9
yeast_promoter91.596.198.8
human_tx_s26_snp32.164.584.2
human_tx_l2522.051.482.0
at_enhancer13.852.181.0
human_3utr29.852.260.7
human_tx_a2515.436.959.4
tomato_enhancer2.423.952.9
sorghum_enhancer6.120.950.7
maize_enhancer4.022.942.2
at_promoter_j210.84.035.0
prokaryote_promoter0.34.019.5
sorghum_promoter0.41.311.6
human_5utr2.25.510.1
maize_promoter0.71.88.6
human_splicing0.21.35.6
at_promoter_v24_native0.00.25.5
mouse_tx0.51.73.8

Two analyses indicate that this overlap does not explain the pretrained models' gains. First, restricting each test set to oligos whose 2,048 bp window never overlaps training leaves each pretrained model's gain over the From-Scratch Baseline nearly unchanged (median change in ΔrGlobal\Delta r_{\mathrm{Global}} of −0.001, interquartile range −0.012 to 0.007, across 269 model–task pairs from tasks with at least 100 overlapping and 100 non-overlapping test oligos). Second, a Large-Context CNN trained from scratch, which extends the From-Scratch Baseline with dilated layers, gains a median of only 0.008 in rGlobalr_{\mathrm{Global}} across the 19 source tasks when given the same 2,048 bp window (new Figure S11). AlphaGenome, given that window, exceeds the From-Scratch Baseline by a median of 0.123. Because every primary comparison caps input at 2,048 bp, these analyses cover all of them. Longer windows enter only the AlphaGenome sweep in Figure S10.

Reviewer_1i98OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces REPORTER, a set of 22 transfer-learning tasks that is used by the Authors to evaluate several genomic language models (gLMs) and AlphaGenome, as pretrained backbones. The task data are curated from 14 publicly available massively parallel reporter assay experiments that altogether screened over 2.5 million DNA sequences from across the tree of life for transcriptional, splicing, or UTR-mediated regulatory activity. The Authors find that the most capable gLMs can outperform models trained from scratch on nearly all of the considered tasks, but fall short of AlphaGenome on most of our animal tasks.

Critical strengths

The task of evaluating foundational models for genomes is of great practical importance in the molecular design. Such models often serve as backbones to train derivative models generating cells of the particular genomic profiles that in turn can inform such mechanisms as CRISPS on the optimal set of genomic edits for the particular downstream functionality (e.g. altered genome expression to modulate autoimmune response in the autoimmune diseases). To the best of Reviewer's knowledge, this is the first such comprehensive set of benchmarks to quantitatively evaluate those foundational backbones. Furthermore, the REPORTER benchmark led to several interesting findings, as the Authors admit: the REPORTER distinguishes gLMs with similar performance on existing evaluations, revealing capabilities not captured by conventional benchmarks. Furthermore, it shows that the tested gLMs fall short of AlphaGenome on most of our animal tasks. The empirical section is comprehensive, with particularly useful findings on the transfer performance across zero-shot VEP strata (see: Fig. 3). The finding that benchmarked models only partially capture cell-line-specific regulatory effects is surprising, yet of great importance to the entire Community.

Critical weaknesses

It would be great if that comprehensive benchmark can be translated to the new set of tasks that can be used to further finetune gLMs and improve their generalization properties. The Reviewer would be very interested to learn what the Authors' view on that is. More detailed description of how the 22 transfer-learning task were chosen and curated would be very useful.

Questions for authors

Please see the "Critical Weaknesses" section above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 1i98

Thank you for the review. We share your interest in using these data to improve gLMs, not only to evaluate them, and respond to both suggestions below.

It would be great if that comprehensive benchmark can be translated to the new set of tasks that can be used to further finetune gLMs and improve their generalization properties.

We agree that improving gLMs’ representations with a second stage of supervised learning on functional-genomics data is a very promising idea (indeed, Nucleotide Transformer, one of the models we evaluated, achieved better results after doing this with non-MPRA data). Our MPRA datasets can be certainly used for this purpose as well, and our modeling framework provides one well-justified way to formulate the task of learning from these data. However, there are considerations—such as the possibility of forgetting information after fine-tuning, for example—that warrant more careful thinking to demonstrate feasibility. We briefly acknowledged this in the Discussion section, and have expanded on this point in the revised manuscript (based on your feedback).

More detailed description of how the 22 transfer-learning tasks were chosen and curated would be very useful.

In our revised Methods section, we now touch on how we settled on our tasks. To put it plainly, we curated as many tasks as possible. Within each study, we defined tasks and evaluation metrics to match the questions the study was designed to address, such as variant effects in studies of genetic variants and cell-line differences in studies that assayed several cell lines. We were fortunate to have found enough suitable data to assemble a comprehensive benchmark. (Note that curating data from studies originally conducted for other purposes was an aspect of this project that required substantial domain expertise, creativity, and effort.) We look forward to expanding the set of tasks as more data are made available.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their thoughtful comments and have revised the manuscript accordingly.

Summary of changes (reviewer IDs mark responses to specific comments)

Benchmark data and partitions

  • The pseudocount reconstruction for the five studies that released processed activities is now fully specified, with each task's normalization constant in new Table S1 (YcM7).
  • Methods detail study-specific processing for all 14 studies, including replicate summation, multi-locus sequences, and all exclusions (YcM7).
  • Sequence clusters now use normalized global edit similarity with task-specific thresholds of 60–85% (previously 90% Hamming similarity for all tasks). Each cohort uses the lowest threshold keeping the largest cluster at ≤2% of source records (four documented exceptions).
  • Partitions now assign whole clusters to validation and test sets of fixed size (2,000, 5,000, or 20,000 records by task), and all remaining records go to training. Table S2 reports the realized sizes and each task's clustering threshold (YcM7).
  • The tasks human_tx_s26_satmut, at_promoter_v24_gatc, and ecoli_promoter_scanmut are now test-only. Models are trained and selected on the paired source task and applied without refitting to mutants whose parent is in the source test set.
  • The task human_3utr now keeps only the reference-background allele pair for each variant.

Training and evaluation

  • Training length is set by sampled examples (at most 3,072,000 per run), not epochs.
  • Adapted backbones use a learning rate of 1e-4 (previously 3e-4), and the From-Scratch Baseline uses 1e-3. A single rule replaces per-model exceptions: a run with constant validation predictions restarts once at 1e-5.
  • Total input, including genomic flanks, is capped at 2,048 bp for every backbone (previously up to 32,768 bp).
  • The From-Scratch Baseline, the Species-Specific MLM Baselines, and AlphaGenome are trained with two seeds, and Figure 2 reports their Fisher-z average. The seeds agree closely (Pearson r≥0.97r \geq 0.97, new Figure S6). All other backbones use one seed.
  • All results were regenerated under the revised partitions and protocol.

Models

  • We now benchmark 28 gLM backbones alongside AlphaGenome, adding Carbon-3B, Carbon-8B, PlantCAD2-Large, and the long-context Evo 2-7B release, which we report separately from Evo 2-7B Base.
  • Methods now explain the choice of a CNN for the From-Scratch Baseline: all seven source studies that trained a deep model on their data used a CNN (dWZb).

Results and figures

  • Figure 2 has two new panels: (C) balanced gains in rGlobalr_{\mathrm{Global}} over the From-Scratch Baseline within each organism category and (D) training compute relative to that baseline. Former panel C (direct vs. transfer training) is removed, as those three tasks are now test-only.
  • Figure 2 now shows ArGamba-dual, the top gLM on human_3utr and mouse_tx, instead of Caduceus-PS (now in Figure S7); the Abstract and Results name it.
  • Figure 3 now plots each backbone's balanced Human and Mouse gain against its zero-shot VEP performance on ClinVar and both TraitGym sets, scored by likelihood ratio (14 backbones) or embedding distance (18 backbones). Evo 2 VEP scores now use the same checkpoints we adapt for REPORTER.
  • The cell-line analysis now compares global and cell-line correlations directly and reports saturation-mutagenesis cell-line metrics on the full target library (Figure S9).
  • New Table S4 reports each task's share of test-set genomic positions overlapping training, from the oligo alone to 8,192 bp windows. Excluding test oligos whose 2,048 bp window overlaps training leaves pretrained models' gains nearly unchanged (YcM7).
  • New Figure S11 shows that genomic context alone does not explain these gains: a from-scratch Large-Context CNN gains a median of only 0.008 in rGlobalr_{\mathrm{Global}} from a 2,048 bp window (YcM7).
  • Figure S10 (AlphaGenome context sweep, 512–8,192 bp) is rebuilt on the revised partitions.
  • Figure S5 now reports the agreement between test NLL and each auxiliary correlation. Former Figure S5C (frozen vs. LoRA-adapted backbones), not repeated under the revised protocol, is removed.
  • Figures S3, S4, and S7–S9 are regenerated. The task-level VEP and aggregation-sensitivity figures are removed.

Text

  • The Introduction defines MLM at first use and splits the sentence summarizing the benchmarked models (dWZb), and cites work supporting each motivating application of gLMs.
  • Methods now state that we included every published study we could identify that met our inclusion criteria (1i98).
  • The Background cites GUANinE, BEND 2.0, and Tang et al. (2025) as evaluations that include MPRA data, and distinguishes likelihood-ratio from embedding-distance VEP scores.
  • The reproducibility statement describes the supplementary archive: partitioning and scoring code, configurations, partition assignments with checksums, processed-data samples, and a worked scoring example (YcM7).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.