REPORTER Benchmarks Genomic Representations against Regulatory Activity Experiments

Pretraining is increasingly being used to improve the performance of genomic sequence models, but assessing whether pretrained representations broadly capture sequences' gene-regulatory functions remains a challenge. Here, we introduce REPORTER, a set of 22 transfer-learning tasks that we use to evaluate several genomic language models (gLMs) and AlphaGenome—a leading functional-genomics model—as pretrained backbones. Our task data are curated from 14 publicly available massively parallel reporter assay experiments that altogether screened over 2.5 million DNA sequences from across the tree of life for transcriptional, splicing, or UTR-mediated regulatory activity. We find that the most capable gLMs can outperform models trained from scratch on nearly all of our tasks, but fall short of AlphaGenome on most of our animal tasks. Our results provide empirical insight into pretraining, backbone selection, and transfer learning. Looking forward, REPORTER can serve model developers as a standard benchmark and design target.