Beyond Activity Prediction: Evaluating Compositional Regulatory Generalization in Genomic Foundation Models

Genomic foundation models claim that they can recognize regulatory sequences and predict activity, but whether they learn transferable rules governing combinations of regulatory elements remains unclear. We introduce a matched-perturbation evaluation framework that measures regulatory effects and interactions rather than individual sequence properties alone. This distinction is important because regulatory function is inherently combinatorial and context dependent: the effect of a motif, enhancer, or variant can change when other regulatory elements or promoter contexts change. The benchmark spans a hierarchy of increasingly complex capabilities, including single-element perturbations, pairwise interactions, configuration changes, promoter-context transfer, and higher-order combinations involving three to eight regulatory features, covering 14 pretrained model configurations and 59,851 complete combinatorial perturbation sets. Across this hierarchy, we identify a consistent dissociation between activity prediction and compositional regulatory generalization. Models capture sequence-level signals and some local effects, but their ability to predict transferable interactions declines sharply as regulatory relationships become more context dependent. Notably, models that accurately predict construct-level activity fail to recover promoter-dependent interaction changes and do not consistently outperform zero-interaction baselines for higher-order effects. These results reveal that current genomic foundation models learn regulatory activity without reliably capturing the compositional principles that govern how regulatory elements combine and transfer across contexts, motivating perturbation-based evaluation as a necessary complement to standard activity benchmarks.