acceptodds
Under review as a conference paper at ICLR 2027

Beyond Activity Prediction: Evaluating Compositional Regulatory Generalization in Genomic Foundation Models

Abstract

Genomic foundation models claim that they can recognize regulatory sequences and predict activity, but whether they learn transferable rules governing combinations of regulatory elements remains unclear. We introduce a matched-perturbation evaluation framework that measures regulatory effects and interactions rather than individual sequence properties alone. This distinction is important because regulatory function is inherently combinatorial and context dependent: the effect of a motif, enhancer, or variant can change when other regulatory elements or promoter contexts change. The benchmark spans a hierarchy of increasingly complex capabilities, including single-element perturbations, pairwise interactions, configuration changes, promoter-context transfer, and higher-order combinations involving three to eight regulatory features, covering 14 pretrained model configurations and 59,851 complete combinatorial perturbation sets. Across this hierarchy, we identify a consistent dissociation between activity prediction and compositional regulatory generalization. Models capture sequence-level signals and some local effects, but their ability to predict transferable interactions declines sharply as regulatory relationships become more context dependent. Notably, models that accurately predict construct-level activity fail to recover promoter-dependent interaction changes and do not consistently outperform zero-interaction baselines for higher-order effects. These results reveal that current genomic foundation models learn regulatory activity without reliably capturing the compositional principles that govern how regulatory elements combine and transfer across contexts, motivating perturbation-based evaluation as a necessary complement to standard activity benchmarks.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
genomic foundation models, compositional generalization, gene regulation, higher-order interactions, out-of-distribution generalization, perturbation-based evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_S1B3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a benchmark for evaluating whether genomic foundation models capture compositional regulatory effects, rather than simply sequence recognition or activity prediction. The benchmark suite is set up as a hierarchy of increasing complexity, ranging from local perturbation effects, to pairwise motif interactions, to higher-order interactions. The analyses in this paper test both native models' likelihood scores, and fine-tuned heads on top of these models' embeddings. The authors conclude that these models captures local effects, but fail to predict interactions, particularly when context changes and for higher-order combinations.

Critical strengths

Supervised probes test multiple archtictures

This paper evaluates a broad set of models, and uses both native likelihodos and fine-tuned heads on frozen embeddings. The fine-tuned heads also test both linear and simple CNN architectures. These baselines are particularly useful, for example showing that strong sequence recognition does not require large-scale pretraining. This leads to some nice insights on the limitations of foundation models which are important to consider.

Focus on functional predictions and combinatorial effects is rather novel

Most existing benchmarks focus on sequence classification, activity prediction, and variant-effect prediction on individual sequences. Evaluating interactions is an underexplored area, as well as transference across contexts. Applying the same contrast to experimental and predicted values is a clean way to set up the experiment. The framing of compositional generalization as distinct from activity prediction is a valuable contribution in itself.

Treatment of dataset production is clean and well-documented

The authors take care to group related examples during dataset splitting, and they are explicit and thorough in their description of data provenance/preparation in the appendix. The authors are also generally clear when datasets are imperfect or irrecoverable (e.g. for RC4B).

Critical weaknesses

It is difficult to understand what the takeaways of each analysis are

Each RC section mostly lists ranges of metrics, and then states a conclusion. It doesn't explain how the numbers support that conclusion. Here are two examples:

  • RC2: "Direction-discrimination AUROC ranges from 0.507 to 0.714, whereas signed effect-strength correlations remain weak (Spearman ρ=0.006−0.305\rho = 0.006-0.305)."
  • RC4A: "four-factor macro BDA ranges from 0.491 to 0.525, while macro Spearman correlations range from −0.050 to 0.077." Neither section really interprets these results or explains how these results lead to the conclusion being drawn. Each RC would be easier to follow with explanations which guide the reader to understand the conclusion being drawn. Most of the numbers are visible from the figures, anyways, so there's no need to call out so many of them in the main text.

Specific jargon is used without introduction

Many terms appear in the main text without definition. Some are defined only in the appendix, and some are never defined anywhere. For example, "direction discrimination", "BDA", "intact-motif preference", etc. I assume this was done to save space, but this relates heavily to the first point above, which is that there isn't enough explanation as to what the analyses are doing, and it is difficult to understand without putting a lot of effort into reading through the appendix and making inferences. For example, the y-axis of Figure 4g has terminology that isn't defined anywhere (that I could find).

RC2's native evaluation adds little beyond RC1

The native part of RC2 compares ΔZ\Delta Z, the likelihood difference between a control sequence and changed one, against ΔW\Delta W, the change in PWM scanning. This is already very similar to RC1's real-vs-shuffled analysis, because it's effectively another measurement of whether or not a model can distinguish between "real functional" vs "functionally broken" sequences. Additionally, the PWM scoring is brittle because of the need to have a high-confidence PWM match for it to be meaningful, and the difficulties inherent in choosing the right PWMs a priori. Furthermore, although experimentally measured activity changes are available, ΔZ\Delta Z from the native models are not compared to them, but this would help distinguish RC2 from RC1, and actually test local perturbation effects.

There are many explanations for a model's failing to predict interaction effects

Particularly for RC3, the paper starts with the premise that models can recognize individual regulatory elements by masuring intact-motif preference (i.e. whether or not a model can distinguish between intact and broken motifs). But this classification has rather poor accuracy, close to random baseline in many cases. It is also a likelihood preference, not a prediction of the motif's effect on activity. Furthermore, even if this premise is accepted, the claim that accurate prediction of individual motifs does not reliably predict interactive effects, is true even without this experiment. Specifically, a purely additive model can achieve high performance on predicting single-motif effects, but always predict ϵZ=0\epsilon_{Z} = 0. A different model can predict every single effect wrong, but always predict ϵZ=ϵexp\epsilon_{Z} = \epsilon_{exp} correctly. So a mismatch between single-motif and double-motif prediction accuracy does not tell us very much. Another confounding factor here is also how predictable ϵexp\epsilon_{exp} is to begin with. Even if a model has the capability of learning interaction effects, perhaps epistatis in motifs is so rare or noisy that it is indistinguishable from a modeling standpoint. A similar issue can be found in later RCs, as well, where a low ρ\rho between ϵZ\epsilon_{Z} and ϵexp\epsilon_{exp} is interpreted as the model failing to capture interactions, but there are other explanations for such a result (e.g. experimental results have low replicate agreement, supervised training could not generalize given the limited training set, etc.).

Several datasets are out of distribution for many models

Some of the datasets are on rather OOD species, such as Drosophila, yeast, and E. coli. To my knowledge, many of these foundation models are trained on mostly human (perhaps with mice) sequences. This OOD gap could be responsible for many of the low-performing results seen in these experiments.

Difficulty of capturing combinatorial effects is confounded by other factors

One of the paper's main claims is that model alignment (how well the foundation models capture functional effects) decreases with combinatorial complexity. This seems to be the case given the experiments, but there are a lot of confounding factors, as well. As we go up the hierarchy and reach more combinatorially complex experiments, the readouts get less training data, the holdouts get harder, the species become more out-of-distribution, the labels become noisier (higher-order differences combine more measurements), etc. So this is an expected result given the setup, and it remains hard to tease apart if these models truly are suffering more at complex tasks (I believe they are, but it's hard to conclude this from the data given).

Questions for authors
  • RC1: Native likelihood scores sometimes beat out supervised models, even on the same test population. Might this be the result of pooling of the embedding layer, or because the readout's training objective?
  • RC2: How did you decide which PWMs to use? How sensitive are the results to the choice of PWMs used?
  • RC3: How reproducible are the experimental interaction values ϵexp\epsilon_{exp}? For example, what is the agreement between replicates? This would give a ceiling on how well any model could recover them.
  • RC3: How large is ϵZ\epsilon_{Z} on average and does this differ by architecture or by distance between the motifs/features? This coudl show whether native-model tests can detect interactions at all (some architectures may not be able to detect interactions beyond certain distances)
  • Figure 4g: What are "Promoter-independent", "Identity oracle", "P-additive", "P-interaction" and "No interaction", and how were their scores computed?
  • RC5: The hypercube for different values of kk share a dataset; how dense is this dataset? That is, can we reasonable expect generalization given the dataset split for each combinatorial combination?
Reviewer_fX6uOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a benchmark of DNA foundation models on combinatorial regulatory effects and epistatic interactions, extending existing DNA benchmarks which look only at individual sequences and single-variant effects (BEND from Marin et al., DART-Eval from Patel et al.). The proposed benchmark is organized into five tasks of increasing complexity: sequence recognition (RC1), local perturbation responses (RC2), pairwise interactions (RC3), contextual transfer (RC4), and higher-order combinations (RC5), in which R[1,2,3,5] are meant to measure regulatory capability and R4 is meant to measure context generalization. The authors evaluate 14 genomic foundation model configs (11 genomic/phylogenetic models and 3 sequence-to-function models) using both zero-shot scores and supervised readouts on frozen representations. While models perform well at sequence recognition (RC1) and can achieve high construct-level activity accuracy when promoter contexts are seen during training (RC4B), correlations with ground truth pairwise and higher-order interactions are typically weak (RC3, RC5). The authors interpret these results as evidence that strong activity prediction does not imply that models can learn compositional regulatory rules.

Critical strengths
  1. The benchmark is a meaningful addition to the DNA foundation model evaluation literature. To my knowledge, it is the first benchmark to evaluate DNA foundation models against experimentally measured non-additive interactions and their transfer across regulatory contexts. Prior work either looks at model-internal interactions (GIA from Koo et al.; CREME from Toneyan & Koo; SQUID from Seitz et al.) or tests predefined rules (Mechanistic Invariance Test from Cheng & Zhang), which the authors are careful to cite. The benchmark's tasks are well designed and show the importance of directly evaluating combinatorial regulation; accurate activity prediction does not necessarily imply accurate interaction prediction. The benchmark tests model capabilities that existing evaluations (i.e. BEND, DART-Eval) do not directly measure.

  2. The benchmark includes broad coverage of DNA foundation models, and leakage is controlled carefully. The benchmark includes 14 model configurations including genomic language models, a phylogenetic model, and sequence-to-function models, plus from-scratch baselines (k-mer based models, CNN, MPRA-LegNet) and zero-interaction baselines. The authors take meaningful steps to avoid train/test leakage by keeping related perturbation states together and using task-specific holdouts (e.g. by chromosome, enhancer, transcription-factor pair, or promoter context). In the higher-order interaction eval, sequences with shared DNA, reverse complements, or parent families are grouped together. These controls strengthen confidence in the results.

Critical weaknesses
  1. Measurement noise is a real concern in this study. The pairwise and higher-order contrasts in equations 2 and 5 combine multiple activity measurements with alternating signs, which can reduce signal to noise. This is confirmed by the reliability analysis in Appendix C.1, which reports weak replicate correlations for third-order e2MPRA effects (Spearman 0.086 to 0.175) and N8 strata (-0.129 to 0.237). The authors correctly note that "low model correlation in weakly reproducible strata does not by itself establish absence of higher-order information" in the Appendix but do not mention this caveat adequately in the main text. Results are not normalized by a noise ceiling or restricted to reproducible strata. The paper’s main finding, a gap between activity prediction and interaction prediction, is also what one would expect if activity labels are reliable and interaction labels are noisy. Ceiling-normalized metrics and reporting results only on reproducible strata would help establish that the findings can be attributed to model capabilities. Several key results also rely on small sample sizes or limited biological diversity:

    • (a) RC3 interaction correlations are estimated from only 20 families spanning 11 enhancers, with shared constructs across families (Table D4). This limits precision of the native-score correlations (-0.398 to 0.386). Confidence intervals for the three sequence-to-function model correlations also include zero (Table D5).
    • (b) Several RC5 strata are too small to support the study's conclusions: King k=4 interactions include only 23 cubes and the two QeFS conditions contain only three and five cubes (Fig. 5). These strata provide limited evidence about higher-order prediction capability.
    • (c) About 95% of the 59,851 higher-order sets cited in the abstract (57,044) come from Yu et al.’s bacterial promoter dataset (Table A4). This is a vastly different setting from the rest of the benchmark, which targets eukaryotic regulation.
  2. The benchmark lacks a positive control for recovering interactions. Although the authors report experimental repeatability in Appendix C.1, they do not show that the evaluation pipeline can recover known interaction structure when it exists. Using a dataset with known epistasis could test this recovery, in which a task-trained model could establish an upper bound on performance. Existing baselines (k-mer ridge, CNNs) rarely outperform the zero-interaction baseline and hence do not fill this role. The reported baseline results suggest that the performance bottleneck could be the data or eval protocol rather than model failures. Relatedly, models are evaluated only with native (zero-shot) scores and supervised readouts trained on frozen representations (Appendix B.3). No models are fine-tuned, which can undersell their capabilities. DART-Eval found that conclusions depended on the adaptation setting (zero-shot vs. probing vs. fine-tuning), with probed frozen embeddings often underperforming other approaches. Where possible, fine-tuning is an important setting to include before attributing generalization failure to model capabilities.

Questions for authors

Clarifications:

  1. Where possible, can the authors report model performance relative to a replicate-based noise ceiling, e.g. by using one replicate to predict another replicate or by using a split-half reliability estimate?
  2. Do the authors' conclusions still hold when RC3-RC5 are restricted to strata with sufficient replicate agreement, e.g. King k = 3 (Spearman 0.71-0.74)?
  3. Regarding positive controls, can the evaluation pipeline recover interactions when they are known to exist? For example, by using simulated data with planted epistasis, or with a model trained directly on interaction targets? If no method beats the zero-interaction baseline, it is difficult to distinguish model failure from targets that can’t be predicted.
  4. In RC4B, why were the genomic and phylogenetic models not re-evaluated on the corrected-v2 build? This should be done to confirm the findings in this section.
  5. As noted above, about 95% of the 59,851 higher-order interactions come from Yu et al. Do conclusions for RC5 change if Yu is excluded or reported separately? More generally, why would a bacterial promoter library be suitable for evaluating models pretrained mostly on eukaryotic/human data?
  6. Where possible, can the authors fine-tune any models either shallowly or end-to-end (even on one task, e.g. RC4B)? If frozen readouts are the performance bottleneck, fine-tuning could change the conclusions or relative rankings of models.
  7. Can the authors report confidence intervals for the native-score correlations in Table D4?
  8. How sensitive are the RC4B results to the choice of predictor used to combine separately embedded components? For example, do results change with a nonlinear head (e.g. an MLP over component embeddings) or more expressive ways of combining components?

No suggestions for future work not already captured above.

Reviewer_KAxrOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates whether genomic foundation models capture compositional regulatory behavior which is a meaningful lens to evaluate a foundational model apart from sequence recognition and activity prediction. It introduces five evaluation levels covering recognition, local perturbations, pairwise non-additive effects, regulatory configuration transfer, and higher-order interactions. Several pretrained configurations are tested using native predictions and supervised readouts of frozen representations. The main finding is that strong sequence recognition does not necessarily translate into accurate prediction of regulatory interactions.

Critical strengths
  1. The research question is very well thought out and an important problem to address. Recognizing regulatory sequences is not equivalent to understanding how multiple regulatory elements interact. Evaluating non-additive effects and generalization to unseen promoter contexts is a fresh take beyond standard sequence classification.

  2. The evaluation includes multiple pretrained models, biological settings, and interaction orders. The use of simple additive and zero-interaction baselines helps establish whether predictions capture anything beyond individual sequence effects.

Critical weaknesses
  1. The evaluation levels are not directly comparable. Dataset sizes, experimental protocols, and measurement reliability vary considerably across RC1–RC5. RC3 contains only 20 complete four-state families, while higher-order experiments contain many overlapping perturbation sets. This makes it difficult to attribute declining performance specifically to increasing interaction complexity rather than dataset characteristics or measurement noise.

  2. The conclusions about learned representations need stronger support. The experiments primarily evaluate model predictions or downstream readouts, rather than directly investigating regulatory information encoded in pretrained representations. Poor interaction prediction does not necessarily establish that the foundation model lacks that information. The readout architecture, additive reference model, and activity measurement scale could all influence the result.

  3. There is also a scalability limitation: exact higher-order contrasts require (2^k) perturbation combinations for (k) binary features. These contrasts measure statistical non-additivity, not necessarily direct biological interactions.

Questions for authors
  1. Can the authors account for dataset size and experimental repeatability when comparing performance across RC1–RC5? Does the same decline remain after these controls?

  2. How can the authors distinguish missing regulatory information in pretrained representations from information that the downstream readout fails to recover? This would help strengthen the contribution.

  3. How sensitive are the higher-order conclusions to the chosen activity scale and additive reference model?

  4. Can the proposed evaluation scale beyond eight interacting features without exhaustively enumerating all perturbation combinations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.