- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes a benchmark for evaluating whether genomic foundation models capture compositional regulatory effects, rather than simply sequence recognition or activity prediction. The benchmark suite is set up as a hierarchy of increasing complexity, ranging from local perturbation effects, to pairwise motif interactions, to higher-order interactions. The analyses in this paper test both native models' likelihood scores, and fine-tuned heads on top of these models' embeddings. The authors conclude that these models captures local effects, but fail to predict interactions, particularly when context changes and for higher-order combinations.
Critical strengths
Supervised probes test multiple archtictures
This paper evaluates a broad set of models, and uses both native likelihodos and fine-tuned heads on frozen embeddings. The fine-tuned heads also test both linear and simple CNN architectures. These baselines are particularly useful, for example showing that strong sequence recognition does not require large-scale pretraining. This leads to some nice insights on the limitations of foundation models which are important to consider.
Focus on functional predictions and combinatorial effects is rather novel
Most existing benchmarks focus on sequence classification, activity prediction, and variant-effect prediction on individual sequences. Evaluating interactions is an underexplored area, as well as transference across contexts. Applying the same contrast to experimental and predicted values is a clean way to set up the experiment. The framing of compositional generalization as distinct from activity prediction is a valuable contribution in itself.
Treatment of dataset production is clean and well-documented
The authors take care to group related examples during dataset splitting, and they are explicit and thorough in their description of data provenance/preparation in the appendix. The authors are also generally clear when datasets are imperfect or irrecoverable (e.g. for RC4B).
Critical weaknesses
It is difficult to understand what the takeaways of each analysis are
Each RC section mostly lists ranges of metrics, and then states a conclusion. It doesn't explain how the numbers support that conclusion. Here are two examples:
- RC2: "Direction-discrimination AUROC ranges from 0.507 to 0.714, whereas signed effect-strength correlations remain weak (Spearman )."
- RC4A: "four-factor macro BDA ranges from 0.491 to 0.525, while macro Spearman correlations range from −0.050 to 0.077." Neither section really interprets these results or explains how these results lead to the conclusion being drawn. Each RC would be easier to follow with explanations which guide the reader to understand the conclusion being drawn. Most of the numbers are visible from the figures, anyways, so there's no need to call out so many of them in the main text.
Specific jargon is used without introduction
Many terms appear in the main text without definition. Some are defined only in the appendix, and some are never defined anywhere. For example, "direction discrimination", "BDA", "intact-motif preference", etc. I assume this was done to save space, but this relates heavily to the first point above, which is that there isn't enough explanation as to what the analyses are doing, and it is difficult to understand without putting a lot of effort into reading through the appendix and making inferences. For example, the y-axis of Figure 4g has terminology that isn't defined anywhere (that I could find).
RC2's native evaluation adds little beyond RC1
The native part of RC2 compares , the likelihood difference between a control sequence and changed one, against , the change in PWM scanning. This is already very similar to RC1's real-vs-shuffled analysis, because it's effectively another measurement of whether or not a model can distinguish between "real functional" vs "functionally broken" sequences. Additionally, the PWM scoring is brittle because of the need to have a high-confidence PWM match for it to be meaningful, and the difficulties inherent in choosing the right PWMs a priori. Furthermore, although experimentally measured activity changes are available, from the native models are not compared to them, but this would help distinguish RC2 from RC1, and actually test local perturbation effects.
There are many explanations for a model's failing to predict interaction effects
Particularly for RC3, the paper starts with the premise that models can recognize individual regulatory elements by masuring intact-motif preference (i.e. whether or not a model can distinguish between intact and broken motifs). But this classification has rather poor accuracy, close to random baseline in many cases. It is also a likelihood preference, not a prediction of the motif's effect on activity. Furthermore, even if this premise is accepted, the claim that accurate prediction of individual motifs does not reliably predict interactive effects, is true even without this experiment. Specifically, a purely additive model can achieve high performance on predicting single-motif effects, but always predict . A different model can predict every single effect wrong, but always predict correctly. So a mismatch between single-motif and double-motif prediction accuracy does not tell us very much. Another confounding factor here is also how predictable is to begin with. Even if a model has the capability of learning interaction effects, perhaps epistatis in motifs is so rare or noisy that it is indistinguishable from a modeling standpoint. A similar issue can be found in later RCs, as well, where a low between and is interpreted as the model failing to capture interactions, but there are other explanations for such a result (e.g. experimental results have low replicate agreement, supervised training could not generalize given the limited training set, etc.).
Several datasets are out of distribution for many models
Some of the datasets are on rather OOD species, such as Drosophila, yeast, and E. coli. To my knowledge, many of these foundation models are trained on mostly human (perhaps with mice) sequences. This OOD gap could be responsible for many of the low-performing results seen in these experiments.
Difficulty of capturing combinatorial effects is confounded by other factors
One of the paper's main claims is that model alignment (how well the foundation models capture functional effects) decreases with combinatorial complexity. This seems to be the case given the experiments, but there are a lot of confounding factors, as well. As we go up the hierarchy and reach more combinatorially complex experiments, the readouts get less training data, the holdouts get harder, the species become more out-of-distribution, the labels become noisier (higher-order differences combine more measurements), etc. So this is an expected result given the setup, and it remains hard to tease apart if these models truly are suffering more at complex tasks (I believe they are, but it's hard to conclude this from the data given).
Questions for authors
- RC1: Native likelihood scores sometimes beat out supervised models, even on the same test population. Might this be the result of pooling of the embedding layer, or because the readout's training objective?
- RC2: How did you decide which PWMs to use? How sensitive are the results to the choice of PWMs used?
- RC3: How reproducible are the experimental interaction values ? For example, what is the agreement between replicates? This would give a ceiling on how well any model could recover them.
- RC3: How large is on average and does this differ by architecture or by distance between the motifs/features? This coudl show whether native-model tests can detect interactions at all (some architectures may not be able to detect interactions beyond certain distances)
- Figure 4g: What are "Promoter-independent", "Identity oracle", "P-additive", "P-interaction" and "No interaction", and how were their scores computed?
- RC5: The hypercube for different values of share a dataset; how dense is this dataset? That is, can we reasonable expect generalization given the dataset split for each combinatorial combination?