- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes biological rule faithfulness (BRF) for evaluating genomic foundation models using RNA splicing. Controlled sequence edits identify component-associated subspaces, whose functional relevance is examined through propagated erasure and matched random interventions. Representations of experimentally assayed sequence states are then substituted into a common reference along joint component subspaces. BRF measures agreement between the resulting model interaction directions and experimental interaction directions. The paper also tests whether fixed, relation-specific BRF rankings help select models on external experimental tasks when conventional benchmark rankings disagree.
Critical strengths
- The paper asks whether genomic models capture biological relationships that conventional predictive benchmarks may miss. Using splicing as a concrete test system gives this question a clear scientific motivation and makes the framework relevant to biologically informed model assessment.
- The authors' internal analyses show that similar predictive performance can coexist with different dependence on the selected component subspaces. Their evaluation on external experimental data further suggests that relation-specific scores can provide useful model-selection information. Together, these findings support the framework as a complement to standard benchmarks within the studied tasks.
Critical weaknesses
The paper could offer a clearer interpretation of the BRF differences between models. The authors already acknowledge in Section 7 that pretraining objectives, architectures and data may all contribute. Connecting that discussion more directly to the observed results, including the role of the intervention and readout, would help readers understand what the comparison supports. The numerical reference and random-control results described in Appendix E, along with the training details of the human-PSI and yeast readouts used for BRF, also remain insufficiently reported. These details matter to the interpretation, even though the current set of checkpoints cannot isolate the causal effects of individual training choices.
Questions for authors
-
Could the authors point to or provide the numerical reference and matched-random control results described in Appendix E, and briefly clarify the training data, prediction targets, data splits and fitting procedure for the human-PSI and yeast readouts used in the BRF evaluation? This would help assess the basis for the reported model differences.
-
Could the authors briefly connect Section 7 to the observed BRF differences, considering bidirectional MLM versus causal next-token prediction, parameter scale and pretraining data? In particular, how might contextual access interact with the intervention and pooled readout, and what role might species coverage and splice-relevant sequence content play? I would find it helpful to distinguish patterns supported by the existing results from possible explanations, given the confounding among objectives, architectures and data.
-
The authors might also briefly discuss publicly released models such as NTv3’s MLM-pretrained models, GENERator and Caduceus-PS to place the work in a broader context. I would welcome such a discussion in the final version, without expecting the authors to undertake additional model experiments during the rebuttal period.
Response to Question 3
We thank the reviewer for raising this question. We sincerely apologize that the original manuscript did not sufficiently explain how BRF can be used when component knowledge is incomplete. BRF requires components that can be defined and perturbed, together with corresponding experimental measurements, but does not require prior knowledge of the complete molecular mechanism or interaction direction. We conducted the following additional experiments to examine this issue.
1. Candidate regions provide a starting point when component definitions are unclear.
Using Fan’s regulatory-sequence data [1], we compared two approaches to constructing component subspaces. One used the authors’ component-targeted edits; the other introduced edits only within two predefined 15 bp regions, without using transcription factor names, motif rules, or precise target positions, while matching the number of edits and nucleotide substitution types. Across four models and three cell lines, 83.33% of pairwise model orderings were preserved. The top-ranked model remained unchanged in GM12878 and Jurkat but changed in K562. Region-based BRF differed from component-based BRF by −1.55 to +1.81 percentage points, and the Spearman correlations between their continuous interaction estimates ranged from 0.54 to 0.88.
These results show that, in this experiment, replacing precise component-targeted edits with candidate-region edits preserved most pairwise model orderings. This reduces reliance on precise component annotations, although the candidate regions themselves must still be specified in advance.
2. Combinatorial experiments establish the reference when interactions are unclear.
Under the same background and conditions, four states are measured: the reference state, A-only perturbation, B-only perturbation, and joint A/B perturbation. These measurements test whether the effect of A depends on the state of B. The interaction direction is obtained from the measurements, without prespecifying whether the components should act synergistically or inhibit one another, or first identifying the complete mechanism underlying the relationship. An incomplete understanding of how components interact therefore does not preclude evaluation; the key requirement is that they can be perturbed separately and their combined effects measured.
3. Settings where controlled component perturbations cannot be defined remain to be explored.
We acknowledge the challenge raised by the reviewer. At present, BRF requires, at a minimum, controlled component perturbations defined at a coarse level, such as candidate regions, together with corresponding combinatorial measurements. When components are entirely unknown and such perturbation comparisons cannot yet be established, the current workflow lacks a basis for identifying component subspaces. Without combinatorial measurements, an interaction reference cannot be established either. Future work might use experimental perturbations and their responses to guide subspace searches and subsequently localize candidate components. However, this possibility remains to be validated and lies beyond the scope of the present study.
References
[1] Fan et al. Expanding the DNA Motif Lexicon of the Transcriptional Regulatory Code. bioRxiv, 2025 (preprint).