- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
BioInvariant evaluates if frozen genomic representations respond appropriately to biological transformations. It tests four models on reverse-complement consistency, coding consequence, clinical variant effects, and regulatory motif disruption. The main finding is that raw embedding similarity can give misleading conclusions. The contribution is an evaluation protocol and empirical analysis.
Critical strengths
1.The disappearance of HyenaDNA’s motif effect when control substitutions occur upstream is a useful finding. It shows how an apparent biological signal can arise from the interaction between architecture and measurement. 2.Dinucleotide matching, centered cosine similarity, local pooling, and context-length analyses make the empirical observations more informative. The discussion also acknowledges where these checks change the conclusions.
Critical weaknesses
1.The distinction from existing work is overstated. DART-Eval already evaluates frozen, mean-pooled embeddings using cosine-distance comparisons between motif-containing sequences and matched controls (Patel et al., 2024, Section 4.2). BioInvariant adds other biological axes and useful audits, but should establish more clearly what methodological or empirical insight goes beyond that earlier evaluation. 2.The biological interpretation of a failed sensitivity test remains uncertain. A near-chance result establishes that the chosen distance does not separate the groups; it does not establish that the representation lacks the relevant information. Although the limitations acknowledge it, labels such as “false invariance” still seem stronger than the evidence supports.
Questions for authors
1.How does the motif sensitivity test differ from DART-Eval’s controlled cosine-distance test, and which conclusions specifically depend on the new design? 2.Have simple sequence representations or independently validated biological representations been evaluated on the four main axes? Such results would help distinguish limitations of the models from limitations of the measurement. 3.Can variants appear in more than one matched pair? What are the resampling units for confidence intervals and significance tests, and how is dependence among variants from the same gene handled?
5 comments
3. Simple and validated representations (Q2).
We added k-mer counts (k = 1-6, over the whole window and over 128 bp around the edit) and a GC-only scorer on all four axes (SI Table 21):
| Axis | Best simple baseline | Models | Reading |
|---|---|---|---|
| RC, 2% GC bins | GC-only: 77.7% | 7.4–36.9% | This test can be solved from GC alone, which limits the measurement |
| RC, exact GC count (new) | GC-only: 4.8% (chance) | 38.5% / 44.9% / 16.8% / 9.6% | Three models carry partner information beyond GC |
| Coding | k-mers: AUROC 0.46–0.53 | 0.45–0.53 | Neither shows a significant missense-over-synonymous ordering |
| Clinical | Local 3–5-mers: ≈ 0.55 | NT 0.630, HyenaDNA 0.538 | Only NT exceeds the baselines |
| Motif | None significant at both control positions | DNABERT-2 significant at both | Gain over baselines only upstream |
Because of the first row, exact GC-count matching is now the primary RC analysis (Table 2). For a representation with a known answer, the RC axis has two: canonical k-mer counts and the RC-equivariant Caduceus-PS (with its RC-invariant readout) both reach 100% Retrieval@1, while a naive mean over Caduceus-PS's raw states is at chance (4.3%). These validate the RC measurement only. The other three axes have no validated representation; the Limitations say so, and this gap remains.
4. Pair reuse, resampling units, dependence (Q3).
- Reuse: We do not reuse genes or variants across pairs. In every same-gene pair set (coding 750 pairs; clinical 1,000; clinical missense-only 1,000), each gene contributes exactly one pair and no variant appears in more than one pair.
- Resampling unit: the pair for paired statistics, the variant for unpaired AUROC, and the anchor for the RC and motif experiments.
- Dependence within genes: in the unpaired analyses one gene can contribute several variants (at most 11 coding, 19 clinical). We added a bootstrap of the AUROC that resamples whole genes (1,260 and 1,301 genes) or chromosomes. Gene-clustered intervals are 0.96–1.02× as wide as i.i.d. intervals, chromosomeclustered ones at most 1.15×, and no significance call changes after Holm correction. Example: clinical NT 0.630 [0.606, 0.653] (SI Table 17).
Summary of changes in the revised paper (shown in blue):
- Section 2 corrected for DART-Eval; a same-motif control at the lowest-information position of the same motif (Table 5, SI Table 23) (W1, Q1).
- The "false invariance/sensitivity" and Type I–IV labels are removed, and the Limitations say that negative results are specific to the readouts tested (W2).
- k-mer and GC-only baselines on all four axes (SI Table 21) and two RC positive controls, canonical k-mers and Caduceus-PS (Table 2) (Q2).
- Pair construction is stated in the captions (Table 3: one pair per gene; SI Tables 4 and 9: each variant in at most one pair), and the AUROC is cluster-bootstrapped by gene or chromosome (SI Table 17) (Q3).
Response to Rebuttal 1
Thank you; these analyses resolve my main concerns. I have two minor clarifications about interpreting the new results:
-
For the linear-probe analysis in Table 7 and SI Table 24, are the training and test partitions gene-disjoint? If variants from the same gene can occur in both partitions, please clarify whether the reported performance could partly reflect recognition of gene or reference-context characteristics rather than transferable variant-level information.
-
For the motif analysis, was the default upstream/downstream control draw fixed before examining the model results? Please also clarify how multiplicity is handled across the four encoders, two control positions, alternative readouts, and control-position redraws. I understand that the redraws are primarily robustness checks rather than additional independent confirmatory tests.
We thank the reviewer for a constructive review. Each of the three questions led to a new analysis. Changes are shown in blue in the revised PDF and SI; "Table n" refers to the revised PDF.
1. Difference from DART-Eval's test (W1, Q1).
We agree that Section 2 overstated the difference, and we corrected it. The designs differ as follows. DART-Eval inserts a whole consensus motif into a neutral background and compares it with a shuffled motif at the same position. We change one base (the highest-information position of a JASPAR MA0139.1 hit) of a real, ChIP-supported CTCF site in its native 3 kb context, and compare it with the same base change outside the motif.
Because our control sits at a different position, we can test whether a conclusion depends on control placement. Three conclusions depend on this design (Table 5, SI Table 26):
We also added a control closer to DART-Eval's: the same base change at the lowest-information position of the same motif. No encoder ranks the core edit above it with the control on both sides of the core (DNABERT-2 d = 0.035, p = 0.018 overall, significant only with downstream controls; NT −0.043; HyenaDNA −0.082 upstream and +0.190 downstream; Caduceus −0.216). So the motif axis shows that DNABERT-2 separates a motif edit from a non-motif edit, not that any encoder reads the motif's information content (Table 5, SI Table 23).
2. Interpreting failed sensitivity tests (W2).
We agree. A near-chance distance shows only that this readout does not separate the groups. The paper no longer uses "false invariance", "false sensitivity" or the Type I-IV labels. To see how far the negative coding result depends on the readout, we tried five others on the same variants (Table 7):
No distance readout shows a significant missense-over-synonymous ordering. The probe does separate the classes, so your point is correct: some information is present that distance does not show. But a description of the variant that needs no model (substitution type, trinucleotide context and local 3-periodic base frequencies) reaches 0.781, and under this probe adding the encoder's direction to that description gives no incremental predictive gain (AUROC 0.718–0.780, i.e. −0.063 to −0.002). The paper states that this "does not show that the information is absent". A further probe supports that caveat: adding NT's reference embedding, which carries no variant information, to the model-free description raises AUROC by 0.083 [0.064, 0.103] (0.032 [0.014, 0.051] on same-gene pairs; no gain for the other encoders). This is predictive information in the 3 kb reference context, not a variant-specific signal (SI Table 24).