acceptodds
Under review as a conference paper at ICLR 2027

BioInvariant: Evaluating Appropriate Biological Invariance and Sensitivity in Genomic Foundation Models

Abstract

Genomic foundation models are increasingly used as general-purpose DNA encoders, but they are still evaluated primarily through downstream accuracy. Predictive accuracy alone does not reveal whether representation geometry reflects known biological structure or is dominated by simpler sequence statistics such as GC content. We introduce BioInvariant, a counterfactual evaluation framework for frozen genomic representations. BioInvariant tests two complementary properties: appropriate invariance under transformations that preserve biological meaning, and appropriate sensitivity under perturbations that disrupt biological meaning, always relative to composition-matched controls. We evaluate DNABERT-2, Nucleotide Transformer, HyenaDNA, and Caduceus across four axes: reverse-complement consistency, coding consequence, clinical variant signal, and regulatory motif disruption. Raw embedding similarity can be misleading: Caduceus attains the highest raw reverse-complement similarity, but its reverse-complement-specific signal nearly disappears after GC-matched correction; DNABERT-2 and Nucleotide Transformer show stronger corrected specificity. Sensitivity results are also axis-dependent, and no model is consistent across all axes. Additional audits show that context length, GC shortcuts, background perturbations, pooling, and control position substantially affect interpretation; in particular, HyenaDNA's apparent motif sensitivity depends on where the control substitution is placed, and only DNABERT-2's motif effect is significant for both upstream and downstream controls. BioInvariant therefore complements downstream benchmarks with a representation-level criterion: whether frozen genomic encoders move for biologically appropriate reasons under controlled, matched counterfactuals.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Genomic Foundation Models, DNA Language Models, Representation Learning, Genomics, Biological Invariance

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 6 reviews, above 68% of the others.
Reviewer_tLQEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BioInvariant evaluates if frozen genomic representations respond appropriately to biological transformations. It tests four models on reverse-complement consistency, coding consequence, clinical variant effects, and regulatory motif disruption. The main finding is that raw embedding similarity can give misleading conclusions. The contribution is an evaluation protocol and empirical analysis.

Critical strengths

1.The disappearance of HyenaDNA’s motif effect when control substitutions occur upstream is a useful finding. It shows how an apparent biological signal can arise from the interaction between architecture and measurement. 2.Dinucleotide matching, centered cosine similarity, local pooling, and context-length analyses make the empirical observations more informative. The discussion also acknowledges where these checks change the conclusions.

Critical weaknesses

1.The distinction from existing work is overstated. DART-Eval already evaluates frozen, mean-pooled embeddings using cosine-distance comparisons between motif-containing sequences and matched controls (Patel et al., 2024, Section 4.2). BioInvariant adds other biological axes and useful audits, but should establish more clearly what methodological or empirical insight goes beyond that earlier evaluation. 2.The biological interpretation of a failed sensitivity test remains uncertain. A near-chance result establishes that the chosen distance does not separate the groups; it does not establish that the representation lacks the relevant information. Although the limitations acknowledge it, labels such as “false invariance” still seem stronger than the evidence supports.

Questions for authors

1.How does the motif sensitivity test differ from DART-Eval’s controlled cosine-distance test, and which conclusions specifically depend on the new design? 2.Have simple sequence representations or independently validated biological representations been evaluated on the four main axes? Such results would help distinguish limitations of the models from limitations of the measurement. 3.Can variants appear in more than one matched pair? What are the resampling units for confidence intervals and significance tests, and how is dependence among variants from the same gene handled?

5 comments

We thank the reviewer for a constructive review. Each of the three questions led to a new analysis. Changes are shown in blue in the revised PDF and SI; "Table n" refers to the revised PDF.

1. Difference from DART-Eval's test (W1, Q1).

We agree that Section 2 overstated the difference, and we corrected it. The designs differ as follows. DART-Eval inserts a whole consensus motif into a neutral background and compares it with a shuffled motif at the same position. We change one base (the highest-information position of a JASPAR MA0139.1 hit) of a real, ChIP-supported CTCF site in its native 3 kb context, and compare it with the same base change outside the motif.

Because our control sits at a different position, we can test whether a conclusion depends on control placement. Three conclusions depend on this design (Table 5, SI Table 26):

  • HyenaDNA's apparent motif effect differs between upstream- and downstream-control strata: d = 0.004 (p = 0.87) with upstream controls, 0.161 with downstream controls; difference 0.157 [0.101, 0.217], reproduced in two redraws of the control positions (0.179 and 0.234). The strata contain different anchors, so we do not claim that position is the only cause.
  • DNABERT-2 is the only encoder with a significant positive effect for both control positions (0.085 / 0.060, at the default control draw). Tested directly against the strongest k-mer baseline, it is ahead with upstream controls (+0.059 [0.007, 0.110]) but not with downstream controls (−0.008 [−0.068, 0.060]). In two redraws its upstream effect stays positive (0.107, 0.132) while its downstream effect is 0.048 and 0.026, the latter not significant. The paper states the both-sides result only for the default draw.
  • With the original outside-motif controls, all single-base effects are small (|d| < 0.17).

We also added a control closer to DART-Eval's: the same base change at the lowest-information position of the same motif. No encoder ranks the core edit above it with the control on both sides of the core (DNABERT-2 d = 0.035, p = 0.018 overall, significant only with downstream controls; NT −0.043; HyenaDNA −0.082 upstream and +0.190 downstream; Caduceus −0.216). So the motif axis shows that DNABERT-2 separates a motif edit from a non-motif edit, not that any encoder reads the motif's information content (Table 5, SI Table 23).

2. Interpreting failed sensitivity tests (W2).

We agree. A near-chance distance shows only that this readout does not separate the groups. The paper no longer uses "false invariance", "false sensitivity" or the Type I-IV labels. To see how far the negative coding result depends on the readout, we tried five others on the same variants (Table 7):

Missense vs synonymous AUROC (750 / 750)DNABERT-2NTHyenaDNACaduceus
Mean over window (main)0.4890.4610.5270.446
Mean over 128 bp around the variant0.4900.4520.5000.428
Token containing the variant0.5140.5210.5230.474
Max over positions0.4600.4830.4840.479
CLS token0.5000.497——
Linear probe: direction of f(alt) − f(ref) + substitution type/context0.6020.6490.6090.656

No distance readout shows a significant missense-over-synonymous ordering. The probe does separate the classes, so your point is correct: some information is present that distance does not show. But a description of the variant that needs no model (substitution type, trinucleotide context and local 3-periodic base frequencies) reaches 0.781, and under this probe adding the encoder's direction to that description gives no incremental predictive gain (AUROC 0.718–0.780, i.e. −0.063 to −0.002). The paper states that this "does not show that the information is absent". A further probe supports that caveat: adding NT's reference embedding, which carries no variant information, to the model-free description raises AUROC by 0.083 [0.064, 0.103] (0.032 [0.014, 0.051] on same-gene pairs; no gain for the other encoders). This is predictive information in the 3 kb reference context, not a variant-specific signal (SI Table 24).

3. Simple and validated representations (Q2).

We added k-mer counts (k = 1-6, over the whole window and over 128 bp around the edit) and a GC-only scorer on all four axes (SI Table 21):

AxisBest simple baselineModelsReading
RC, 2% GC binsGC-only: 77.7%7.4–36.9%This test can be solved from GC alone, which limits the measurement
RC, exact GC count (new)GC-only: 4.8% (chance)38.5% / 44.9% / 16.8% / 9.6%Three models carry partner information beyond GC
Codingk-mers: AUROC 0.46–0.530.45–0.53Neither shows a significant missense-over-synonymous ordering
ClinicalLocal 3–5-mers: ≈ 0.55NT 0.630, HyenaDNA 0.538Only NT exceeds the baselines
MotifNone significant at both control positionsDNABERT-2 significant at bothGain over baselines only upstream

Because of the first row, exact GC-count matching is now the primary RC analysis (Table 2). For a representation with a known answer, the RC axis has two: canonical k-mer counts and the RC-equivariant Caduceus-PS (with its RC-invariant readout) both reach 100% Retrieval@1, while a naive mean over Caduceus-PS's raw states is at chance (4.3%). These validate the RC measurement only. The other three axes have no validated representation; the Limitations say so, and this gap remains.

4. Pair reuse, resampling units, dependence (Q3).

  • Reuse: We do not reuse genes or variants across pairs. In every same-gene pair set (coding 750 pairs; clinical 1,000; clinical missense-only 1,000), each gene contributes exactly one pair and no variant appears in more than one pair.
  • Resampling unit: the pair for paired statistics, the variant for unpaired AUROC, and the anchor for the RC and motif experiments.
  • Dependence within genes: in the unpaired analyses one gene can contribute several variants (at most 11 coding, 19 clinical). We added a bootstrap of the AUROC that resamples whole genes (1,260 and 1,301 genes) or chromosomes. Gene-clustered intervals are 0.96–1.02× as wide as i.i.d. intervals, chromosomeclustered ones at most 1.15×, and no significance call changes after Holm correction. Example: clinical NT 0.630 [0.606, 0.653] (SI Table 17).

Summary of changes in the revised paper (shown in blue):

  • Section 2 corrected for DART-Eval; a same-motif control at the lowest-information position of the same motif (Table 5, SI Table 23) (W1, Q1).
  • The "false invariance/sensitivity" and Type I–IV labels are removed, and the Limitations say that negative results are specific to the readouts tested (W2).
  • k-mer and GC-only baselines on all four axes (SI Table 21) and two RC positive controls, canonical k-mers and Caduceus-PS (Table 2) (Q2).
  • Pair construction is stated in the captions (Table 3: one pair per gene; SI Tables 4 and 9: each variant in at most one pair), and the AUROC is cluster-bootstrapped by gene or chromosome (SI Table 17) (Q3).
Reviewer_tLQEOpenReview ↗

Response to Rebuttal 1

Thank you; these analyses resolve my main concerns. I have two minor clarifications about interpreting the new results:

  1. For the linear-probe analysis in Table 7 and SI Table 24, are the training and test partitions gene-disjoint? If variants from the same gene can occur in both partitions, please clarify whether the reported performance could partly reflect recognition of gene or reference-context characteristics rather than transferable variant-level information.

  2. For the motif analysis, was the default upstream/downstream control draw fixed before examining the model results? Please also clarify how multiplicity is handled across the four encoders, two control positions, alternative readouts, and control-position redraws. I understand that the redraws are primarily robustness checks rather than additional independent confirmatory tests.

Reviewer_xZKTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BioInvariant evaluates frozen genomic representations through reverse-complement (RC) consistency, coding consequence, ClinVar labels, and CTCF motif disruption. It uses cosine-based comparisons and matched controls to examine whether apparent invariance or sensitivity survives changes to the evaluation protocol. At the main 3 kb setting, Caduceus-Ph has the highest raw RC similarity but the lowest true-partner retrieval among the four tested checkpoints. Separating upstream from downstream motif controls also changes the interpretation of HyenaDNA's aggregate effect. The paper supports its analysis with alternative controls, pooling, matched variant comparisons, and context-length, sample-size, and control-seed audits. Its contribution is a controlled empirical study of this representation readout.

Critical strengths
  • The paper shows that Caduceus-Ph can have high raw RC similarity while offering weaker true-partner discrimination than the other evaluated checkpoints. This changes how the checkpoint would be characterized from raw similarity alone. The comparison of corrected margins is supported by dinucleotide matching, centering, and central pooling, making the distinction between strand alignment and partner specificity more convincing.

  • The authors use the CTCF experiment to examine how the interpretation of the reported sensitivity depends on control placement. With the same base substitution inside and outside the motif, HyenaDNA has a positive aggregate effect whose interpretation changes when upstream and downstream controls are considered separately. DNABERT-2 retains small positive effects in both subsets. The authors appropriately qualify their biological conclusions in light of this result. Together with the RC comparison, this gives the paper useful empirical value.

Critical weaknesses
  • The paper's contribution seems more incremental than its framing suggests. BEND [1] already uses focal-position ref/alt cosine distances for zero-shot variant scoring, DART-Eval [2] compares embedding changes from motif and shuffled-motif insertions, and Tokenization to Transfer [3] examines mutation sensitivity and pooling. BioInvariant adds RC controls, explicit within-gene variant pairs, and single-base CTCF edits in natural genomic sequences. I would like a clearer account of what these additions establish beyond earlier evaluations.

  • Some biological claims also need more careful wording. RC preserves exact GC content, but matching within 2% GC bins can leave residual GC differences. The dinucleotide and centered-cosine checks are reassuring, though they do not fully establish a beyond-composition interpretation. The paper already acknowledges readout-specific limitations, but “false invariance” in §5 still seems too strong for the coding result. I would describe it as a lack of robust missense-over-synonymous ordering at 3 kb under this readout.

[1] Frederikke I. Marin, Felix Teufel, Marc Horlacher, Dennis Madsen, Dennis Pultz, Ole Winther, and Wouter Boomsma. BEND: Benchmarking DNA Language Models on Biologically Meaningful Tasks. International Conference on Learning Representations (ICLR), 2024. Version consulted: arXiv:2311.12570v4, 9 April 2024. https://arxiv.org/abs/2311.12570v4.

[2] Aman Patel, Arpita Singhal, Austin Wang, Anusri Pampari, Maya Kasowski, and Anshul Kundaje. DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA. arXiv preprint arXiv:2412.05430v1, 6 December 2024. https://arxiv.org/abs/2412.05430v1.

[3] Kirill Vishniakov, Karthik Viswanathan, Aleksandr Medvedev, Praveenkumar Kanithi, Marco A. F. Pimentel, Ronnie Rajan, and Shadab Khan. Tokenization to Transfer: Do Genomic Foundation Models Learn Good Representations? International Conference on Learning Representations (ICLR), 2026. Official proceedings paper.

Questions for authors
  • Could the authors clarify whether the RC conclusion concerns partner discrimination under the stated matching rules or biological specificity beyond composition more generally? Explicitly adopting the former would address the scope concern. If the stronger interpretation is intended, a GC-only retrieval baseline on the actual candidate sets would help assess the contribution of exact GC information. This could use the existing candidate information without new genomic-model inference.

  • For the final version, it would be useful to check whether the main RC and CTCF findings also hold for one newer model, such as Evo 2 [1], a eukaryotic GENERator checkpoint [2], or a pre-trained NTv3 checkpoint [3], if access and a comparable evaluation are feasible. I see this as an optional extension, not something required during rebuttal or as a condition for acceptance.

[1] Garyk Brixi, Matthew G. Durrant, Jerome Ku, et al. Genome modelling and design across all domains of life with Evo 2. Nature, 652(8112):1349–1361, 2026. DOI: 10.1038/s41586-026-10176-5.

[2] Wei Wu, Qiuyi Li, Yuanyuan Zhang, et al. GENERator: A Long-Context Generative Genomic Foundation Model. arXiv preprint arXiv:2502.07272, 2025. Version consulted: v5, revised 31 January 2026. https://arxiv.org/abs/2502.07272v5.

[3] Sam Boshar, Benjamin Evans, Ziqi Tang, et al. A foundational model for joint sequence-function multi-species modeling at scale for long-range genomic prediction. bioRxiv, 2025. DOI: 10.64898/2025.12.22.695963. Version 1.

1 comment

We thank the reviewer for the careful reading and concrete suggestions. We adopted each of them. Changes are shown in blue in the revised PDF and SI; "Table n" refers to the revised PDF.

1. Contribution relative to prior work (W1).

We agree that the contribution is narrower than our framing. The revised paper cites Vishniakov et al. (ICLR 2026), corrects Section 2, and presents the work as a controlled audit of embedding-distance readouts. What each addition establishes:

  • RC controls: high raw RC similarity does not imply partner discrimination. Caduceus-Ph has the highest raw similarity (0.9985) and the lowest Retrieval@1 (9.6%, chance 4.8%). A new positive control agrees: Caduceus-PS has raw similarity 0.997 but is at chance (4.3%) with a naive mean, and reaches 100% with its RC-invariant readout.
  • Single-base CTCF edits with position-stratified controls: for a causal model the motif effect differs between upstream- and downstream-control strata. HyenaDNA has d = 0.004 (p = 0.87) with upstream controls and 0.161 with downstream controls (difference 0.157 [0.101, 0.217]; SI Table 26). The strata contain different anchors, so we do not claim that position is the only cause. At the default control draw, only DNABERT-2 is significant for both positions (0.085 / 0.060), and all effects against outside-motif controls are small (|d| < 0.17).
  • Same-gene pairs: NT's clinical win rate falls from 0.602 to 0.545 when pairs are also matched on consequence class, and then no longer exceeds a local composition baseline (0.560).

We also state a limit: with a new control at a low-information position of the same motif, no encoder ranks the core edit higher with the control on both sides of the core (DNABERT-2: d = 0.035, p = 0.018 overall, but significant only with downstream controls; the others are null, position-dependent or negative; Table 5).

2. Scope of the RC claim (W2, Q1).

We adopt the narrower reading: partner discrimination under the stated matching rule. We also ran the GConly baseline you suggested, and it confirms your concern: it reaches Retrieval@1 = 77.7% [75.3, 80.1] inside the 2% bins (chance 3.8%), above every model (30.3 / 36.9 / 14.5 / 7.4%). We therefore matched GC count exactly (20 distractors per anchor, 488 anchors), which puts the GC-only scorer at chance:

GC-onlyDNABERT-2NTHyenaDNACaduceus
Retrieval@1 (chance 4.8%)4.8%38.5% [34.2, 42.8]44.9% [40.4, 49.4]16.8% [13.5, 20.3]9.6% [7.0, 12.3]
ΔRC+0.0364+0.0117+0.0043−0.0003

The sign of ∆RC and the model ranking are the same as in the submission. This is now the primary RC result (Table 2); the 2%-bin result sits next to the GC-only baseline in SI Table 19. We also note that canonical k-mer counts reach 100%, so Retrieval@1 "measures partner discrimination under the stated matching rule, not biological understanding".

3. Wording in Section 5 (W2).

We used your formulation and removed "false invariance", "false sensitivity" and the Type I–IV labels throughout. New readouts on the same variants support the careful wording (Table 7): local pooling, the variant token, max pooling and the CLS token all give missense-vs-synonymous AUROC between 0.43 and 0.53. So the statement is about what these readouts recover, not about what the representation contains.

4. A newer model (Q2, optional).

We could not complete a comparable evaluation of a new model family (Evo 2, GENERator, NTv3) within the discussion period; the Limitations state that our conclusions are limited to the evaluated checkpoints.

Summary of changes in the revised paper (shown in blue):

  • Section 2 cites Tokenization to Transfer (Vishniakov et al., 2026) and describes BEND and DART-Eval correctly; the contribution is stated as a controlled audit of zero-shot distance readouts (W1).
  • The RC claim is narrowed to partner discrimination under exact GC-count matching (Table 2); the 2%-bin result sits next to the GC-only baseline (SI Table 19) (W2, Q1).
  • Section 5 uses your formulation; the "false invariance/sensitivity" and Type I-IV labels are removed (W2).
  • The Limitations state that our conclusions are limited to the evaluated checkpoints (Q2).
Reviewer_tiPqOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces BioInvariant, a counterfactual evaluation framework for frozen genomic representations. BioInvariant tests two complementary properties: appropriate invariance under transformations that preserve biological meaning, and appropriate sensitivity under perturbations that disrupt biological meaning, always relative to composition-matched controls. Their evaluation includes DNABERT-2, NT, HyenaDNA, and Caduceus.

Critical strengths
  1. Interesting problem: The paper introduces an interesting problem, i.e., a counterfactual evaluation framework for frozen genomic representations. It designs 4 experiments for this.

  2. Ablation of GC content: The paper considers the effect of GC content and designs a method to control it.

Critical weaknesses
  1. Robustness: The paper only evaluates 4 models, and they are not even the SOTA models. I am wondering about the performance of other models, like Evo 2, Evo, DNABERT-S, ...

  2. The proposed metrics serve only as a limited representation or proxy for biological information. Most conclusions rely on the cosine distance between embeddings from the final layer after mean-pooling. A small distance, or a lack of discernible difference between variant classes, does not necessarily imply that the model has failed to encode relevant biological distinctions. Such information might instead reside within local token representations, the directional shifts in these representations, or be revealed through alternative "readout" methods beyond simple "unsupervised distance." While the authors acknowledge some of these limitations and explore alternative pooling strategies, the framework would be more compelling if the conclusions were further validated against other accessible readout approaches. This limitation affects the core interpretation of the negative findings regarding "coding consequences."

Questions for authors
  1. Can you test more models for this?

  2. Can you test some other embedding methods, like special token embeddings (e.g., starting 4-mer motif embedding)?

3 comments

We thank the reviewer for the review. We ran the three alternative readouts named in the second weakness and a special-token (CLS) embedding in response to Q2, all on the same variants as the paper. Changes are in blue in the revised PDF and SI, whose table numbers we use.

1. Limits of pooled cosine distance (W2).

We agree that a non-separating distance does not show that information is absent. The paper no longer uses the labels that implied this ("false invariance", "false sensitivity"), and states that the negative coding result concerns the readout. We tested each alternative you list (Table 7).

(a) Local and other unsupervised readouts. AUROC of the embedding distance:

DNABERT-2NTHyenaDNACaduceus
Coding (missense vs synonymous; 750 per class)
Mean over window (main)0.4890.461*0.5270.446*
Mean over 128 bp around variant0.4900.452*0.5000.428*
Token containing the variant0.5140.5210.5230.474
Max over positions0.460*0.4830.4840.479
CLS token0.5000.497——
Clinical (pathogenic vs benign; 1,000 per class)
Mean over window (main)0.5030.630*0.538*0.448*
Mean over 128 bp around variant0.4850.654*0.533*0.488
Token containing the variant0.4840.544*0.5170.493
Max over positions0.4840.576*0.5060.502
CLS token0.5200.584*——
(* p < 0.0125, i.e. 0.05 with Bonferroni correction over four models.)

No distance readout shows a significant missense-over-synonymous ordering, so the negative coding result is not specific to mean pooling. One thing does change: the reversed effects of NT and Caduceus disappear at the variant token, and the paper now calls them pooling-dependent. On the clinical axis NT is the only model with a signal under every readout.

(b) Direction of the shift (supervised). A linear probe on the unit-normalized direction of f(alt) − f(ref), with 5-fold cross-validation that holds out whole chromosomes:

Missense vs synonymous, AUROCDNABERT-2NTHyenaDNACaduceus
Direction alone0.5250.6000.5470.605
Direction + substitution type and context0.6020.6490.6090.656
(A) No model: substitution type and context + local 3-periodic base frequencies0.7810.7810.7810.781
(B) Direction added to (A)0.7180.7630.7680.780
Gain, (B) − (A)−0.063−0.018−0.014−0.002

Your hypothesis is partly right: the direction separates the classes although the distance does not. But the no-model description (A) reaches 0.781, and under this probe adding the direction of any encoder to it (B) gives no incremental predictive gain. The 3-periodic features are base frequencies at offsets 0, 1, 2 (mod 3) within ±60 bp of the variant: a sequence-only proxy for reading frame, not an annotation. On the clinical axis the direction adds a small significant amount for NT (+0.039 [0.023, 0.054]); none persists on same-gene missense-only pairs (SI Table 24). This does not show that the information is absent: adding NT's reference embedding (no variant information) to (A) gives +0.083 [0.064, 0.103], and +0.032 [0.014, 0.051] on same-gene pairs. That is context information, not a variant-specific signal (SI Table 24).

2. Other embeddings (Q2).

We tested the CLS (summary) token of DNABERT-2 and NT as an additional readout, together with the three other embeddings in table (a). With CLS, the coding axis is at chance (0.500 and 0.497) and NT's clinical signal remains (0.584, weaker than 0.630 with mean pooling). None of these embeddings changes the main conclusions. We are not sure this is the readout you meant: if "starting 4-mer motif embedding" refers to the first sequence token and not the summary token, we have not tested it and would be glad to add it.

3. More models (W1, Q1).

We could not complete an evaluation of a new model family within the discussion period. Evo 2 does not run on our hardware (its checkpoints need FP8 on Hopper-class GPUs or exceed our GPU memory). Evo was trained on prokaryotic and phage genomes, so human variants and CTCF sites are out of its domain. We did not complete DNABERT-S in time. The Limitations say that our conclusions are limited to the evaluated checkpoints.

Why these four: they are the encoders most often compared in earlier evaluations (BEND, DART-Eval, Tokenization to Transfer), and they span three tokenizations and three architecture types. Our findings are about how conclusions depend on controls and readouts, not about which model is best.

Summary of changes in the revised paper (shown in blue):

  • Four further distance readouts on identical variants (128 bp local mean, the token containing the variant, max pooling, CLS token; Table 7, SI Table 7); the reversed NT and Caduceus effects are called poolingdependent (W2, Q2).
  • A direction probe against a model-free baseline (Table 7, SI Table 24) (W2).
  • The "false invariance/sensitivity" labels are removed, and the negative coding result is stated as concerning the readout (W2).
  • The Limitations state that our conclusions are limited to the evaluated checkpoints (W1, Q1).

Thanks for the detailed response. These solve my most concerns. I raise my score to 2.

Reviewer_fWdwOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

BioInvariant evaluates frozen genomic embeddings using composition-matched reverse-complement controls and retrieval, coding-consequence contrasts, ClinVar labels, and CTCF motif perturbations. It compares four checkpoints using pooled cosine geometry and matched controls. Its strongest results show that raw reverse-complement similarity can obscure sequence specificity and that pooling and control position can change sensitivity conclusions (main PDF pp. 4-8, Tables 1-7). This is an empirical diagnostic framework, not a new encoder or a validated measure of biological understanding.

Critical strengths
  1. The reverse-complement analysis makes a useful distinction between high raw similarity and sequence-specific retrieval. Caduceus-Ph has raw similarity 0.9985 but Retrieval@1 of 7.4%, compared with 36.9% for Nucleotide Transformer. Dinucleotide matching, centering, and alternative pooling qualify the interpretation rather than hiding its instability (main p. 4, Tables 1-2; SI p. 7, Table 15).
  2. The sensitivity analyses reveal consequential dependence on evaluation choices. HyenaDNA's motif effect is near zero for upstream controls and positive for downstream controls (main p. 8, Table 7). Caduceus's clinical AUROC is 0.488 in the local-pooling analysis, versus 0.448 in the main analysis (SI p. 3, Table 7; main p. 7, Table 6). However, the samples also differ in size, so this comparison does not isolate pooling alone. The authors appropriately avoid a universal model ranking.
Critical weaknesses
  1. The novelty comparison understates the closest peer-reviewed prior work. Section 2 describes DART-Eval mainly as predicting external labels. DART-Eval already uses frozen final-layer mean embeddings, compares motif-versus-control cosine distances, and evaluates reference-versus-alternate cosine distance. BioInvariant adds composition-matched RC retrieval, natural-locus single-base motif disruptions with position-stratified controls, and a combined set of axes. These are useful distinctions, but their incremental value needs a more direct explanation (main p. 3, Section 2; Patel et al., DART-Eval, NeurIPS 2024, Sections 3, 4.2, 4.5 and Appendix E.2).
  2. The benchmark is not yet sufficiently specified or calibrated to support its biological interpretation. The supplied PDFs omit the ClinVar release, ENCODE accessions and cell type, PWM identifier and hit threshold, and pair-reuse rules. They also do not show simple sequence baselines on the four primary axes or a biologically informed positive control. Thus, failure of pooled cosine distance to order variants does not establish absence of biological information. Success is also not shown to exclude tokenization or local sequence effects. The authors partly acknowledge these limits on main pp. 6 and 9. However, the labels "false invariance" and "false sensitivity" go beyond the demonstrated calibration. This matters most for the small motif effects (main pp. 3, 8-9, Table 8 and Section 5; SI pp. 1-8).
Questions for authors
  1. What precisely is new relative to DART-Eval's zero-shot motif and variant embedding tests? Can the existing experiments isolate what the new controls reveal that those tests would miss?
  2. Please provide the data releases, accessions, PWM parameters, cohort identifiers, and pair-reuse policy. What statistic is resampled in SI Table 17? Are the 1,000 same-gene missense pairs drawn from a different pool than the 346/119 missense variants in SI Table 11? Do the whole-window and local-pooling conclusions hold on identical variants?
  3. Do the main conclusions survive base-weighted pooling and simple composition/k-mer controls on the primary axes? Does a biologically informed positive control behave as expected?

I would be open to a higher rating if the response establishes the added value over DART-Eval and addresses the benchmark calibration concerns. Clear cohort definitions and evidence supporting the biological interpretation would be particularly helpful.

3 comments

We thank the reviewer for stating clearly what would change the assessment. We ran new analyses for each of the three conditions. Changes are shown in blue in the revised PDF and SI; "Table n" refers to the revised PDF.

  1. Added value over DART-Eval (W1, Q1).

We agree that DART-Eval already uses cosine distance between frozen mean-pooled embeddings. Section 2 now says so, and we no longer present that readout as new.

What we add is an audit of that score. When the distance is larger, is it biology, or something simpler such as GC content, the gene, or the position of the edit? Each readout is paired with a control for the simpler explanation. In two cases the control changes the conclusion on the same data:

  • Reverse complement. Caduceus-Ph has the highest raw RC similarity (0.9985) but the lowest partner retrieval (Retrieval@1 9.6%, vs 44.9% for NT; chance 4.8%; 488 anchors, Table 2). DART-Eval has no RC test.
  • Position of the control. HyenaDNA's pooled motif effect (d = 0.083, significant) is d = 0.004 (p = 0.87) with upstream controls and d = 0.161 with downstream controls; the difference is 0.157 [0.101, 0.217] (Table 5, SI Table 26). DART-Eval holds position fixed by design, so it does not measure this. The two strata contain different anchors, so we do not claim that position is the only cause.

We also added a control inside the same motif: the same substitution at the lowest-information position of the same PWM hit (4,385 anchors; paired d, positive = core edit moves the embedding more):

DNABERT-2NTHyenaDNACaduceus
All anchors+0.035 (p = 0.018)−0.043 (p = 3e-4)−0.020 (p = 0.20)−0.216 (p < 1e-4)
Control upstream / downstream of core+0.029 / +0.065+0.003 / −0.088−0.082 / +0.190−0.265 / −0.071

No encoder meets the rule we apply to motif effects (positive and significant with the control on either side of the core). DNABERT-2's positive overall effect is significant only with downstream controls (upstream p = 0.10), and the other three are negative, null or change sign with position. The paper reports this as a limitation of the motif axis (Table 5, SI Table 23).

  1. Specification (W2, Q2).

Everything requested is now in Table 1 and SI Section 1:

  • ClinVar: GRCh38 SNVs, release 2026-08-22, no review-status filter. Restricting to ≥1-star variants changes every AUROC by ≤0.004 (SI Table 22).
  • CTCF: UCSC/ENCODE TF ChIP-seq clusters (2020-01-27), pooled over 114 cell types.
  • PWM: JASPAR MA0139.1 (19 bp), pseudocount 0.8, best hit on either strand with score ≥0.75 × the maximum.
  • Pairs: one pair per gene. SI Table 25 lists the data sources and versions.

On the three specific questions:

  • SI Table 17 resampled the mean VES of one class, which is not the statistic our conclusions rest on. It now cluster-bootstraps the AUROC itself, by gene or chromosome. Intervals are at most 1.15× as wide as i.i.d. intervals, and no significance call changes after Holm correction.
  • Pools. The 346/119 missense variants of SI Table 11 belong to the 1,000-per-class AUROC sample. The 1,000 same-gene missense pairs are sampled separately; 10 of their 2,000 variants overlap.
  • Pooling on identical variants (750 per class, SI Table 7):
Clinical AUROCDNABERT-2NTHyenaDNACaduceus
Whole window0.5150.6400.5390.447 (p = 4e-4)
128 bp around the variant0.4960.6620.5350.488 (p = 0.42)

Caduceus's below-chance ordering disappears under local pooling, which confirms a pooling effect on identical variants. DNABERT-2 and NT keep their conclusions; HyenaDNA's small positive effect is no longer significant after correction.

  1. Calibration (W2, Q3). In short: under exact GC-count matching the model ranking is preserved and the RC claim is narrowed to partner discrimination; NT's clinical signal exceeds every composition baseline; the motif evidence is limited; and only the RC axis has a positive control.

(a) Simple baselines (k-mer counts, k = 1--6, and a GC-only scorer; SI Table 21).

  • RC. The concern was justified: in our original 2% GC bins a GC-only scorer reaches Retrieval@1 = 77.7% [75.3, 80.1] (chance 3.8%), above every model (30.3 / 36.9 / 14.5 / 7.4%). With 20 distractors that have exactly the anchor's GC count, it is at chance, and this is now the primary result (Table 2):
GC-onlyDNABERT-2NTHyenaDNACaduceus
Retrieval@1 (chance 4.8%)4.8%38.5% [34.2, 42.8]44.9% [40.4, 49.4]16.8% [13.5, 20.3]9.6% [7.0, 12.3]
  • Coding. No baseline shows a significant missense-over-synonymous ordering (AUROC 0.459–0.533), and neither do the models.
  • Clinical. NT (AUROC 0.630) exceeds the strongest baseline (local 4-mer, 0.551) by 0.079 [0.046, 0.113]; HyenaDNA does not (−0.013 [−0.050, 0.023]; SI Table 26). On same-gene missense-only pairs, NT is not distinguishable from a local 1-mer baseline.
  • Motif. Composition is matched exactly by design. DNABERT-2 exceeds every baseline with upstream controls (+0.059 [0.007, 0.110]) but not with downstream controls (−0.008 [−0.068, 0.060]), so we call this evidence limited.

(b) Base-weighted pooling. Weighting each DNABERT-2 token by the bases it covers leaves the results almost unchanged: Retrieval@1 37.5% vs 38.5%, motif d = 0.071 vs 0.073.

(c) Positive control. On the RC axis, two representations with a known answer reach 100% Retrieval@1: canonical k-mer counts, and Caduceus-PS (RC-equivariant by construction) with its RC-invariant readout. A naive mean over Caduceus-PS's raw states is at chance (4.3%) although its raw RC similarity is 0.997. So on the RC axis the measurement recovers a known RC property, and raw similarity alone does not. This validates the RC measurement only: the coding, clinical and motif axes have no validated positive control, so their biological calibration remains an unresolved limitation, and the Limitations now say that negative results there are specific to the readouts tested. Five further readouts on identical variants also show no significant missense-over-synonymous ordering (Table 7).

  1. Labels (W2).

We agree. The paper no longer uses "false invariance", "false sensitivity" or the Type I–IV labels. Results are described directly, for example "a lack of robust missense-over-synonymous ordering at 3 kb under pooled cosine distance". The motif effects against outside-motif controls (|d| < 0.17, below the conventional 0.2 for a small effect) are reported as effect sizes without a label.

Summary of changes in the revised paper (shown in blue):

  • Section 2 now describes DART-Eval, BEND and Tokenization to Transfer correctly, and the paper is presented as a controlled audit of existing zero-shot readouts (W1, Q1).
  • New Table 1 specifies cohorts, sample sizes, controls and tests; data provenance is in SI Table 25 (W2, Q2).
  • Exact GC-count matching is the primary RC analysis (Table 2). GC-only and k-mer baselines cover all four axes (SI Tables 19, 21), with direct model-versus-baseline tests (SI Table 26) (Q3).
  • Positive controls on the RC axis (canonical k-mers, Caduceus-PS; Table 2), a same-motif control (Table 5, SI Table 23), and a Limitations statement that the other axes have no validated positive control (Q3).
  • A cluster bootstrap of the AUROC by gene or chromosome (SI Table 17).
  • The "false invariance/sensitivity" and Type I--IV labels are removed (W2).

Thank you for the detailed response and the additional experiments. They have addressed most of my concerns, and I have raised my score accordingly. Good luck!

Reviewer_C58xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BioInvariant to evaluate biological invariance and sensitivity in genomic foundation models. It tests four models on reverse-complement consistency, coding consequence, clinical variants, and motif disruption. The results show that model rankings depend on the biological task and evaluation controls.

Critical strengths
  1. This paper studies an important question beyond downstream accuracy. Table 1 shows that raw RC similarity and corrected similarity give different model rankings.

  2. The robustness analyses are useful. For example, HyenaDNA's motif sensitivity disappears with upstream controls. Caduceus's clinical reversal also disappears under local pooling.

Critical weaknesses
  1. The GC controls are not sufficient. RC preserves exact GC content, but the negative controls only match within 2% GC bins. A model encoding only GC content could still achieve positive corrected RC scores and high retrieval accuracy. The additional checks in Table 2 do not fully rule this out. This weakens the main RC-specificity claim.

  2. The novelty is unclear. DART-Eval already evaluates motif perturbations using embedding distances. BEND also uses embedding cosine distance for variant effects. It is unclear what BioInvariant adds beyond these existing evaluations.

Questions for authors
  1. Do the RC results hold under exact GC-count matching? How does a GC-only baseline perform?

  2. What new findings does BioInvariant provide beyond DART-Eval and BEND?

  3. Why should missense variants necessarily produce larger cosine distances? Could coding consequence instead be encoded in the direction of the embedding change?

3 comments

We thank the reviewer for a focused review. We ran both analyses requested in Q1 and the direction analysis suggested in Q3. Changes are in blue in the revised PDF and SI, whose table numbers we use.

1. GC control (W1, Q1).

We agree that matching within 2% GC bins was not sufficient. In our original 2% GC bins (chance 3.8%), a scorer that uses only the GC count reaches Retrieval@1 = 77.7% [75.3, 80.1], above every model (30.3 / 36.9 / 14.5 / 7.4%, in the column order below). So that test did not establish specificity beyond GC.

We re-ran the experiment with 20 fresh distractors per anchor that have exactly the anchor's GC count (488 anchors). The GC-only scorer is then at chance:

ChanceGC-onlyDNABERT-2NTHyenaDNACaduceus
Retrieval@14.8%4.8%38.5% [34.2, 42.8]44.9% [40.4, 49.4]16.8% [13.5, 20.3]9.6% [7.0, 12.3]
ΔRC+0.0364+0.0117+0.0043−0.0003

Under exact GC-count matching, three models stay far above chance, and the model ranking and the signs of ΔRC are the same as in the submission. Other candidate sets agree (SI Table 20). The exact-count analysis is now the primary RC result (Table 2), and the claim is narrowed to "partner discrimination under exact GC-count matching".

Composition baselines for the other three axes are in SI Table 21.

2. What is new beyond DART-Eval and BEND (W2, Q2).

We agree that Section 2 misdescribed both, and we corrected it. The paper is now presented as a controlled audit of embedding-distance scores. The matched controls give three findings that appear in neither benchmark:

  • High raw RC similarity does not mean the model recognizes the partner. Caduceus-Ph has the highest raw RC similarity (0.9985) and the lowest partner retrieval (9.6%, chance 4.8%). A new positive control shows the same from the other side: the RC-equivariant Caduceus-PS has raw similarity 0.997 but is at chance (4.3%) with a naive mean, and reaches 100% only with its RC-invariant readout (Table 2). Neither benchmark has an RC test.
  • For a causal model, the motif effect differs between upstream- and downstream-control strata. HyenaDNA: d = 0.004 (p = 0.87) with upstream controls, 0.161 with downstream controls; difference 0.157 [0.101, 0.217] (Table 5, SI Table 26). The strata contain different anchors, so we do not claim that position is the only cause. DART-Eval holds position fixed, so its design avoids this; ours covers comparisons across positions, as in mutagenesis scans.
  • The clinical signal shrinks when pairs are matched more tightly. NT's win rate is 0.602 on samegene pairs and 0.545 on same-gene missense-only pairs, where it no longer exceeds a local base-composition baseline (0.560; difference −0.015 [−0.057, 0.029]).

We also report a limit: with a new control at a low-information position of the same CTCF motif, no encoder ranks the core edit higher with the control on both sides of the core (DNABERT-2 does so only with downstream controls; Table 5). The practical message: an embedding-distance score should be reported with a composition-matched control, the position of the edit relative to its control, and a simple k-mer baseline.

3. Magnitude vs direction (Q3).

This is not required, because a representation can separate the two classes by the direction of the shift and not by its magnitude. The paper now presents "missense > synonymous in distance" as a hypothesis about this readout that we test, not as a requirement on a good representation.

We tested your alternative with a linear probe on the unit-normalized direction of f(alt) − f(ref), with 5-fold cross-validation that holds out whole chromosomes (750 / 750 variants):

Missense vs synonymous, AUROCDNABERT-2NTHyenaDNACaduceus
Distance (main readout)0.4890.4610.5270.446
Direction alone0.5250.6000.5470.605
Direction + substitution type and context0.6020.6490.6090.656
(A) No model: substitution type and context + local 3-periodic base frequencies0.7810.7810.7810.781
(B) Direction added to (A)0.7180.7630.7680.780
Gain, (B) − (A)−0.063−0.018−0.014−0.002

Your intuition is partly right: the direction supports linear separation of the classes while the distance does not. But the no-model description (A) reaches 0.781, and under this probe adding the direction of any encoder to it (B) gives no incremental predictive gain. The 3-periodic features are base frequencies at offsets 0, 1, 2 (mod 3) within ±60 bp of the variant: a sequence-only proxy for reading frame, not an annotation. This does not show that the information is absent: adding NT's reference embedding (no variant information) to (A) gives +0.083 [0.064, 0.103], and +0.032 [0.014, 0.051] on same-gene pairs. That is context information, not a variant-specific signal (Table 7, SI Table 24).

Summary of changes in the revised paper (shown in blue):

  • Exact GC-count matching is the primary RC analysis, and the RC claim is narrowed to partner discrimination (Table 2, SI Table 20); the 2%-bin result is reported next to the GC-only baseline (SI Table 19) (W1, Q1).
  • Composition baselines for the coding, clinical and motif axes (SI Table 21).
  • Section 2 corrected for DART-Eval and BEND; new RC positive control Caduceus-PS (Table 2); motif results by control position and a same-motif control (Table 5; SI Tables 23, 26) (W2, Q2).
  • "Missense > synonymous in distance" is stated as a tested hypothesis about the readout; the direction probe is in Table 7 and SI Table 24 (Q3).

thanks for the revision and clarifications. i have also read the other reviews and author responses. i remain unconvinced that the paper meets the bar for acceptance. I believe the paper would benefit from further revision and polishing, so i will keep my score.

Reviewer_tPavOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents BioInvariant, a counterfactual evaluation framework for genomic foundation models, specifically the sequence representations these models output. The two properties that BioInvariant evaluates are (a) the invariance of sequence representations to the transformations that preserve biological meaning and (a) sensitivity to biologically meaningful perturbations. These are complementary properties, both of which are highly desirable in a globally utile biological sequence representation.

BioInvariant evaluates foundation models against composition-matched controls across four axes - reverse-complement consistency, coding consequence, clinical variant signal and regulatory motif disruption. Four models, namely DNABERT-2, Nucleotide Transformer, HyenaDNA and Caduceus, were evaluated under this framework with the results demonstrating that there is no consistent winner on all four axes of BioInvariant. This serves to round out the evaluation on downstream benchmarks that are typically conducted in works on genomic foundation models.

Critical strengths

The greatest strength of this paper is the problem that it seeks to address. Genomic foundation models have begun to be applied to a variety of biological use cases, with purported improvements constantly being made. Thus far, most of the evaluations of these models have been conducted on benchmarks, comprising a variety of downstream tasks with known ground truth. However, the choice of benchmarks and evaluations is neither consistent nor complete, as it is difficult to ascertain whether the benchmark performance translates to the general utility of these models.

BioInvariant seeks to address this by directly evaluating the sequence representations induced by the genomic foundation models in terms of how they behave with various transformations. The key assumption here is that a "good" representation should be sensitive to only biologically meaningful transformations. If this were indeed true, it follows that for downstream tasks related to the tested biological semantics, such "good" representations should generalize well.

Critical weaknesses

There are several weaknesses, largely in terms of writing and presentation of the manuscript that will require extensive revision to correct, and one potential technical issue. I have laid them out in a point-wise fashion here:

Writing:

The early parts of the paper are well-written, with the introduction clearly outlining the motivation and goals of this work. However, the quality of writing deteriorates sharply once we reach portions of the manuscript directly describing the presented framework, BioInvariant. Specifically, thew "Results" section is somehow both lacking in details and overfull with numerical metrics. Some examples of this are:

  1. In Section 3.1.1, how are windows held out? What are the 2% GC-wide bins?
  2. Why is Retrieval@1 a suitable metric here? And how is this computed?
  3. VES is defined for the first time on Page 5 despite being first mentioned on Page 3.
  4. The psignp_{sign} values in Tables 3 and 4 don't appear to have been discussed in the text.
  5. The discussion in Section 3.4.2 is not anchored in any of metrics in that it isn't clear why those values of d support the inferences made here.
  6. The discussion about background sensitivity on Page 8 is devoid of any of the numerical results.
  7. What is the PWM referred to in Section 3.4.1?
  8. The results in Tables 3 and 4 don't appear to match - shouldn't Δˉm=mean VES missense−mean VES synonymous\bar{\Delta}_m = \text{mean VES missense} - \text{mean VES synonymous}?

Technical:

There are two possible technical issues in this paper.

  1. The assumption that a "good" representation should be sensitive to only biologically meaningful transformations is one way of defining goodness. However, this does not directly translate into generalizability. Firstly, it may not always be clear what representation-level qualities are indicative of generalization across downstream tasks; reverse complementarity naturally makes sense but the other axes seem to be inspired by some downstream tasks goodness on those axes would be utile for. Secondly, it is possible that representation-level sensitivity to seemingly biologically irrelevant transformations could actually be desirable. In fact, it may even be likely because our knowledge of biologically meaningful transformations is limited. Rather, it might be more desirable for known biological transformations to be more identifiable or interpretable at the representation level.
  2. BioInvariant evaluates representations in terms of mean embeddings across the sequence length. This loses out on capturing the variation in representations across the sequence position. For instance, a sequence embedding with the mean embedding fm(X)f_m(X) at each position would be evaluated the same way as a sequence embedding that is aggregated as fm(X)f_m(X).
Questions for authors
  • Are composition-matched controls only used for the reverse complement test? If yes, why? If not, what were the controls used for the other axes?
  • The discussion of results at the end of the introduction may be premature as a reader doesn't have an understanding of the proposed framework at this point in the manuscript.
4 comments

We thank the reviewer for the detailed list. We agree that the Results section gave many numbers without the definitions needed to read them. Each item below led to a specific change, shown in blue in the revised PDF and SI. "Table n" refers to the revised PDF; the motif table (submitted Table 7) is now Table 5.

W1. Held-out windows and GC bins. No model is trained in this paper. "Held out" only means that chromosomes 1–16 were used to choose the window length (3,000 bp), and every reported RC number comes from chromosomes 17–22 (500 windows in the original analysis). A "2% GC bin" groups windows whose GC fraction differs by at most 0.02; controls were drawn from the anchor's bin. Because a scorer that uses only the GC count reaches 77.7% Retrieval@1 inside such bins, the revised paper uses a stricter control: 20 fresh distractors from the same chromosomes with exactly the anchor's GC count (488 eligible anchors). Under this control the model ranking is preserved, and the claim is narrowed to partner discrimination under exact GC-count matching.

W2. Retrieval@1. For each window X we rank RC(X) against the reverse complements of the matched windows, by cosine similarity to f(X). It is a hit if the true partner ranks first; chance is 4.8% with 20 distractors. We need it because raw RC similarity is high whenever all embeddings are close to each other, which is the case at 3 kb. Retrieval@1 is rank-based, so it is comparable across models:

DNABERT-2NTHyenaDNACaduceus
Retrieval@1 (chance 4.8%)38.5%44.9%16.8%9.6%

Caduceus has the highest raw similarity (0.9985) and the lowest Retrieval@1. The metric also recovers a known answer: the RC-equivariant Caduceus-PS reaches 100%. The formula is now in Section 3.1.1, before the results.

W3. VES. Thank you for catching this. VES = 1 − cos(f(ref), f(alt)) is now defined in a "Metrics" paragraph in Section 3, with the other metrics, before any experiment.

W4. Values in Tables 3 and 4. Section 3.2.2 now explains every column. Raw distances are on each model's own scale (about 1e-8 to 1e-4) and cannot be compared across models, so our conclusions rest on two scalefree statistics: the win rate (Table 3, sign test) and the AUROC (Table 4, Mann–Whitney test), with Holm correction over the four models. The reading: no model shows a missense-over-synonymous effect. NT and Caduceus show a significant effect in the opposite direction in Table 4 (AUROC 0.461 and 0.446; Holm-adjusted p = 0.026 and 0.001); in Table 3 only Caduceus remains significant. DNABERT-2 and HyenaDNA are not distinguishable from chance.

W8. Why Tables 3 and 4 need not match. They use independently drawn samples: Table 4 uses 750 + 750 independently drawn variants, Table 3 uses 750 same-gene pairs. The mean paired difference therefore need not equal the difference of class means, but the conclusions agree.

A second apparent mismatch is within each table: DNABERT-2's mean VES is higher for missense, yet its AUROC (0.489) and win rate (0.496) are below 0.5. This is because VES is strongly right-skewed (skewness 6.6 and 8.1), so a few large values drive the mean. The class medians are almost identical (5.35e-5 for missense, 5.36e-5 for synonymous), and the 8 pairs with the largest absolute differences (1% of 750) account for 26% of the summed absolute difference. That is why we rely on rank statistics. The paper now says both things.

W5. How d supports the inferences. The section now states the quantity and a decision rule first. For each CTCF site, d (paired Cohen's d) compares the embedding change after a one-base edit in the motif with the change after the same edit outside the motif; d > 0 means the motif edit moves the embedding more. A motif effect counts as position-robust only if d is positive and significant both when the control is upstream and when it is downstream, because a causal model propagates an edit only to later positions. Each inference is tied to its numbers (Table 5):

  • HyenaDNA: d = 0.004 (p = 0.87) upstream and 0.161 (p < 1e-4) downstream; the difference is 0.157 [0.101, 0.217] (SI Table 26). Its pooled value (0.083) mixes two different effects, so we do not count it as motif sensitivity.
  • DNABERT-2: d = 0.085 and 0.060, both significant; the only encoder that meets the rule. Its downstream effect is not significant in one of two redraws of the controls, so we do not claim robustness across draws.
  • All effects against outside-motif controls are below 0.2, the conventional threshold for a small effect.

W6. Background sensitivity. New Table 6 gives the numbers (4,100 CTCF anchors; D_bg(m) = distance after m base changes outside the CTCF footprint):

DNABERT-2NTHyenaDNACaduceus
Share of anchors where one background change exceeds the motif disruption32%41%46%50%
Median D_bg(4) / D_motif2.554.364.9512.8
Median D_bg(16) / D_motif9.726.338.1181

For every model, the median ratio D_bg(4) / D_motif exceeds 1: at the median anchor, four background changes move the embedding more than one disruption of the motif core. This is why raw sensitivity must be compared with a matched control.

W7. The PWM. It is the CTCF matrix MA0139.1 from JASPAR (19 bp), converted to log-odds with pseudocount 0.8 against a uniform background. We scan both strands of each ChIP-seq peak and keep the best hit if it scores at least 0.75 × the maximum. The "motif disruption" changes the base at the highest-information position to the least likely base (Section 3.4.1).

Technical point 1: what "good" means, and identifiability. We agree with both parts. The paper no longer claims that passing these tests implies downstream generalization, and no longer treats sensitivity to "irrelevant" changes as an error: synonymous variants, for example, can affect splicing, mRNA stability and translation (Sauna & Kimchi-Sarfaty, 2011). Each axis is now a test of one stated hypothesis about how embedding distance is already used as a zero-shot score, always against a matched control.

We tested your identifiability proposal directly, with a linear probe on the direction of f(alt) − f(ref), holding out whole chromosomes. Missense and synonymous variants are identifiable to some degree: AUROC 0.52–0.61 from the direction alone and 0.60–0.66 with substitution type and context, although the distance is near chance (0.43–0.53). But a description of the variant that needs no model (substitution type, trinucleotide context and local 3-periodic base frequencies) reaches 0.781, and under this probe we find no incremental predictive gain from adding the direction of any encoder to it (AUROC 0.718–0.780); this does not show that the information is absent. A further probe supports that caveat: adding NT's reference embedding, which carries no variant information, to the model-free description raises AUROC by 0.083 [0.064, 0.103] (0.032 [0.014, 0.051] on samegene pairs; no gain for the other encoders). This is predictive information in the 3 kb reference context, not a variant-specific signal. The paper reports this next to the distance results (Table 7, SI Table 24).

Technical point 2: mean pooling. We agree that mean pooling hides positional variation. It is our default because it is the most common way to get one vector per sequence and did best in a recent benchmark (Feng et al., 2025). We added three readouts that do not average over the window (the token containing the variant, max pooling, the CLS token), on the same variants; the full table is Table 7:

  • Coding (750 variants per class): every readout gives AUROC 0.43–0.53, and none shows a significant missense-over-synonymous ordering.
  • Clinical (1,000 variants per class): NT is significant under every readout (AUROC 0.544–0.654). HyenaDNA is significant only under the two mean readouts (0.538, 0.533).

So the main conclusions do not depend on mean pooling. Your concern is justified for a secondary finding: the reversed effects of NT and Caduceus disappear at the variant token, and the paper now calls them pooling-dependent.

Q1. Controls on each axis. Every axis has a matched control, but what is matched differs by axis, so the abstract now says "matched controls" instead of "composition-matched controls" (new Table 1):

AxisMatched control
Reverse complementReverse complements of 20 other windows with exactly the same GC count
CodingA synonymous variant in the same gene, with one pair per gene
ClinicalA benign variant in the same gene. In a second set, the control also matched the consequence class (missense vs missense).
MotifThe same base change outside the motif, preserving the base-count change. The new control uses the same base change at the lowest-information position of the same motif.

RC distractors match the GC count exactly, not the full base or k-mer composition. Motif and control edits induce the same base-count change. Coding pairs are matched on gene; clinical pairs on gene and, in the missense-only set, on consequence class. Composition on the variant axes is assessed separately with k-mer baselines (SI Table 21).

Q2. Results preview in the Introduction. We agree. The framework overview now comes first, and the preview is three plain sentences without metric names or numbers.

Summary of changes in the revised paper (shown in blue):

  • A "Metrics" paragraph in Section 3 defines VES and the other metrics before any result; Retrieval@1 is defined in Section 3.1.1; exact GC-count matching is the primary RC analysis (Table 2) (W1–W3).
  • Section 3.2.2 explains every column of Tables 3 and 4 and why they need not match (W4, W8).
  • The motif section states the quantity and decision rule first; background sensitivity is in new Table 6; the PWM (JASPAR MA0139.1) and its scan are specified (W5–W7).
  • No claim that passing the tests implies generalization; each axis is a stated hypothesis; the direction probe is in Table 7 and SI Table 24 (T1).
  • Three readouts that do not average over the window (Table 7); the reversed NT and Caduceus effects are called pooling-dependent (T2).
  • Table 1 lists the matched control of each axis, the abstract says "matched controls", and the Introduction preview is rewritten (Q1, Q2).
Reviewer_tPavOpenReview ↗

Detailed rebuttal

First off, i'd like to thank the authors for choosing to take all the reviewer comments and feedback in their stride, and subsequently act upon them in a bid to improve their manuscript.

This has led to a fairly extensive revision of the submitted manuscript, which is decidedly clearer and makes more of a focused case. In light of that, I'm willing to raise my score by a point.

However, I would like to point out that such a substantial revision raises two concerns -- one specific to this manuscript, and the other more general.

  • My understanding is that the rebuttal/revision phase is meant to rough out any rough edges and clarify some questions about the work, and not an opportunity to submit a dramatically different revision. This is because (a) it is unrealistic reviewers to reread such revisions as that is tantamount to effectively reviewing a brand-new paper and (b) it suggests that the submission required significant work to be publication-ready to start with.
  • The updates to the manuscript have resulted in more focused claims, but they cast doubts on the exact utility of the proposed framework. For instance, the effect of pooling in the results suggests that it is a dimension that should be a lever/setting in BioInvariant.

Comments on the paper

1 comment

Hi reviewers,

Thank you for your reviewing service. The authors have provided fairly extensive rebuttals. Please confirm that you have read them, state which of your concerns are or are not addressed, and whether that changes your recommendations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.