acceptodds
Under review as a conference paper at ICLR 2027

What Your Benchmark Cannot See: Non-Identifiability of Backbone Symmetry Under Invariant Readouts

Abstract

Standard benchmark scores can make a structural collapse look like an ordinary performance drop. We show that this ambiguity is fundamental: under a fixed invariant readout, the score alone cannot reveal whether directional information is absent from the backbone or erased by the readout, making backbone symmetry non-identifiable. For targets that swap under reverse complementation, this mismatch forces orbit-paired accuracy to exactly zero on orbit-closed free orbits, independent of loss, optimisation, decoding, or readout capacity, even while Matthews correlation coefficient (MCC), accuracy, and macro-F1 remain non-zero. We characterise how channel symmetrisation and position-symmetric pooling jointly remove directional information at readout, and test the predictions with parameter-matched readouts across three DNA foundation models on a new direction-aware promoter benchmark. Linear probes recover directionality from Caduceus representations; a symmetry-matched equivariant readout restores positive orbit-paired accuracy and exact equivariant agreement across all three models; and a matched invariant control exhibits no structural obstruction. Thus the problem is not symmetry itself, but using a readout symmetry that the target does not obey—and benchmark scores alone need not expose the mistake.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
DNA foundation models, reverse-complement equivariance, invariant readout mismatch, benchmark non-identifiability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.60 over 5 reviews, above 10% of the others.
Reviewer_dxh1OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies, mainly in the context of DNA sequence models, the effect of different symmetry constraints on the model's readouts. The authors build on the idea that under a fixed invariant readout, tasks whose targets are direction-sensitive show worsened performance purely due to the symmetry constraint on the readout. They build an empirical case by comparing invariant, free and equivariant readouts across three DNA foundations models, as well as synthetic scenarios and an MNIST demonstration.

Critical strengths
  • The mathematics are correct, as far as I could verify
  • The experiments address the research question adequately and are reported over multiple seeds
Critical weaknesses
  • My main critique for this paper is on novelty, significance and presentation. The theoretical contributions are presented in an overly complicated manner relative to the thin contribution they carry. A substantial body of definitions, theorems, corollaries and lemmas are introduced to argument results that in my opinion are direct and simple. Theorem 2, for instance, can be stated in one sentence: "Since a flip-invariant classifier gives the same output for a given input x and its flip Rx, and they carry different labels each, then the model is wrong on at least one of them". The remaining bounds and Theorem 3 follow directly from this. I appreciate the formality maintained throughout the paper, and it does make the paper easy to follow, but the theory could be simplified considerably and presented in more accessible prose. Once compressed/simplified, it perhaps becomes more clear that the theoretical contribution, while correct, is limited. I have not done a thorough literature research for results shown here, but I would not be surprised if several of these results have been derived before or are simply considered standard. Due to the latter point, I would consider the contribution modest.
  • Some claims do not entirely follow from the experimental results and should be relaxed or nuanced. For instance, abstract mentions "...a target-matched equivariant readout restores paired success and improves aggregate performance.", yet for frozen NTv3 and Caduceus in Table 2, HfreeH_{free} ties or even improves over HequiH_{equi}.
Questions for authors
  • In essence, the paper argues that the readout's symmetry must match how the labels transform, which I agree. However, this is standard knowledge in the group equivariance/invariance scientific community. My question is: what should a practicioner do differently after reading this paper? What the authors argue is true, but I struggle to see a clear value out of it. One way to strengthen the paper, besides a narrative perhaps a bit more focused around this question, would be to identify cases where invariant readouts are applied to direction-sensitive targets in practice: public checkpoints, published studies, benchmarks, or pipelines. For instance, the authors show that the public Caduceus head is equivalent to the invariant HinvH_{inv}. Are there published works or benchmarks that apply this head (or another invariant one) to direction-sensitive targets? I am not convinced that practicioners familiar with these models and with equivariance would overlook this, but such instances could strengthen the narrative.
  • I agree that standard benchmark metrics can hide symmetry-induced readout errors. In practice, though, prior knowledge usually identifies the cause. Consider an MNIST classifier on rotated 6s and 9s with a fully SO(2) invariant readout. Here, digit classification suffers, and we know why: SO(2) invariance conflates the two labels (a more appropriate prior would be partial invariance over a small interval on SO(2) like ±45º\pm 45º) . Your section 2.8 does something similar: it says that to find out when invariant readouts are harmful, we can compute ΔG\Delta_G prior to training, which depends only on the label statistics. Given this, why would a score-based audit (with OPA_A) be needed e.g. for a readout that is invariant by construction? Simply checking the readout's construction against the task definition, as usual, gives it.
Reviewer_LrhwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies a general evaluation problem. When an input undergoes a symmetry transformation, its label may also be exchanged. For example, reverse complementing (RC) a DNA sequence swaps the transcription direction from rightward to leftward. If the readout is invariant, it cannot distinguish these labels. Benchmark scores cannot reveal this issue, making it impossible to tell whether the backbone has learned the relevant information. The authors theoretically prove this for any finite group acting through a permutation representation, using DNA foundation models as an example. DNA benchmarks often attach the same lightweight readout to different backbones. For validation, the authors construct a direction-aware promoter benchmark and test three backbones. Invariant readouts all have zero orbit-paired accuracy (OPAA), while equivariant readouts resolve the problem. Changing only the readout also changes backbone rankings. Two additional experiments using synthetic data and MNIST show that this phenomenon is not limited to DNA.

Critical strengths
  1. They identifies an issue that common readout-based evaluation practice overlooks. When the input symmetry swaps the labels, an invariant readout can never predict both a sequence and its flipped version correctly. While conventional metrics stay non-zero. Benchmark scores therefore cannot tell whether the backbone encodes the relevant information. The experiments results support both sides of this claim.

  2. They provides a new direction-aware promoter benchmark, built from K562 GRO-cap data, in which each sequence carries a two-channel direction label.

Critical weaknesses
  1. The core insight is not such new. Mallet and Vert (2021) already characterize reverse-complement equivariant layers whose channels can change sign under reverse complement, and they apply equivariant networks to base-pair resolution profile prediction. The accuracy ceiling for invariant classifiers also follows from Wang et al. (2023).

2.The paper does not show that the issue affects any existing downstream task. The collapse is demonstrated only on a benchmark the authors constructed for this purpose, and existing tasks with strand-dependent labels, such as BEND gene finding, are not evaluated, so it remains unclear whether any reported benchmark result is actually misleading.

Mallet, V. and Vert, J.-P. Reverse-complement equivariant networks for DNA sequences. In Advances in Neural Information Processing Systems, volume 34, pp. 13511–13523, 2021.

Marin, F. I., Teufel, F., Horlacher, M., Madsen, D., Pultz, D., Winther, O., and Boomsma, W. BEND: Benchmarking DNA language models on biologically meaningful tasks. In The Twelfth International Conference on Learning Representations, 2024.

Wang, D., Zhu, X., Park, J. Y., Jia, M., Su, G., Platt, R., and Walters, R. A general theory of correct, incorrect, and extrinsic equivariance. In Advances in Neural Information Processing Systems, volume 36, pp. 40006–40029, 2023.

Questions for authors
  1. BEND’s gene finding task already has strand-specific labels that swap under reverse complement. Does the collapse happen there, for example with Caduceus?

  2. All three backbones are small. Do the results, especially the gain of the equivariant readout and the ranking change, hold for larger models such as NT-500M or NT-2.5B? Rerunning the experiments on larger models is not necessary, but could you discuss how you expect the results to change with scale.

  3. Was the free readout trained with reverse-complement augmentation that swaps the labels?

Reviewer_5DpMOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that when a downstream readout is invariant under an input transformation but the target transforms non-trivially, the benchmark score cannot show whether the backbone lacks the information or the readout erased it. The test case is reverse-complement (RC) symmetry in DNA. Direction-resolved promoter labels (a→, a←) swap under RC.

Critical strengths
  1. clear definition and motivation of the paper
  2. the diagram of figure 1 clearly defines the invariance and equvariance.
  3. different model choice to highlight the behavior did not correlate with the model
Critical weaknesses
  1. the conclusion is expected and obvious. I agree the experiments are solid to support the claim. But I have strong concerns can this be a conference paper instead of a workshop. There is almost no techinical insights or novelty in this paper.

  2. The authors built a task where invariance must fail, and then invariance failed. Do any existing DNA benchmarks actually have direction-dependent labels evaluated with an invariant head? Or in other words, is this really important in terms of biology??

  3. The test set has 1,776 sequences, with only about 441 free orbit pairs. Only the SEM over 5 seeds is reported, with no confidence intervals for test-set sampling.

Questions for authors

I still did not get the importance of this work. I think this is so straightforward and can not be a paper.

Reviewer_AxbqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether RC-invariant readouts can suppress directional information encoded by the DNA foundation model and cause benchmark evaluation errors. The author constructs a dataset using K562 GRO-cap data and does a four-state directional promoter prediction task. Within the task, the author compares three different readouts: free, invariant and equivariant across Caduceus, HyenaDNA and NTv3. The paper theoretically shows that invariant predictions cannot correctly classify the directional input and further use linear probes to demonstrate that directional information remains accessible in Caduceus representation. The experiment result shows equivariant readouts improve prediction performance. Additional experiments on MNIST and H3K27me are provided to validate the proposed idea.

Critical strengths

The paper addresses an important issue in foundation model evaluation, and the analysis of directional information preservation is potentially useful. However, the practical usage and significance of the invariant readout limitation is not sufficiently analyzed. Several experimental validations and comparisons are needed to support the paper claims.

Critical weaknesses

Major Concerns:

  1. The practical significance of the proposed benchmark analysis is not sufficiently justified. The paper shows that an RC-invariant readout fails on a newly constructed RC-equivariant directional promoter prediction task. However, this failure is expected because the readout and the target follow different experiment assumptions. Although the experiments show how readout design can affect backbone evaluation, it is still unclear whether such mismatches actually occur in DNA foundation model benchmarks or have a misleading conclusion. The authors should provide evidence from prior works that the identified issue exists in practice and affects model evaluation. Otherwise, the work mainly illustrates the expected consequences of applying an inappropriate constraint rather than identifying a previously unrecognized limitation of existing benchmarking practices.
  2. The experiments lack comparisons with existing RC-equivariant baselines. There exist several prior works such as Zhou et al. (MLCB 2022) that already investigated RC conjoining and output alignment for strand-specific predictions. The equivariant readout used in this paper follows a similar approach. The authors provide a three readouts theoretical analysis and additional evaluations, but it remains unclear what new methodological insights or practical improvement are offered beyond existing works. The authors should clarify the novelty of their approach and provide comparisons with related methods to better support its additional contributions.
  3. The benchmark lacks sufficient label-sensitivity and per-class error analysis. The four-state labels are constructed using strand-specific GRO-cap TSS annotations. However, the distinction between these categories may depend on TSS detection thresholds and annotation criteria rather than clear biological differences. The authors should test the sensitivity of the labels to these criteria and identify potential reverse-strand signals below the detection threshold. In addition, Table A1 only reports rightward and leftward recall, without a complete four-class confusion matrix or separate results for background and two-side promoters. A more detailed per-class error analysis would help clarify which promoter categories benefit from the equivariant readout.

Minor Concerns:

  1. The readout design introduction is not clear in the method section. HyenaDNA and NTv3 require two forward passes for the original and RC sequence whereas Caduceus uses native RC-related features from a single forward pass. The authors should clarify this distinction in the main text and explain how the logits are obtained for each Backbone.
  2. The definition of the Invariant Readout is inconsistent in the Position-aware experiments. Equation 5 defines the Invariant Readout as averaging the predictions of the Original and RC Sequences but Appendix A.3 uses the same name for a Channel-symmetric Projection that is no longer an invariant classifier under Position-aware Pooling. The authors should use different terminology to avoid confusion.
  3. The reason why adds 256-N-padding on both sides is unclear. Each 512-bp DNA Sequence is padded with 256 N Tokens on each side to obtain a 1,024-nt input. The authors should explain why this specific configuration is necessary instead of directly using the original 512-bp sequence, and whether the additional padding influences feature extraction and Pooling Operations.
Questions for authors

The paper identifies a meaningful issue in evaluating DNA foundation models through fixed readouts, and its theoretical analysis and representation-level experiments support the central claim. However, the current experiments do not sufficiently demonstrate that this problem affects existing downstream benchmarks in practice. The authors should include comparisons with established RC-equivariant baselines and provide additional biological validation to support the paper's claims.

Reviewer_tgEgOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines invariant readouts for tasks whose labels change under an input transformation. Its main example is strand-specific promoter prediction, where reverse complementation exchanges the directional labels. An invariant predictor must give the same prediction to both inputs, making it impossible to classify both members of a directional pair correctly. The authors formalize this observation through orbit-paired accuracy (OPAA), an accuracy ceiling, and an OPAA-based non-identifiability result. They construct a directional promoter benchmark and compare invariant, unconstrained, and equivariant readouts on Caduceus[1], HyenaDNA, and NTv3. Probing and pooling interventions examine directional information in Caduceus representations, with synthetic, MNIST-derived, and RC-invariant genomic tasks serving as controls.

Critical strengths
  1. I found the Caduceus representation analysis useful. Figure 2 and Table A2 show that directional information remains accessible through antisymmetric channels or position-aware features, even when globally averaged symmetric features cannot distinguish the two orientations. This supports the specific claim that failure through one readout does not establish that the backbone lacks the relevant information.
  2. The experiments include useful checks on the proposed mechanism. The authors report decision-level and logit-level transformation audits, compare readouts with matched parameter counts, and include an RC-invariant genomic task where the same constraint does not force paired failure. These controls help verify the implementation and clarify why the effect depends on how the target transforms.
Critical weaknesses
  1. My main concern is that the paper does not establish a substantial theoretical or methodological advance over existing work. The zero-OPAA result follows immediately from the assumptions: an invariant predictor assigns the same label to both members of a pair, while their ground-truth labels differ. The invariant-readout part of Theorem 3 then follows because this metric is zero for every backbone. Formalizing these observations makes the assumptions explicit, but does not by itself provide a substantial new account of backbone non-identifiability. The result concerns a particular functional that is forced to be constant; Appendix B.1.6 acknowledges that aggregate risks can still differ across backbones.

The accuracy bound also has a direct precedent. Equation (12) closely matches the orbit-majority bound in Wang et al., ICLR 2023, Proposition 4.1 [2], and the more general treatment in Wang et al., NeurIPS 2023 [3]. The bound 1-\pi_A/2 is the corresponding two-element-orbit case. The proposed output-aligned repair is similarly close to Zhou et al. [4]: their Figure 3 explicitly explains why strand-specific predictions must be transformed into a common orientation before averaging, and Section 4.1 discusses directional promoter elements and early representation merging. The manuscript cites relevant prior work, but needs to explain more clearly what it adds beyond these specific results. In my assessment, the new diagnostics and application-specific experiments do not yet justify the strength of the central theoretical claims.

  1. The empirical contribution is narrower than the benchmark framing suggests, and much of the reported failure is imposed by construction. For HyenaDNA and NTv3, the invariant arm explicitly averages predictions on x and Rx. Its zero OPAA is therefore guaranteed regardless of the backbone. These experiments check that the imposed constraint behaves as expected; they do not show that the models’ ordinary evaluation pipelines inadvertently discard directional information. The Caduceus two-axis analysis is more informative, but depends on its particular RC-equivariant representation structure. Mean pooling alone does not impose this restriction on a generic sequence model, whose contextual features can already encode order.

The paper also does not identify and audit a specific existing benchmark task where the implemented readout conflicts with the target’s RC transformation rule. It instead demonstrates the failure on a newly constructed directional task. This is a valid stress test, but it does not establish that an existing benchmark comparison was misleading. In particular, the original Caduceus downstream setup explicitly assumed RC-invariant labels. The manuscript needs to distinguish the risks of transferring that readout to a different task from evidence that its original use was inappropriate.

Finally, the paper lacks a direct comparison with an unconstrained model trained using correctly label-transformed RC augmentation and made equivariant only at inference through output-aligned averaging. This is a particularly relevant baseline given [4], and would help separate the benefit of the proposed training intervention from an established post-hoc procedure. Computational cost should also be compared, since matching parameter counts does not match the cost of processing both orientations.

Overall, my main concern is the contribution itself, rather than a small number of missing experiments or presentation issues. The paper contains new experiments, particularly the Caduceus probes, but I do not find that they establish a sufficiently substantive advance beyond existing theory and methods. Confirming the forced failure on more backbones does not resolve this concern.

References: [1] Schiff Y, Kao C H, Gokaslan A, et al. Caduceus: Bi-directional equivariant long-range dna sequence modeling[J]. Proceedings of machine learning research, 2024, 235: 43632. https://pmc.ncbi.nlm.nih.gov/articles/PMC12189541/pdf/nihms-2005895.pdf [2] Wang et al. The Surprising Effectiveness of Equivariant Models in Domains with Latent Symmetry. ICLR 2023, https://openreview.net/pdf?id=P4MUGRM4Acu [3] Wang et al. A General Theory of Correct, Incorrect, and Extrinsic Equivariance. NeurIPS 2023, https://proceedings.neurips.cc/paper_files/paper/2023/file/7dc7793c89b93887e126a86f22ef63c6-Paper-Conference.pdf [4] Zhou, Shrikumar, and Kundaje. Towards a Better Understanding of Reverse-Complement Equivariance for Deep Learning Models in Genomics. PMLR 165, 2022, https://proceedings.mlr.press/v165/zhou22a/zhou22a.pdf

Questions for authors
  1. What is the main new insight beyond the orbit-wise bounds in [2,3] and the strand-specific output alignment in [4]? Please distinguish results that follow directly from these ideas from genuinely additional theoretical or empirical findings.
  2. Is there an existing published benchmark task whose actual implementation uses an invariant readout despite requiring a nontrivial target action? If so, please identify the task, implementation, and reported conclusion affected by the mismatch. If the intended contribution is instead a new stress test, what evidence establishes its practical importance beyond demonstrating the expected consequence of the imposed constraint?
  3. What RC augmentation is used to train the unconstrained baseline? How does the trained equivariant readout compare with that baseline after target-aligned post-hoc averaging at inference, including computational cost?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.