- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies, mainly in the context of DNA sequence models, the effect of different symmetry constraints on the model's readouts. The authors build on the idea that under a fixed invariant readout, tasks whose targets are direction-sensitive show worsened performance purely due to the symmetry constraint on the readout. They build an empirical case by comparing invariant, free and equivariant readouts across three DNA foundations models, as well as synthetic scenarios and an MNIST demonstration.
Critical strengths
- The mathematics are correct, as far as I could verify
- The experiments address the research question adequately and are reported over multiple seeds
Critical weaknesses
- My main critique for this paper is on novelty, significance and presentation. The theoretical contributions are presented in an overly complicated manner relative to the thin contribution they carry. A substantial body of definitions, theorems, corollaries and lemmas are introduced to argument results that in my opinion are direct and simple. Theorem 2, for instance, can be stated in one sentence: "Since a flip-invariant classifier gives the same output for a given input x and its flip Rx, and they carry different labels each, then the model is wrong on at least one of them". The remaining bounds and Theorem 3 follow directly from this. I appreciate the formality maintained throughout the paper, and it does make the paper easy to follow, but the theory could be simplified considerably and presented in more accessible prose. Once compressed/simplified, it perhaps becomes more clear that the theoretical contribution, while correct, is limited. I have not done a thorough literature research for results shown here, but I would not be surprised if several of these results have been derived before or are simply considered standard. Due to the latter point, I would consider the contribution modest.
- Some claims do not entirely follow from the experimental results and should be relaxed or nuanced. For instance, abstract mentions "...a target-matched equivariant readout restores paired success and improves aggregate performance.", yet for frozen NTv3 and Caduceus in Table 2, ties or even improves over .
Questions for authors
- In essence, the paper argues that the readout's symmetry must match how the labels transform, which I agree. However, this is standard knowledge in the group equivariance/invariance scientific community. My question is: what should a practicioner do differently after reading this paper? What the authors argue is true, but I struggle to see a clear value out of it. One way to strengthen the paper, besides a narrative perhaps a bit more focused around this question, would be to identify cases where invariant readouts are applied to direction-sensitive targets in practice: public checkpoints, published studies, benchmarks, or pipelines. For instance, the authors show that the public Caduceus head is equivalent to the invariant . Are there published works or benchmarks that apply this head (or another invariant one) to direction-sensitive targets? I am not convinced that practicioners familiar with these models and with equivariance would overlook this, but such instances could strengthen the narrative.
- I agree that standard benchmark metrics can hide symmetry-induced readout errors. In practice, though, prior knowledge usually identifies the cause. Consider an MNIST classifier on rotated 6s and 9s with a fully SO(2) invariant readout. Here, digit classification suffers, and we know why: SO(2) invariance conflates the two labels (a more appropriate prior would be partial invariance over a small interval on SO(2) like ) . Your section 2.8 does something similar: it says that to find out when invariant readouts are harmful, we can compute prior to training, which depends only on the label statistics. Given this, why would a score-based audit (with OPA_A) be needed e.g. for a readout that is invariant by construction? Simply checking the readout's construction against the task definition, as usual, gives it.