acceptodds
Under review as a conference paper at ICLR 2027

SymSelect: Recovering Symmetry Alignment When Validity Depends on the Sample

Abstract

Candidate transformations can provide useful structural supervision even when their validity varies across samples. When pair validity is unobserved, aligning every candidate can impose invalid constraints, while discarding transformed pairs loses the benefit carried by valid relations. We study how much of the benefit of supervised valid-pair alignment can be recovered without observing pair validity. We introduce SymSelect, which ranks candidate relations by representation similarity and aligns only the most similar pairs. On MNS and I-RAVEN, SymSelect recovers 77%–92% of the accuracy gain from supervised valid-pair alignment across the evaluated backbones. Our analysis shows that selecting mostly valid pairs is not sufficient to inherit the oracle benefit: the retained valid subset must also preserve a useful alignment update. Local and finite-batch results separate invalid-pair contamination from bias among the retained valid pairs and characterize the effects of retention and contribution variability. Matched controls show that aggregate validity enrichment alone does not explain SymSelect's recovery, while checkpoint measurements on I-RAVEN show increasing batch precision, decreasing selected-valid gradient discrepancy, and positive local alignment effects. These results characterize selective symmetry alignment when transformation validity is available structurally but hidden at the level of individual applications.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
symmetry learning, representation alignment, adaptive invariance, selective training, structured reasoning, visual reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_R5SWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper proposes SymSelect, a framework that selects high-similarity transformed pairs for representation alignment when transformation validity is unobserved. This paper's theoretical analysis separates invalid-pair contamination from selection bias among valid pairs and provides conditions for inheriting the oracle’s local improvement. Experiments on MNS and I-RAVEN show that SymSelect recovers 77%–92% of the accuracy gain from supervised valid-pair alignment without using validity labels.

Critical strengths
  1. The theoretical analysis is sound under the stated assumptions, with clear proofs that distinguish invalid-pair contamination from selection bias among valid pairs.

  2. The method is simple to implement and achieves strong empirical results, recovering most of the supervised alignment gain without access to pair-validity labels.

Critical weaknesses
  1. The introduction and problem formulation would benefit from clearer motivating examples and a more explicit distinction between candidate transformations and valid alignment relations, making the setting more accessible to readers unfamiliar with the area.

  2. Although the paper provides quantitative gradient diagnostics and validity-matched controls, it does not establish whether Theorem 1’s sufficient condition holds in practice or assess the tightness and predictive value of Theorem 2’s finite-batch bound.

  3. Although the paper includes oracle-based augmentation and several selection controls, it lacks comparisons with learned augmentation or input-dependent invariance methods, such as Augerino [1] and InstaAug [2], leaving the practical advantage of similarity-based selection over existing adaptive approaches unclear.

[1] Benton, Gregory, et al. "Learning invariances in neural networks from training data." Advances in neural information processing systems 33 (2020): 17605-17616.

[2] Miao, Ning, et al. "Learning instance-specific augmentations by capturing local invariances." arXiv preprint arXiv:2206.00051 (2022).

Questions for authors

1, How sensitive is SymSelect to the choice of similarity measure? Could the author evaluated alternative metrics for pair selection and alignment?

2, How sensitive is SymSelect to the choice of k? Could the author provide some ablation study on k?

3 comments

Dear Reviewer R5SW,

We would like to respectfully offer a reminder and invite you to join the further discussion. We believe that most concerns have been reasonably addressed or substantially resolved. If you have a different view, we can give further clarification. Alternatively, we would also welcome you to raise the score or adjust the confidence.

We would also like to share important additional experimental results that may change your view of the paper. Following your suggestions, we strengthened the empirical analysis of selective alignment, including gradient diagnostics, validity-matched controls, and sensitivity to the selection rate.

  • Direct evidence of beneficial alignment updates. On I-RAVEN, every measured selected alignment step improves the diagnostic objective. Across six checkpoints, the relative discrepancy between selected valid and oracle gradients decreases from 0.93 to 0.51, supporting the local improvement mechanism studied in Theorem 1.
  • Benefits beyond validity enrichment. With matched aggregate selected validity composition, SymSelect outperforms randomized selection by 1.66 accuracy points on MNS and 1.76 points on I-RAVEN. This demonstrates that the identity of selected pairs matters beyond their validity rates.
  • Robustness to selection rate. Our new ablation shows that selective alignment performs best at relatively small retention rates. On I-RAVEN, selecting 9.4% of candidate pairs achieves 78.97% accuracy, compared with 75.79% for Align All. On MNS, selecting 4.7% achieves 39.46%, compared with 37.54% for Align All.

We have also clarified the motivating examples, theoretical scope, and relationship to adaptive augmentation methods.

Full experimental details are provided in our responses and revision. We look forward to your reply and will be happy to answer any further questions.

Thanks for the detailed explanations and experiments, my concerns are addressed. I have raised the score to 3.

Thank you!

Reviewer_Yfm7OpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper considers a known family of transformations whose validity depends on the input. SymSelect ranks pairs of original and transformed examples by cosine similarity and aligns only the top kk pairs, alongside supervised classification on the original examples. The analysis separates two sources of discrepancy from oracle alignment: invalid pair contamination and bias within the retained valid subset. Experiments on MNS and I-RAVEN report recovery of 77% to 92% of the oracle accuracy gain. Additional controls and gradient measurements examine whether validity precision alone explains the improvement.

Critical strengths
  1. I found the distinction between validity and usefulness interesting. Even a subset containing only valid pairs need not preserve the oracle update. The counterexample makes this point clearly, and separating contamination from retained valid bias is useful for understanding relation selection.

  2. The controlled datasets allow exact validity measurements, making the oracle comparison meaningful. The matched composition controls are also well motivated. The improvements over aligning all pairs across four MNS backbones, together with the accuracy gains on valid transformed examples, support the usefulness of selection in these settings. I appreciate that the authors also measure local gradient effects rather than relying only on final accuracy.

Critical weaknesses
  1. My main concern is the gap between the theoretical characterization and the proposed selection rule. Theorem 1 bounds the discrepancy between the selected and oracle gradients and then applies smoothness. This is a reasonable argument, but it assumes a positive oracle margin and requires the selection error to be sufficiently small. It does not establish when cosine ranking makes this error small. Similarly, the finite batch bound applies to any subset with the stated composition.

In particular, ϵsel\epsilon_{\mathrm{sel}} contains an important part of the question we would like to understand: why should the most similar valid pairs give a representative or useful update? Proposition 2 shows that this can fail even with perfect validity ranking. Table 9 is helpful evidence, but it follows one I-RAVEN trajectory and uses a particular diagnostic JJ constructed through masking. I am therefore not yet convinced that the analysis explains the algorithm’s advantage beyond giving conditions under which a selected update would work.

I would also appreciate a clearer discussion of gradient preserving subset selection, especially CRAIG (Mirzasoleiman et al., 2020) and the already cited GRAD-MATCH (Killamsetty et al., 2021). Hidden validity is a different setting, but preserving a reference gradient through subset selection is an established idea. The additional theoretical insight specific to this setting needs to be clearer.

  1. The empirical operating range remains unclear. The results support the method on the evaluated tasks, but the transformations are manually specified and validity is available from the task rules, then (I assume) deliberately hidden. This is useful for controlled experiments. Still, I would like a stronger explanation of the practical setting this models, where candidate transformations are known but checking their validity is difficult.

The choice k/B=6/64k/B=6/64 is important here. Appendix C shows that a batch is very likely to contain enough valid pairs to fill the selected set, but this does not explain the choice of retention rate. A sweep over k/Bk/B would directly test the proposed tradeoff between contamination and retained valid bias. The candidate family size experiment addresses a different question. It would be particularly informative to see where the method stops helping, or where similarity favors valid but unhelpful relations. Higher contribution variability allows both unusually good and unusually bad subsets; the bound does not tell us which one the selector finds.

  1. Finally, no supplementary code is provided. The appendix gives useful implementation details, but code is particularly important for reproducing the validity checks, matched controls and gradient diagnostics.

References:

Mirzasoleiman, B., Bilmes, J., and Leskovec, J. Coresets for Data-efficient Training of Machine Learning Models. ICML, 2020.

Killamsetty, K., S, D., Ramakrishnan, G., De, A., and Iyer, R. GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training. ICML, 2021.

Questions for authors
  1. Regarding Weakness 1, is the relationship between similarity and local contribution measured within the valid pair population, across seeds and both datasets? This would help separate usefulness from validity enrichment.

  2. Regarding Weakness 2, are retention rate sweeps or empirical failure cases available? Please point to the relevant results if already included.

  3. Is the sufficient condition in Theorem 1 verified at any checkpoint, or are the measurements intended as qualitative support for the decomposition?

  4. Can the implementation be shared? Also, does the projection head include batch normalization? It appears in Appendix B’s description but is omitted in Appendix C.

Reviewer_yFpXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The authors demonstrate improved accuracy on MNS and I-RAVEN using the novel method of SymSelect.

Critical strengths

Strength 1: the authors appear to recover improved performance using selected metrics.

Critical weaknesses

Weakness 1: the methodology is not fully described mathematically. Current literature dealing with symmetry is careful to define quantities precisely, from a mathematical point of view, which the paper in its current form does not fully espouse. For example, a candidate transformation famility T\mathcal{T} is not defined in section 3.1. The quantity TT is also not defined when used in equation 1, unless the reader is meant to look to the caption of figure 1 for the general relationship that T∈TT \in \mathcal{T}, whatever T\mathcal{T} is. The loss LCEL_{\text{CE}} is also not defined, leaving the reader to guess the authors' conventions. In fact, the paper would even benefit from a precise definition of a visual reasoning dataset, if one can be given, which would help solidify the precise nature of T\mathcal{T}.

Weakness 2: the precise problem addressed by the paper is not immediately clear. For example, in section 2, the authors assert that their "question" is different (lines 091-095) than work which seeks to learn symmetry: they "fix the candidate family and study hidden application validity relative to a supervised valid pair alignment oracle." No further description is offered (such as the meaning of hidden application validity), robbing what would otherwise be a key statement of important meaning for the reader. My opinion is that the paper would benefit from a more meticulous problem description.

Questions for authors

Question 1: What does "original" mean in the context of figure 2? Are we seeing the baseline outperform the novel method?

Question 2: What is the connection between your work and symmetry discovery? If your question is different, what makes the literature relevant?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.