acceptodds
Withdrawn from ICLR 2027

Invariance Is Sample-Conditional: Risk-Controlled Consistency for Transformation–Example Pairs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A transformation family is not uniformly label preserving: the same crop, occlusion, or attribute edit can preserve one example and remove the defining evidence from another. Always-on consistency therefore injects incorrect invariance gradients for semantically invalid example–transformation pairs. We formulate invariance validity as a sample–transformation relation and introduce SELECTIVE INVARIANCE, a compatibility gate whose threshold controls false acceptance on a held-out audit split. Accepted pairs receive consistency supervision; rejected pairs retain the clean supervised objective without being forced invariant. On CIFAR-100, SELECTIVE INVARIANCE reaches 79.1% clean accuracy and 69.4% shift accuracy with 4.8% false acceptance and an invalid invariance gradient ratio (IIGR) of 8.6 at 74.6% coverage. AROID reaches 77.8%, 67.2%, 15.4%, and 21.0 at 75.2% coverage, while a matched random gate reaches 77.0%, 65.7%, 22.7%, and 28.9 at the same 74.6% coverage. The matched-coverage control shows that semantic compatibility, rather than reduced regularization mass alone, drives the improvement.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Selective Invariance, Consistency Regularization, Sample-Conditional Invariance, Data Augmentation, Robust Representation Learning, Distribution Shift, Risk Control, Semantic Compatibility

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_vGhUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies a simple but important issue with consistency regularization: the same transformation may preserve the label for one image and remove the relevant evidence from another. The authors propose Selective Invariance, which predicts whether an image–transformation pair is valid and applies consistency loss only to accepted pairs. A threshold is selected on a held-out audit set under a false-acceptance risk constraint. Rejected pairs still receive clean supervised training, and the compatibility head is removed at inference.

The paper reports results on CIFAR-100, including clean and shifted accuracy, false acceptance, coverage, and an invalid invariance gradient ratio (IIGR). The reported improvements are encouraging, but several central parts of the method and evaluation are not specified well enough to assess the conclusions.

Critical strengths
  1. The motivation is clear. The crop and color-edit examples in the Introduction explain why transformation validity should depend on the individual image. The method addresses this issue directly by gating the consistency term while keeping the clean supervised objective (Section 3, Eq. 4).

  2. The matched random gate is a useful control. In Table 1, both the proposed method and the random gate have 74.6% coverage. Selective Invariance reports gains of 2.1 percentage points in clean accuracy and 3.7 points in shifted accuracy. This comparison provides more information than a comparison against always-on consistency alone.

  3. The paper is explicit about some important limits. Sections 3 and 7 state that the risk control depends on the audit protocol and pair distribution, and does not automatically transfer to new tasks or transformation mixtures. Reporting false acceptance and IIGR alongside accuracy also helps readers examine the intended behavior of the gate.

Critical weaknesses
  1. The technical and experimental presentation is very short and leaves out essential details. The introduction, method, and all experimental results fit into three pages. The paper does not explain how validity labels are obtained, how many pairs are audited, or how ambiguous transformed images are handled. The backbone, training settings, transformation distribution, and shift suite are also unspecified. Table 1 contains only aggregate CIFAR-100 results, without seed counts, variability estimates, or component ablations. Section 7 acknowledges several of these missing details, and the Reproducibility Statement says configurations and raw logs are not included. These omissions make it difficult to assess or reproduce the reported gains.

  2. The central risk-control claim is not specified well enough to check. Section 3 introduces UCB(R_FA) ≤ α without defining R_FA precisely or specifying the bound estimator, confidence level, or α. It also does not explain how the confidence bound accounts for choosing the threshold on the same audit set. Without these details, I cannot tell what the reported 4.8% false acceptance means or whether the claimed risk constraint is satisfied.

  3. The matched-coverage comparison does not fully support the mechanism claim. Section 6 treats equal coverage as matched regularization mass, and the Conclusion says the random-gate comparison rules out reduced consistency as the explanation. However, accepting the same proportion of pairs does not imply equal consistency-loss or gradient magnitude. Eq. 5 is a gradient decomposition, and lower IIGR alone does not establish why accuracy improves. The results suggest that pair selection helps, but the stronger explanation needs additional controls.

Questions for authors
  1. How are validity labels obtained? Please specify the audit size, labeling protocol, treatment of ambiguous images, and the separation between compatibility training, threshold calibration, and final evaluation.

  2. What is the exact definition of R_FA? Which UCB estimator, confidence level, α, and minimum coverage are used? How is threshold selection handled statistically, and what happens if no threshold satisfies both constraints?

  3. What backbone, transformations, consistency distance D, and training settings are used? What exactly is included in the shift evaluation, and over how many runs are the results averaged?

  4. Can the authors report consistency-loss magnitude and accepted-gradient magnitude for the proposed and random gates, or add a control that matches effective regularization strength? This would help assess the claim in Section 6.

  5. How much do the uncertainty penalty and risk-controlled threshold contribute beyond the compatibility score alone? Comparisons at matched coverage, together with risk–coverage curves, would help answer this.

Reviewer_LsRzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how to make a model's output invariant to transformations of the same input. In computer vision, models are often trained with a consistency loss that pushes the model to produce the same output across different transformed versions of a given input. For example, semi-supervised learning pushes for consistent predictions on augmented unlabeled data, while self-supervised learning maximizes the similarity between two augmented views of the same image. This paper focuses on the supervised setting, where a consistency loss is added on top of the classification loss. It asks one main question: can we avoid enforcing consistency on destructive transformations, that is, those that remove the information that defines the label, and apply it only to pairs where the label is preserved?

To answer this, the paper proposes selective invariance, which trains a module to estimate whether a pair is valid and uses this estimate to gate the consistency loss. A pair (an input and its transformed version) is valid if the transformed input keeps the original label, and invalid otherwise. The module sits on top of the classifier and takes as input the detached features of the input and of its transformed version, their difference, and the transformation parameters. From these, it predicts the probability that the pair is valid, together with an uncertainty estimate. During training, the consistency loss is applied only to pairs whose predicted validity, minus a scaled uncertainty, exceeds a threshold chosen on a held-out audit set of labeled pairs. Note that while rejected pairs are excluded from the consistency loss, they still contribute to the standard supervised loss.

The method is evaluated on CIFAR-100, measuring accuracy on the clean test set and on an (unspecified) shifted test set. It improves over always-on consistency on both clean and shifted accuracy (+1.9 / +3.5). Against a random gate that accepts the same fraction of pairs (74.6%), the gains are similar (+2.1 / +3.7), and false acceptance drops from 22.7% to 4.8%. The authors use this comparison to argue that the gain comes from skipping invalid pairs, not simply from applying less consistency. For reference.

Critical strengths

The paper tackles a real problem: augmentation validity varies across examples, and forcing invariance on pairs where the transformation destroys the label injects wrong gradients.

Critical weaknesses

The paper has many critical weaknesses:


Quality and presentation: the writing is not clear, and it takes multiple passes to understand many parts of the paper. Many key details are also missing:

  • how the model and the validity module are trained
  • how shifted accuracy is measured
  • which backbone and transformations are used
  • the hyperparameter values
  • the size of the audit set and how it is labeled

On top of this, no code, configurations or experimental details are provided


Limited novelty and missing related work: handling transformations that may not preserve the label is a well-studied problem. Prior work has, for example:

  • softened the target for strong transformations (Soft Augmentation [1], AutoLabel [2]);
  • learned per-input augmentation policies (InstaAug, AdaAug, and MetaAugment [3], which already learns per-input and per-transformation weights);
  • designed augmentations that keep the label intact (KeepAugment, TeachAugment).

Gating the consistency loss is not new either. FixMatch [4] already applies consistency only to confident predictions, and its follow-ups (FlexMatch, FreeMatch, SoftMatch [5]) study how to choose this threshold, trading off how many pairs are used against how many are wrong. Finally, picking a threshold on held-out data so that the error stays below a target with high probability is the idea behind selective classification with guaranteed risk [6]. None of these works are cited, and the paper has only six references in total.


Very limited and small-scale results: with about three pages of main content, the paper reads more like a small experimental report than an ICLR paper. Results are on CIFAR-100 only, with no repeated runs or variance. It is therefore unclear whether the method works at larger scale or with more complex transformations. The gains may even be specific to small datasets: with more data, wrong transformations may matter less and simply act as a form of regularization. Key baselines are also missing, most importantly a simple gate that needs no labels, such as using the classifier's own confidence.


The risk control is not justified: the false-acceptance guarantee holds only if the gate stays fixed after the threshold is set. But the gate relies on features from a model that keeps changing during training, so the guarantee may not hold for long. Since risk control is the main contribution, this is a major issue.


The results do not fully support the main claims: several numbers in Table 1 do not add up. Always-on consistency has 28.6% false acceptance, so about 71% of pairs are valid, yet the oracle gate which accepts only valid pairs reports 76.1% coverage. The random gate keeps a random subset of pairs, so its share of invalid pairs should stay around 29%, yet it reports 22.7%. It is also unclear how coverage and false acceptance are computed for AROID, which never accepts or rejects pairs. Finally, the paper never explains how it knows whether each transformed training image still shows its original class, which the oracle gate requires. Since the random gate is the main evidence for the paper's claim, these issues need to be explained.


References

  • [1] Soft Augmentation for Image Classification, CVPR 2023.
  • [2] What Are Effective Labels for Augmented Data? Improving Calibration and Robustness with AutoLabel, arXiv 2023.
  • [3] MetaAugment: Sample-Aware Data Augmentation Policy Learning, AAAI 2021.
  • [4] FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence, NeurIPS 2020.
  • [5] SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learning, ICLR 2023.
  • [6] Selective Classification for Deep Neural Networks, NeurIPS 2017.
Questions for authors

Clarifications

  • How is the validity module trained, and on which labels? Are these labels separate from the audit set used to choose the threshold?
  • Is the threshold chosen once, or updated during training? What values are used for the target false-acceptance level and the other hyperparameters?
  • How big is the audit set, and how were its labels obtained? How do you know, for the training pairs, whether each transformed image still shows its original class?
  • How exactly are false acceptance and coverage computed?
  • What is the shifted test set (CIFAR-100-C?)
  • which backbone and transformations are used?
  • is the classification loss also applied to the transformed images? If not, I think the comparison with the augmentation baselines is not fair, since they train on augmented images.
  • How does the method compare to a simple gate that needs no labels, such as the classifier's own confidence on the transformed image, at the same coverage?
  • What are the results over several seeds, with mean and standard deviation?

Suggestions

  • I think the paper needs a full rewrite. It should explain the method and experimental setup in much more detail, and make clear what is new compared to prior work.
  • As mentioned in the weaknesses, I think the method should be compared with the missing related work, such as FixMatch-style confidence gating and per-input augmentation methods like InstaAug or MetaAugment.
  • The method should be tested on larger datasets, at least ImageNet. This problem may mostly matter at small scale, and with more data, destructive transformations might be less harmful and simply act as a form of regularization.
  • It would also be good to show that the idea works beyond image classification, for example on other vision tasks or other domains such as text or audio, where transformations can also change the label.
Reviewer_x5yfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This incomplete paper proposes Selective Invariance: A compatibility model scores each (example, transformation) pair on whether the transformation preserves the label. A threshold, calibrated on a held-out audit set to maximize coverage subject to an upper confidence bound on false-acceptance risk, decides which pairs receive the consistency loss. Rejected pairs keep only the supervised loss. On CIFAR-100, a single table reports 79.1% clean and 69.4% shift accuracy, against 77.0% and 65.7% for a random gate that accepts the same fraction of pairs.

Critical strengths
  1. A clearly stated question: whether an augmentation is valid depends on the example, and the paper separates a transformation’s usefulness from its validity.
Critical weaknesses
  1. The submission is incomplete. The main text is about 3.5 pages, and the experiments consist of one paragraph and one table. The backbone, training setup, hyperparameters, shift benchmark, seeds and variances are all missing. The claims can be neither assessed nor reproduced.

  2. The validity labels are undefined. V(x, t) requires the true label of each transformed image, which CIFAR-100 does not provide. The “audit protocol” is never described, yet false acceptance, IIGR and the oracle gate all need V for every training pair.

  3. Table 1 is internally inconsistent. The oracle gate’s 0% false acceptance at 76.1% coverage implies that about 23.9% of pairs are invalid. A gate that ignores validity cannot change that share, yet always-on consistency and the random gate report 28.6% and 22.7% false acceptance. False acceptance is never defined, and “coverage” for AROID is undefined...

  4. The threshold rule is standard risk control from selective classification and Learn-then-Test. Closely related methods and natural baselines are neither cited nor compared: Soft Augmentation, TeachAugment, InstaAug, and confidence gating in the style of FixMatch. The paper also does not address whether the calibration stays valid, even though the compatibility model’s input features change during training.

Questions for authors

Please refer to the Weaknesses. At this point, this paper looks incomplete to me and I recommend rejection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.