acceptodds
Withdrawn from ICLR 2027

Align, Then Sample: Reference Region Alignment for Equivariance and Invariance

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Contrary to the common assumption, convolutional neural networks (CNNs) are not inherently shift equivariant or invariant and a small shift in the input can lead to a drastic change in the output. Sampling operations, including downsampling and upsampling, are key components that can disrupt shift equivariance or invariance in CNN architectures. Conversely, sampling operations are important to improve computational efficiency and enlarge the receptive field for more contextual information. Furthermore, convolutional operations themselves break rotation and reflection invariance. Existing solutions typically address this issue at the model level rather than the individual layer level or rely on additional trainable parameters. In this paper, we introduce the reference region alignment (RRA) operation enhancing existing standard sampling methods (e.g., max pooling or average pooling), making them truly equivariant to arbitrary shifts, discrete rotations and horizontal or vertical reflections. When integrated into CNN architectures, the RRA operation instills exact shift and roto-reflection equivariance at the individual layer level even before training, without introducing new trainable parameters. Experimental results demonstrate that integrating RRA with existing standard sampling operators improves CNN performance, yielding accuracy and consistency that are superior or comparable to established shift-equivariant and shift-invariant sampling methods across classification and semantic segmentation tasks, while also exhibiting roto-reflection consistency.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Shift Equivariance, Equivariant Neural Networks, Convolutional Neural Networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yq9qOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Reference Region Alignment (RRA), an operator applied around downsampling operations in ConvNets aiming to make general them translation and roto-reflection equivariant. The method is evaluated on standard ResNet for image classification on CIFAR and ImageNet and semantic segmentation on PASCAL-VOC The paper reports 100% consistency under circular shifts, rotations and reflections, improved consistency under standard translations, and competitive accuracy compared with conventional sampling and approaches such as APS, LPS and LPF.

Critical strengths
  1. RRA introduces no additional trainable parameters and can be combined with standard sampling operators.
  2. RRA is evaluated on several datasets, including classification and segmentation tasks, and reports perfect circular shift-consistency and improved standard-shift consistency, while maintaining competitive accuracy.
Critical weaknesses
  1. The method is introduced as an adaptation to sampling operators in CNNs. However, in experiments involving rotations and reflections, RRA is additionally applied to the input. This appears to serve as canonicalization, addressing the fact that standard convolutions are not rotation/reflection equivariant, but this point is not discussed in the paper. With valid input canonicalization, applying RRA around internal sampling operators is unnecessary to obtain classification invariance. It therefore necessary to compare the method against other canonicalization approaches such as [1]. Additionally, it is unclear whether input RRA is enabled for the accuracy and shift-consistency results in Tables 2 and 3, or only for the rotation/reflection evaluation mentioned in the text. In the latter case, this makes it difficult to assess its effect on accuracy. Additionally, Rotation/reflection consistency is also not reported for the other baselines.

  2. The paper does not engage with more recent anti-aliasing approaches such as [2,3]. In particular, [3] guarantees invariance to both integer and fractional circular translations, whereas the evaluated LPF approach [4] provides more limited guarantees. Additionally, [3] demonstrates that adaptive sampling approaches can be vulnerable to fractional translations, an issue that is not discussed or evaluated here.

  3. In some cases the gains of RRA compared to the other baselines appears to be small, for example the accuracy in Table 2 and S-Consistency in Table 3. The paper does not report standard deviation of the results in Tables 2-4 although the experiments are repeated for 5 seeds.

[1] Equivariant Adaptation of Large Pretrained Models, Mondal et al., 2023.

[2] Alias-Free Generative Adversarial Networks, Karras et al, 2021

[3] Alias-Free Convnets: Fractional Shift Invariance via Polynomial Activations, Michaeli et al, 2023

[4] Making convolutional networks shift-invariant again, Zhang, 2019

Questions for authors
  1. Is RRA applied on the input in the results reported in Tables 2 and 3? if not, can you report how does applying RRA on the input affect the accuracy?
  2. Can you report an input-only RRA baseline and compare with other canonicalization methods? Specifically, is input-only RRA is sufficient for translation and roto-reflection invariance?
  3. Can you report rotation/reflection consistency for the competing methods in Tables 2-3?
  4. How does RRA perform compared to the other baselines under fractional translations, such as those considered in [3]?
  5. Can you report the standard deviation of the results reported in Tables 2 and 3?

[3] Alias-Free Convnets: Fractional Shift Invariance via Polynomial Activations, Michaeli et al, 2023

2 comments

We sincerely thank the reviewer for the constructive feedback and valuable suggestions. We address the specific questions below.

Q1. Input RRA in Tables 2 and 3

RRA was not applied to the input in the experiments reported in Tables 2 and 3; it was incorporated only around the internal sampling operations. Input-level canonicalization was used separately for the roto-reflection consistency evaluation, since standard convolutions are not inherently rotation/reflection equivariant. We acknowledge that evaluating the effect of input-level RRA on classification accuracy would provide valuable additional insights. We appreciate this suggestion and intend to investigate it in the next iteration of our work.

Q2. Input-only RRA and canonicalization baselines

We agree that evaluating an input-only RRA baseline and comparing it with existing canonicalization methods would help distinguish the contributions of input-level and layer-level alignment. In principle, valid input canonicalization can provide classification invariance to the transformations it canonicalizes, whereas internal RRA is designed to preserve transformation consistency at sampling operations. A controlled comparison of input-only, layer-only, and combined configurations would provide a clearer understanding of their respective contributions. We appreciate the reviewer highlighting this important experimental direction and intend to incorporate these comparisons into the next iteration of the work.

Q3. Rotation/reflection consistency of competing methods

The baseline results in Tables 2 and 3 were taken directly from their respective original publications rather than obtained through reimplementation or retraining. The competing methods primarily address shift invariance/equivariance and do not explicitly claim or report rotation/reflection consistency under a comparable evaluation protocol. Consequently, these results were not available for inclusion in our comparisons. We agree that evaluating the competing methods under a common rotation/reflection protocol would provide a more comprehensive assessment and appreciate this suggestion for the next iteration of our work.

Q4. Fractional translations

We thank the reviewer for highlighting this important direction and drawing our attention to the work of Michaeli et al. (2023). Our present formulation of RRA focuses on discrete integer translations and roto-reflections, while fractional translations have not been investigated theoretically or experimentally. Therefore, we cannot make claims about its performance relative to existing methods under fractional translations. We acknowledge the relevance of recent alias-free approaches, including Karras et al. (2021) and Michaeli et al. (2023), and recognise the importance of positioning RRA within this broader literature. We appreciate this suggestion and will consider fractional-translation robustness and comparisons with these methods in the next iteration of our work.

Q5. Standard deviations in Tables 2 and 3

All RRA experiments were repeated across five random seeds, and the corresponding standard deviations are already reported in the manuscript text. However, the baseline results in Tables 2 and 3 were taken directly from their respective original publications, where standard deviations were not consistently available. Reporting comparable standard deviations for these methods would therefore require reimplementing and rerunning the baselines. We hope this clarifies the distinction between our experimental results and the previously published baseline scores.

I thank the authors for their responses. Unfortunately, my main concern remains unresolved. The authors clarify that the roto-reflection consistency results, reported only in the text, were obtained using input-level RRA, whereas the accuracy results in Tables 2 and 3 were obtained without it. Thus, no classification accuracy is reported for the configuration used to obtain roto-reflection consistency, leaving the effect of input-level RRA on accuracy unclear. Since roto-reflection invariance is a central claimed contribution, I consider this an important gap in the evaluation and therefore maintain my original score.

Reviewer_DV1pOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets the problem of transformation (shift, flip, rotation) equivariance/invariance in CNNs, with a specific focus on shift invariance. They propose "Reference Region Alignment". The method can roughly be summarized as follows:

Given an input image (or feature map) a "reference region" and "reference orientation" are identified. First, several "directional signatures" are defined (Eq 8 to 11), which are essentially k \times k kernels applied to the input via convolution, such that each location has a score for each signature. These kernels are further defined such that the score of an input patch when transformed by the dihedral group (which consists of multiples of 90 degree rotations and reflections) can also be inferred, so the patch can be scored under different transformations (without actually applying the transformations). The k \times k region that has the largest sum of these scores is then the "reference region" and the transform that results in this score is the "reference orientation".

The RRA operation is then applied to the entire image (or feature map): which first applies the reference orientation and then circularly shifts the reference region such that it sits at the origin. The standard subsampling operation is then applied (max pooling), before the inverse of the RRA is again applied to reverse the shift and re-orientation of the image. The intuition is that this shift operation consistently aligns the inputs such that the subsampling operation always discards/retains the same information, no matter how the original input is translated, so long as the same reference region and orientation is selected. The effect of this is shift equivariance, which then results in shift invariance for the network when global average pooling is applied at the end. When applied to the input image, the RRA operator can also ensure invariance to dihedral group transformations (again, assuming the same reference region and orientation is selected).

The paper then reports multiple experiments comparing the proposed method to baselines such as Adaptive Polyphase Sampling (APS) and Learnable Polyphase Sampling (LPS), and RRA is shown to generally (slightly) outperform the competitors for image classification and semantic segmentation under various evaluation metrics.

Critical strengths

The method is very interesting, and seems sound and logical. It is also described in sufficient detail (albeit a bit obtusely). The experiments are similarly logical and the results promising, with sufficient breadth: both image classification and semantic segmentation, across various datasets and backbone architectures with various evaluation metrics. I am also impressed by the fact that no additional learnable parameters are introduced and that the method can be seamlessly inserted into any CNN.

Critical weaknesses

My main concerns are about the experimental results:

  1. No hyperparameter tuning is mentioned - some of the baselines could be under-tuned and the visible performance gains simply due to better hyperparameter selection, especially given that many of the gains are very slight. I'd recommend at least a learning rate sweep across the different methods (if these are re-implemented, see Question 1)
  2. The authors state that 5 runs are performed per model and report standard deviations in the text (but not the tables), but this is never reported for the baselines.
  3. Some baseline comparisons are simply omitted without explanation, e.g. ResNet-50 with DDAC in Table 3 and ResNet-101 APS in Table 4.
  4. Some ablations are missing, namely the effect on performance and invariance of 1) the region size k, 2) Applying RRA with and without anti-aliasing, and it would be interesting to measure 3) RRA applied to input versus internal layers versus both.

In terms of presentation, much of the core method is obfuscated by rather dense equations. Figure (1) makes this more understandable but I believe including a more natural description of the method as well would be very beneficial.

Questions for authors
  1. Are the baselines re-implemented/re-trained or do you report the published scores? This would directly address concerns (1) and (2) . If published scores are reported, no additional HP sweep is required, but this should be made clear in the experimental setup.
  2. Why are certain baseline comparisons omitted (Concern (3))? Please make this clear in the text.
  3. Why are two different backbones used for circular and standard shifts in Table 4?
  4. Which sub-parts of the proposed method is novel? Are some ideas re-purposed from prior work? This is not currently clear from the related work section.
  5. Can you perform the requested ablations (at least the first two mentioned in Concern (4))?

My main hesitation about acceptance is the fairness of the experimental comparison (concern 1 and question 1), if these are adequately addressed I will reconsider my current rating.

1 comment

We sincerely thank the reviewer for the constructive feedback and valuable suggestions. We address the specific questions below.

Q1. Source of baseline results

All baseline results were taken directly from their respective original publications rather than obtained through reimplementation or retraining. Therefore, no additional hyperparameter tuning was performed for the baseline methods. We acknowledge that this distinction should have been explicitly stated in the experimental setup and appreciate the opportunity to clarify it here.

Q2. Omitted baseline comparisons

As clarified in our response to Question 1, all baseline results were taken directly from the original publications. Specifically, the DDAC results for ImageNet classification were reported using ResNet-101, while corresponding ResNet-50 results were not provided (Table 3). Similarly, the LPS paper reports APS results for circular-shift semantic segmentation using ResNet-18, but does not report APS results for standard-shift semantic segmentation using ResNet-101 (Table 4). Consequently, these entries were omitted because the corresponding published results were unavailable, rather than selectively excluded. We hope this clarifies the reasons for the missing comparisons.

Q3. Different segmentation backbones

The two backbones were selected to maintain consistency with the established experimental protocols rather than due to any architectural requirement of the proposed method. Specifically, for circular-shift semantic segmentation, we followed the LPS experimental setup (Section 5.4, Table 5 [1]), which uses DeepLabV3+ with an ImageNet-pretrained ResNet-18 backbone, following the circular-shift evaluation protocol inspired by APS. For standard shifts, we followed the LPS setup (Section 5.5, Table 6 [1]), which adopts the DDAC experimental protocol using DeepLabV3+ with an ImageNet-pretrained ResNet-101 backbone. The use of ResNet-101 is also consistent with the original DeepLabV3+ architecture evaluated on PASCAL VOC. These configurations were retained to enable comparisons with the published baseline results under their respective experimental protocols.

Q4. Novel contributions and relationship to prior work

We would like to clarify that RRA was developed independently of APS, LPS, and existing canonicalization approaches. The proposed contributions of RRA include: (1) the formulation of directional signatures (Eqs. 8–11) for identifying a reference region and its orientation under the dihedral group D4D_4; (2) a deterministic, content-dependent mechanism for jointly selecting a spatial reference region and orientation; and (3) the integration of reference-region alignment, sampling, and inverse alignment without introducing additional trainable parameters.

We appreciate the reviewer's observation, which provides valuable guidance for refining the novelty claims and positioning of the next iteration of our work.

Q5. Requested ablation studies

We thank the reviewer for these valuable suggestions. To address the requested ablations, we conducted additional experiments on CIFAR-10 using ResNet-18.

First, we evaluated the effect of reference-region size k∈{5,7,9}k \in \{5, 7, 9\}. RRA consistently achieved 100% circular-shift classification consistency across all three configurations, demonstrating that the observed circular-shift consistency is maintained for the evaluated reference-region sizes.

Second, we evaluated RRA with and without anti-aliasing. Without anti-aliasing, RRA achieved 94.03% top-1 accuracy, whereas incorporating anti-aliasing increased the accuracy to 94.60%. Both configurations maintained 100% circular-shift consistency. These results suggest that anti-aliasing can improve classification accuracy without compromising the circular-shift consistency achieved by RRA.

We appreciate the reviewer's suggestions, which have helped us further examine the proposed method. A more comprehensive investigation of the influence of kk on classification accuracy and standard-shift consistency will be considered in the next iteration of our work.

Additional comment on presentation

We also appreciate the reviewer's suggestion to improve the accessibility of the methodological presentation. We recognise that the mathematical formulation would benefit from a more intuitive explanation of the proposed approach. In the next iteration of the manuscript, we intend to introduce a concise, step-by-step description of reference-region identification, orientation alignment, spatial alignment, sampling, and inverse alignment before presenting the detailed mathematical formulation.

Reference

[1] Renan A Rojas-Gomez, Teck-Yian Lim, Alex Schwing, Minh Do, and Raymond A Yeh. Learnable polyphase sampling for shift invariant and equivariant convolutional networks. Advances in Neural Information Processing Systems, 35:35755–35768, 2022.

Reviewer_kyvuOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The lack of invariance or equivariance to geometric transformations in CNNs is a well-known property. This work aims to achieve equivariance by canonicalizing the layer input at layers where it matters (i.e. where sampling occurs). The method is mathematically guaranteed for equivariance to Zc2⋊D4\mathbb{Z}_c^2 \rtimes D_4, and does not require additional training parameters. Furthermore, it’s also claimed that this work is the first to introduce equivariance into existing sampling operations. Thru the use of an energy function, the RRA score, the method canonicalizes the input to the sampling layer with a reversible transform. The proposed method showed improved performance over prior equivariant work on CNNs for both image classification and segmentation tasks.

Critical strengths

The work presented clear and well-written mathematical details throughout the problem definition and method description, with useful illustrative examples and tabulated outcomes of the proposed method where it helps understanding of the method. These descriptions also include key results and outcomes that were from prior work, filtered down to what’s necessary for interpretation of the current work. The authors clearly stated in various locations that this work is for CNNs only, and this is by design, as it works on the sampling layers. The contribution is scoped and does not over-claim in this aspect.

Critical weaknesses

Firstly, the claim that RRA is the "first approach to introduce equivariance into sampling operation" is an over claim. Both the cited literature, APS and LPS, does this. LPS has trainable parameters, APS does not. Thus the claim of novelty by this definition does not stand. What this work introduces is a potentially different energy or scoring function that APS uses (if we are comparing methods with no trainable parameters), or outside of what LPS can learn (if we want to include trainable methods). Canonicalize then sample (this method) versus baking selection rules into the sampling (LPS and APS) are algebraically isomorphic.

Secondly, while the introduction of the paper mentions Zc2⋊D4\mathbb{Z}_c^2 \rtimes D_4 equivariance, the entirety of the paper only treats Zc2\mathbb{Z}_c^2. Please show clearly experiments, mathematical proofs and illustrative examples where it matters for Zc2⋊D4\mathbb{Z}_c^2 \rtimes D_4 if this equivariance is claimed. The entire flow of this work follows closely the work of both APS and LPS, even though the claim of equivariance is a superset.

Finally, all of the experiments seems to be single seed runs with no variances. Performance gains are small and difficult to tell if they are significant. Even if larger model runs is not computationally feasible, at least show some error bars for the smaller networks like ResNet18.

Questions for authors

As above. Canonicalize then sample vs polyphase component selection are algebraically isomorphic. This alone does not make the proposed method invalid or redundant. However, there's a need to show that this provides either practical benefits (e.g. latency, efficient implementation, etc) or other differences. A more detailed comparison with at least APS should be discussed and also if RRA is a rule that LPS will not be able to learn.

Significant expansion is required to extend the results and experiments here to Zc2⋊D4\mathbb{Z}_c^2 \rtimes D_4 for the rotations and reflections. None of those are discussed in sufficient details and explored in the experiments.

Finally, add error bars (more runs with different seeds), the performance improvements are too small to be called actual improvements.

1 comment

We thank the reviewer for the constructive feedback and valuable suggestions. We address the specific concerns below.

Q1. Relationship to APS/LPS and novelty

We agree that the relationship between canonicalization followed by sampling and polyphase component selection requires a more detailed mathematical comparison. We also acknowledge that APS and LPS already introduce transformation-consistent sampling mechanisms, and that our novelty claims should be more precisely positioned within the existing literature.

RRA introduces a deterministic reference-region selection mechanism based on directional signatures, followed by spatial and orientation alignment, sampling, and inverse alignment without additional trainable parameters. However, we recognise that the parameter-free property is also shared with APS, and we do not claim computational advantages over APS without supporting measurements. Similarly, whether the RRA selection rule can be represented by LPS requires further theoretical investigation.

We appreciate the reviewer highlighting these important questions. A more comprehensive comparison with APS and LPS will help us establish the precise mathematical distinctions and refine the novelty claims in the next iteration of our work.

Q2. Combined translation and roto-reflection equivariance

We agree that a more comprehensive theoretical discussion and experimental evaluation of the combined translation and roto-reflection group Zc2⋊D4\mathbb{Z}_c^2 \rtimes D_4 would further strengthen the paper. We acknowledge that the current rotation/reflection experiments are limited and that a more extensive evaluation is necessary to substantiate the broader transformation-consistency claims.

We appreciate the reviewer highlighting this important limitation. Addressing it requires a more comprehensive theoretical and experimental framework, which we intend to develop in the next iteration of our work.

Q3. Error bars and multiple random seeds

We would like to clarify that all RRA experiments were repeated across five random seeds, and the corresponding standard deviations are already reported in the manuscript text. These values were not included directly in the tables because the baseline results were taken from their respective original publications, where comparable standard deviations were not consistently available.

We agree that the small performance differences should be interpreted cautiously without comparable variability estimates for the competing methods. A more comprehensive statistical comparison would require reimplementing and rerunning the baselines under consistent experimental conditions. We appreciate this suggestion and intend to strengthen the statistical evaluation in the next iteration of our work.

Concluding remarks

We sincerely thank the reviewer for the constructive feedback. The suggestions regarding novelty, mathematical comparisons, and roto-reflection evaluation provide valuable directions for further developing and strengthening our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.