Align, Then Sample: Reference Region Alignment for Equivariance and Invariance

Contrary to the common assumption, convolutional neural networks (CNNs) are not inherently shift equivariant or invariant and a small shift in the input can lead to a drastic change in the output. Sampling operations, including downsampling and upsampling, are key components that can disrupt shift equivariance or invariance in CNN architectures. Conversely, sampling operations are important to improve computational efficiency and enlarge the receptive field for more contextual information. Furthermore, convolutional operations themselves break rotation and reflection invariance. Existing solutions typically address this issue at the model level rather than the individual layer level or rely on additional trainable parameters. In this paper, we introduce the reference region alignment (RRA) operation enhancing existing standard sampling methods (e.g., max pooling or average pooling), making them truly equivariant to arbitrary shifts, discrete $90^{\circ }$ rotations and horizontal or vertical reflections. When integrated into CNN architectures, the RRA operation instills exact shift and roto-reflection equivariance at the individual layer level even before training, without introducing new trainable parameters. Experimental results demonstrate that integrating RRA with existing standard sampling operators improves CNN performance, yielding accuracy and consistency that are superior or comparable to established shift-equivariant and shift-invariant sampling methods across classification and semantic segmentation tasks, while also exhibiting roto-reflection consistency.