acceptodds
Withdrawn from ICLR 2027

Modeling Spatial Transformations Instead of Memorizing Transformed Appearances

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Learning spatial transformations from transformed images can require surprisingly large amounts of data. We study this effect using controlled affine transformations of MNIST and examine how it changes with dataset size, model capacity, and generalization to unseen appearance–transformation combinations. We evaluate several classifier families, including Vision Transformers, ConvNeXts, Spatial Transformer Networks, and group-equivariant CNNs, spanning substantially different spatial inductive biases. We propose a Transform-Parameterized Implicit Neural Representation (TP-INR), which learns appearance while explicitly representing affine transformation structure and inferring its parameters through reconstruction-error minimization. Across architectures, TP-INR preprocessing substantially improves sample efficiency, with the largest gains in low-data regimes. We further find that a ViT can fail to generalize a known transformation distribution to known classes when their combination was not observed during training, while TP-INR partially mitigates this failure. These results suggest that explicitly encoding known transformation structure can make recognition more data- and parameter-efficient.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
sample efficiency, spatial transformations, compositional generalization, implicit neural representations, test-time optimization

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_U3uZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a new method for aligning input images prior to feeding them to a downstream recognition model. The method combines 1) an SIREN-based Implicit Neural Representation (INR) that models each image appearance as a continuous function of spatial coordinates and 2) a learnable six-parameter affine transformation acting on those coordinates. The central idea is to learn a shared INR-initialization for the input images in an shared coordinate frame by encouraging the model to prioritize geometric alignment before learning image appearance. Consequently, image alignment is achieved by finding an affine transformation that explains each input image best in terms of the shared initialization. The shared initialization is learned using a reptile-based meta learning scheme while indiviudal INRs are fitted using an anchoring penalty towards the shared representation and a coarse-to-fine optimization curriculum. At test-time the shared initialization is used to fit the affine parameters for alignment. The authors evaluate TP-INR on MNIST subjected to random affine transformations. The authors study data efficiency, model capacity and compositional generalization for four different architectures trained for digit-recognition, comparing performance with and without TP-INR-based alignment.

Critical strengths
  • S1: The paper presents an interesting approach to image alignment that learns a shared, mean-like representation through a SIREN-based INR across training images and aligns individual samples to this representation through learnable affine transformations.
  • S2: The method requires neither annotated data of canonical targets or from a pretrained foundation model, nor the gradient response of a downstream recognition network. The proposed method is trained solely from the pixel-wise reconstruction error.
  • S3: The data efficiency experiments are promising: across all architectures with substantially different spatial inductive biases TP-INR preprocessing improves recognition accuracy substantially, especially in the low data regime indicating that TP-INR is able to learn a useful inductive bias on a relatively small amount of training samples.
  • S4: The authors further show that larger transformers become increasingly capable of recognizing directly transformed inputs, while TP-INR-preprocessed images yield consistently higher accuracy across all tested model capacities. This supports the claim that recognizers can learn to accommodate spatial variation, but that explicit alignment reduces the model capacity required to achieve a given level of accuracy.
  • S5: The paper appropriately acknowledges that this study is limited to the application of TP-INR on a simple MNIST dataset combined with a controlled set of affine transformations.
Critical weaknesses
  • W1: In contrast to previous alignment (Jaderbergs STNs as cited, FoCal as cited or Schmidt and Stober ICML 2024) or canonicalization methods (Kaba et al as cited, Mondal et al NeurIPS 2023), the training curriculum involving the reptile-based meta-learning, inversely operating learning rate schedules, a coarse-to-fine fitting, the methods training scheme is relatively complex. Moreover, it requires test-time optimization for each input image. While these limitations are not sufficiently discussed, TP-INR is never directly compared to any alignment/canonicalization baseline. Consequently, it remains unclear whether the benefits of TP-INR justify these trade-offs.
  • W2: The baseline in the compositional generalization experiment seems confounded: the highest accuracy (class 5) is only at 1.7% which is significantly lower than random guessing which would give accuracies around 10% for 10 distinct digit classes. However, the model seems to have learned a correlation between pose and parity. It might be the case that transformed odd digits at test-time are automatically classified as even based on the pose-parity correlation rather than discriminative features of the digits. Since this experiment supports the paper’s main claim that recognition models primarily memorize joint appearance–transformation combinations rather than learn reusable representations of appearance and transformation, the potential confound substantially weakens the evidence for this claim.
  • W3: The architectural bias claim overarches (line 448-450): The authors state that "STNs and C4C_4 group-equivariant networks also benefit from TP-INR preprocessing, suggesting the architectural bias alone does not eliminate the cost of learning from transformed appearances." However, C4C_4 group-equivariant networks have built-in equivariance for steps of 90 degrees of rotation while the generated dataset covers a much more diverse set of transformations (rotations $\pm$45 degrees, scales in [0.5, 2.0], shear $\pm$15 degrees, translations $\pm$16 pixels). STNs on the other hand, learn this inductive bias from the loss of a downstream model during training and are not guaranteed to produce consistent canonical images. Therefore, STNs should be treated as a baseline alignment method for comparison with TP-INR, rather than as an architecture with a predefined inductive bias toward a particular transformation group. More than that: Vanilla STNs are very outdated. I would recommend at least a probabilistic STN (Schwöbel et al UAI 2022) baseline.
  • W4: The authors make several design choices that are either not explained or motivated from theoretical considerations but never ablated. For instance, the anchor weighting is an essential parameter that controls regularization strength towards the shared initialization and is only evaluated at λ=1.0\lambda=1.0. Further, the paper states that the transformation task, if unconstrained, is easily absorbed by the INR. This leads to multiple design choices for prioritizing geometric alignment before appearance learning: the anchor penalty, the blurred-target curriculum and the inversely acting learning rate schedules. However, neither the necessity of these components nor the specific choices made in their implementation are validated through ablation studies. Consequently, it remains unclear whether all three components are required and how sensitive the results are to their specific configurations.
  • W5: Reproducibility: The manuscript does not contain enough implementation details regarding the training and evaluation protocols (e.g. inner-loop learning rate schedules). This also applies to the STN architecture, the ViT which is parameter reduced (pretrained?). In the reproducibility statement, the authors mention supplementary material (including informations about the hyper parameters). However, it was neither contained in the main manuscript, nor is there a suppl. material file.
  • W6 As anticipated by the authors in their limitations, results are only shown on MNIST. There are some issues with this: (i) For a venue like ICLR this is simply not sufficient for a paper focusing on empirical findings rather than theory. (ii) It lacks evidence of how the method scales to real world datasets. (iii) MNIST itself comes with a bias which is likely being exploited here. This bias is the background. It is trivial to align or fill a frame when there is no distracting background and only a clear single foreground object. This makes alignment relative easy. The authors need to show at least on CIFAR or comparable how their method would work here.
  • W7 STNs or any regression of affine transformation parameters purely from data tend to collapse to degenerate predictions. Such collapses are either to constant transformation parameters for each input, or collapses in pixel space to extreme signals like zoom-ins until pixelation. The latter comes form singular affine matrices, which can occur as the regressed matrix is an unconstrained real matrix. But I think here the first one is actually more relevant. This raises the question how stable the proposed approach is. In addition to this what is missing entirely are confidence intervals, standard deviations, SEM or some measure to assess variance of a result.
Questions for authors
  • Q1 (W1): How does TP-INR compare directly against alignment and canonicalization baselines (e.g., STN variants, FoCal, Kaba et al., Mondal et al.) on the same downstream tasks? Could you also report training cost and test-time optimization cost (wall-clock time and number of inner-loop steps per image) relative to these baselines, so readers can judge whether the benefits justify the added complexity?
  • Q2 (W1, W7): The authors evaluate their method through the performance of a downstream recognition model. It would be interesting to see the alignment performance isolated through an alignment error compared to the inverse conditioned sample.
  • Q3 (W2): Can you rule out that the low baseline accuracy in the compositional generalization experiment comes from the learned pose–parity shortcut rather than from a failure to compose appearance and transformation? For example, could you report the confusion matrix or the proportion of odd test digits predicted as even, and add a control where pose is decorrelated from parity?
  • Q4 (W3): Given that C4C_4-equivariant networks only cover 90° rotations while the dataset includes continuous rotations, scaling, shear and translation, how do you justify the claim that architectural bias alone does not remove the cost of learning from transformed appearances? Would you consider reframing STNs as an alignment baseline and adding a stronger one, such as a probabilistic STN (Schwöbel et al., UAI 2022), or an architecture equivariant to a higher order transformation group?
  • Q5 (W4): Could you provide ablations for the anchor penalty (including values of λ\lambda other than 1.0), the blurred-target curriculum, and the inverse learning rate schedules, both individually and in combination? Are all three components needed, and how sensitive are the results to their configuration?
  • Q6 (W5): Fig. 3 states that each model contains approximately 1 million parameters. Conversely, the text states that each model contains approximately 100K params. The authors should check which one is correct and ensure consistent parameter counts.
  • Q7 (W5): Exp. 2 states that the amount of training data is fixed but never tells which amount. Please specify.
  • Q8 (W6): How does TP-INR perform on datasets with cluttered backgrounds and natural images, such as CIFAR-10 or a comparable benchmark? In particular, how does the INR fitting handle background content that is not part of the object to be aligned?
  • Q9 (W7): How stable is the transformation estimation across inputs and random seeds? Do you see collapse to constant transformation parameters or degenerate (near-singular) affine matrices? Could you report standard deviations or confidence intervals over multiple seeds for all main results?
Reviewer_UpSqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this draft, the authors studied how much data recognizers need to learn affine transformations, using transformed MNIST. They proposed TP-INR, which fits an implicit neural representation with an explicit affine transform for each image and uses it to align the image before classification. Experiments with several classifiers showed large gains in low-data regimes.

Critical strengths
  1. The question is clear and the experiments are well controlled, including an oracle alignment condition.
  2. TP-INR is a simple, label-free preprocessing step and gives large gains for all four classifiers with little data.
Critical weaknesses
  1. There are no ablations or simple alignment baselines (e.g., moment normalization), so it is not clear what part of TP-INR is needed.
  2. The baselines may be under-trained (fixed 10k steps), and all results are single runs.
  3. Experiment 3 has a shortcut, since being transformed perfectly predicts an even digit during training.
  4. Results are only on MNIST.
Questions for authors

Please refer to the Critical Weaknesses section for details.

Reviewer_zkniOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper empirically investigates how spatial image transformations of data impact training in relevant models, and proposes a new model to more gracefully handle such transformations.

Critical strengths

As far as I can tell, the paper is sound.

Critical weaknesses
  1. I have some concerns about the studied problem. Specifically, affine image transformations are very difficult to apply conceptually beyond MNIST. The black background in MNIST images allows us to skew these images without introducing a weird background (since we can just extrapolate with 'black'). In almost all other settings, such is not the case. This is why more recent approaches go beyond affine transformations, e.g. Deep Diffeomorphic Transformer Networks (CVPR 2018) and variants thereof. I'm not sure if the here-presented findings carry over to these more relevant scenarios.
  2. The paper is fairly hard to read because too many important details are merely given as references. For example, the sentence "Importantly, Reptile operates on the complete SIREN parameter vector" requires an understanding of both Reptile and SIREN, neither of which are explained beyond a high-level description. This lack of detail makes the paper hard to read.
  3. If I understand correctly, the proposed method does alignment/optimization during inference, which sounds rather expensive/impractical.
  4. Instead of using STNs as a baseline, I think more contemporary variants like "Probabilistic STNs" (UAI 2022) would be better.
Questions for authors

Beyond the weaknesses above, I'd like to hear if the authors have any experience with the transformations collapsing (i.e. non-invertible transformations)? Such issues are common in STNs.

Reviewer_srBJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the data and model capacity required to recognize spatially transformed images and proposes Transform-Parameterized Implicit Neural Representations (TP-INR) to improve recognition efficiency. The proposed method uses a meta-learned SIREN to represent image appearance and explicitly parameterizes affine transformations, jointly optimizing both through reconstruction loss and regularization at inference time. Experiments on affine-transformed MNIST show improvements in sample efficiency across several recognition architectures and better generalization to unseen appearance–transformation combination

Critical strengths
  1. Simple and intuitive motivation. Explicitly modeling spatial transformations instead of requiring a recognition model to learn transformation invariance entirely from data is a reasonable approach, particularly in low-data settings.

  2. Evaluation across multiple architectures. The experiments include ViT, ConvNeXt, Spatial Transformer Networks, and group-equivariant CNNs, demonstrating that the proposed preprocessing can improve classification performance across different architectural inductive biases.

Critical weaknesses

1. The proposed appearance–geometry disentanglement is not theoretically justified and lacks direct experimental validation.

The central assumption is that a SIREN can represent appearance while a separately optimized affine transformation captures geometric variation. However, the decomposition is inherently non-unique: different combinations of SIREN weights and affine parameters can reconstruct the same image. The regularization in Eq. (3) merely encourages the adapted SIREN weights to remain close to a meta-learned initialization, without guaranteeing that the inferred affine parameters correspond to the actual transformations.

Furthermore, the SIREN initialization is meta-learned from randomly affine-transformed images rather than canonical appearances. Consequently, the learned initialization may favor arbitrary spatial configurations, and the optimization may recover transformations that minimize changes to the SIREN weights rather than geometrically correct transformations.

Despite this being the central premise of the paper, the authors provide no direct evaluation of transformation recovery accuracy or appearance–geometry disentanglement. Since the applied affine transformations are known in their synthetic dataset, such evaluations should be straightforward. The authors should also test whether independently transformed versions of the same image produce consistent canonicalized representations.

2. The asymmetric coarse-to-fine reconstruction objective is insufficiently justified.

In Section 3.2, the authors progressively reduce Gaussian blurring of the target image while leaving the SIREN-generated image unsmoothed. Consequently, the reconstruction loss compares images with different frequency characteristics. In particular, high-frequency components in the SIREN prediction are penalized even when the corresponding information has been removed from the target.

This mismatch may introduce undesirable optimization behavior, especially for scaling and appearance adaptation. Conventional multiscale image registration typically smooths both images to ensure that alignment is optimized at comparable spatial frequencies. The authors should justify their asymmetric design and compare it against symmetric Gaussian smoothing, preferably with direct measurements of affine transformation recovery.

3. The extremely poor baseline performance raises concerns about the experimental conclusions.

In Experiment 3, the baseline ViT achieves only 1.36% classification accuracy on transformed odd digits, which is surprisingly low even under the proposed distribution shift. The baseline contains approximately 110K parameters, raising concerns about whether the result reflects a fundamental limitation of compositional generalization or simply insufficient model capacity and baseline optimization.

Although TP-INR improves accuracy to 26.84%, this remains relatively poor for MNIST classification. The authors should evaluate stronger and better-tuned recognition architectures, provide confusion matrices, and establish that the baseline has learned the odd-digit classes under their original training conditions. Without these experiments, the claimed improvements in compositional generalization are difficult to interpret.

4. Limited novelty relative to existing approaches.

The proposed method combines several established techniques: SIREN-based implicit neural representations, Reptile meta-learning, explicit affine parameterization, reconstruction-based geometric alignment, and coarse-to-fine Gaussian smoothing. The overall approach is closely related to classical deformable template matching, Active Appearance Models, and optimization-based image registration.

While combining these components may be practically useful, the paper does not clearly identify a sufficiently novel methodological contribution beyond their integration. In particular, the use of a learned appearance prior to guide transformation estimation is not fundamentally new.

5. Limited experimental scope and substantial inference overhead.

All experiments are conducted on affine-transformed MNIST, a highly simplified setting involving isolated objects and a known family of global geometric transformations. The effectiveness of the method on natural images, where appearance variation, background clutter, and local deformations are substantially more complex, remains untested.

Furthermore, TP-INR requires 40 gradient-based optimization steps for every image at inference time. The paper does not provide a quantitative comparison of inference latency or computational cost against alternative preprocessing methods. Given this substantial overhead, the practical value of the reported classification improvements remains unclear.

Questions for authors

NA

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.