- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose a new method for aligning input images prior to feeding them to a downstream recognition model. The method combines 1) an SIREN-based Implicit Neural Representation (INR) that models each image appearance as a continuous function of spatial coordinates and 2) a learnable six-parameter affine transformation acting on those coordinates. The central idea is to learn a shared INR-initialization for the input images in an shared coordinate frame by encouraging the model to prioritize geometric alignment before learning image appearance. Consequently, image alignment is achieved by finding an affine transformation that explains each input image best in terms of the shared initialization. The shared initialization is learned using a reptile-based meta learning scheme while indiviudal INRs are fitted using an anchoring penalty towards the shared representation and a coarse-to-fine optimization curriculum. At test-time the shared initialization is used to fit the affine parameters for alignment. The authors evaluate TP-INR on MNIST subjected to random affine transformations. The authors study data efficiency, model capacity and compositional generalization for four different architectures trained for digit-recognition, comparing performance with and without TP-INR-based alignment.
Critical strengths
- S1: The paper presents an interesting approach to image alignment that learns a shared, mean-like representation through a SIREN-based INR across training images and aligns individual samples to this representation through learnable affine transformations.
- S2: The method requires neither annotated data of canonical targets or from a pretrained foundation model, nor the gradient response of a downstream recognition network. The proposed method is trained solely from the pixel-wise reconstruction error.
- S3: The data efficiency experiments are promising: across all architectures with substantially different spatial inductive biases TP-INR preprocessing improves recognition accuracy substantially, especially in the low data regime indicating that TP-INR is able to learn a useful inductive bias on a relatively small amount of training samples.
- S4: The authors further show that larger transformers become increasingly capable of recognizing directly transformed inputs, while TP-INR-preprocessed images yield consistently higher accuracy across all tested model capacities. This supports the claim that recognizers can learn to accommodate spatial variation, but that explicit alignment reduces the model capacity required to achieve a given level of accuracy.
- S5: The paper appropriately acknowledges that this study is limited to the application of TP-INR on a simple MNIST dataset combined with a controlled set of affine transformations.
Critical weaknesses
- W1: In contrast to previous alignment (Jaderbergs STNs as cited, FoCal as cited or Schmidt and Stober ICML 2024) or canonicalization methods (Kaba et al as cited, Mondal et al NeurIPS 2023), the training curriculum involving the reptile-based meta-learning, inversely operating learning rate schedules, a coarse-to-fine fitting, the methods training scheme is relatively complex. Moreover, it requires test-time optimization for each input image. While these limitations are not sufficiently discussed, TP-INR is never directly compared to any alignment/canonicalization baseline. Consequently, it remains unclear whether the benefits of TP-INR justify these trade-offs.
- W2: The baseline in the compositional generalization experiment seems confounded: the highest accuracy (class 5) is only at 1.7% which is significantly lower than random guessing which would give accuracies around 10% for 10 distinct digit classes. However, the model seems to have learned a correlation between pose and parity. It might be the case that transformed odd digits at test-time are automatically classified as even based on the pose-parity correlation rather than discriminative features of the digits. Since this experiment supports the paper’s main claim that recognition models primarily memorize joint appearance–transformation combinations rather than learn reusable representations of appearance and transformation, the potential confound substantially weakens the evidence for this claim.
- W3: The architectural bias claim overarches (line 448-450): The authors state that "STNs and group-equivariant networks also benefit from TP-INR preprocessing, suggesting the architectural bias alone does not eliminate the cost of learning from transformed appearances." However, group-equivariant networks have built-in equivariance for steps of 90 degrees of rotation while the generated dataset covers a much more diverse set of transformations (rotations $\pm$45 degrees, scales in [0.5, 2.0], shear $\pm$15 degrees, translations $\pm$16 pixels). STNs on the other hand, learn this inductive bias from the loss of a downstream model during training and are not guaranteed to produce consistent canonical images. Therefore, STNs should be treated as a baseline alignment method for comparison with TP-INR, rather than as an architecture with a predefined inductive bias toward a particular transformation group. More than that: Vanilla STNs are very outdated. I would recommend at least a probabilistic STN (Schwöbel et al UAI 2022) baseline.
- W4: The authors make several design choices that are either not explained or motivated from theoretical considerations but never ablated. For instance, the anchor weighting is an essential parameter that controls regularization strength towards the shared initialization and is only evaluated at . Further, the paper states that the transformation task, if unconstrained, is easily absorbed by the INR. This leads to multiple design choices for prioritizing geometric alignment before appearance learning: the anchor penalty, the blurred-target curriculum and the inversely acting learning rate schedules. However, neither the necessity of these components nor the specific choices made in their implementation are validated through ablation studies. Consequently, it remains unclear whether all three components are required and how sensitive the results are to their specific configurations.
- W5: Reproducibility: The manuscript does not contain enough implementation details regarding the training and evaluation protocols (e.g. inner-loop learning rate schedules). This also applies to the STN architecture, the ViT which is parameter reduced (pretrained?). In the reproducibility statement, the authors mention supplementary material (including informations about the hyper parameters). However, it was neither contained in the main manuscript, nor is there a suppl. material file.
- W6 As anticipated by the authors in their limitations, results are only shown on MNIST. There are some issues with this: (i) For a venue like ICLR this is simply not sufficient for a paper focusing on empirical findings rather than theory. (ii) It lacks evidence of how the method scales to real world datasets. (iii) MNIST itself comes with a bias which is likely being exploited here. This bias is the background. It is trivial to align or fill a frame when there is no distracting background and only a clear single foreground object. This makes alignment relative easy. The authors need to show at least on CIFAR or comparable how their method would work here.
- W7 STNs or any regression of affine transformation parameters purely from data tend to collapse to degenerate predictions. Such collapses are either to constant transformation parameters for each input, or collapses in pixel space to extreme signals like zoom-ins until pixelation. The latter comes form singular affine matrices, which can occur as the regressed matrix is an unconstrained real matrix. But I think here the first one is actually more relevant. This raises the question how stable the proposed approach is. In addition to this what is missing entirely are confidence intervals, standard deviations, SEM or some measure to assess variance of a result.
Questions for authors
- Q1 (W1): How does TP-INR compare directly against alignment and canonicalization baselines (e.g., STN variants, FoCal, Kaba et al., Mondal et al.) on the same downstream tasks? Could you also report training cost and test-time optimization cost (wall-clock time and number of inner-loop steps per image) relative to these baselines, so readers can judge whether the benefits justify the added complexity?
- Q2 (W1, W7): The authors evaluate their method through the performance of a downstream recognition model. It would be interesting to see the alignment performance isolated through an alignment error compared to the inverse conditioned sample.
- Q3 (W2): Can you rule out that the low baseline accuracy in the compositional generalization experiment comes from the learned pose–parity shortcut rather than from a failure to compose appearance and transformation? For example, could you report the confusion matrix or the proportion of odd test digits predicted as even, and add a control where pose is decorrelated from parity?
- Q4 (W3): Given that -equivariant networks only cover 90° rotations while the dataset includes continuous rotations, scaling, shear and translation, how do you justify the claim that architectural bias alone does not remove the cost of learning from transformed appearances? Would you consider reframing STNs as an alignment baseline and adding a stronger one, such as a probabilistic STN (Schwöbel et al., UAI 2022), or an architecture equivariant to a higher order transformation group?
- Q5 (W4): Could you provide ablations for the anchor penalty (including values of other than 1.0), the blurred-target curriculum, and the inverse learning rate schedules, both individually and in combination? Are all three components needed, and how sensitive are the results to their configuration?
- Q6 (W5): Fig. 3 states that each model contains approximately 1 million parameters. Conversely, the text states that each model contains approximately 100K params. The authors should check which one is correct and ensure consistent parameter counts.
- Q7 (W5): Exp. 2 states that the amount of training data is fixed but never tells which amount. Please specify.
- Q8 (W6): How does TP-INR perform on datasets with cluttered backgrounds and natural images, such as CIFAR-10 or a comparable benchmark? In particular, how does the INR fitting handle background content that is not part of the object to be aligned?
- Q9 (W7): How stable is the transformation estimation across inputs and random seeds? Do you see collapse to constant transformation parameters or degenerate (near-singular) affine matrices? Could you report standard deviations or confidence intervals over multiple seeds for all main results?