acceptodds
Under review as a conference paper at ICLR 2027

Location: What View Manifolds Can and Cannot Tell Us About Invariant Recognition

Abstract

Deep vision networks could recognise certain objects from every viewpoint well but fail on others, and we hypothesise two possible ways in which recognition invariance is achieved through the viewpoint manifold of the object: gradual collapse in later layers, or reorganisation. We test both possibilities against the networks' classification heads and found three factors influencing readout consistency, which are how far the view set sits from the decision boundary, how widely they spread, and how much of the spread the readout measures on the viewpoint manifold. We measured the factors using 15 ShapeNet objects rendered at forty angles in a full azimuth rotation with ViT-B/16 and ResNet-50. In the two networks tested, correct recognition is rarely achieved, with two and five in fifteen objects, respectively. Relative collapse of the manifold across depth is not a requirement for correct recognition either. Compactness of viewpoints is positively related to the consistency of the recognition and shows no detectable association with its correctness. Correctness is related to where the viewpoint set lies relative to the boundary. Summaries computed only from distances between views cannot certify recognition.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, invariance, representation learning, representation geometry, computer vision

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_6TW4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how neural networks achieve invariant recognition. Specially, it selects ViT-B/16 and ResNet-50 for test, and the two networks’ predictions for different views of the same object are evaluated. It arrives at some conclusions about relative collapse and directional collapse based on carefully designed experiments.

Critical strengths

Investigating the neural mechanism of invariant recognition is important and doing experiments by evaluating the networks’ predictions from different views is interesting.

Critical weaknesses
  1. In terms of writing, some paragraphs contain overly long sentences, which impair the clarity and readability of the argumentation.
  2. This paper reads more like an analytical report. It is suggested to briefly introduce in which downstream tasks the hypothesis raised in the work can be manifested? What are the potential applications of the conclusions presented herein.
  3. The paper mentions sampling object images at different azimuth angles over 360° while keeping the object upright. However, in real-world scenarios, only varying the azimuth angle without accounting for elevation angles and other variations in visual conditions raises the question of whether the conclusions of this paper could be adequately validated.
  4. This work mainly conducts analyses based on existing models and geometric statistics. Although the experiments reveal several valuable phenomena, what are the innovations of this paper compared with existing studies on Neural Collapse and Manifold Untangling, and can these findings guide subsequent representation learning or model design?
  5. The research is limited to 15 object instances, with analyses performed on ViT-B/16 and ResNet-50. Furthermore, key intervention experiments focus solely on Airplane 13. It is unclear to which extent the experimental conclusions are susceptible to data bias.
Questions for authors

To see weakness.

Reviewer_W2AMOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work investigates the latent geometry learned by large scale pre-trained models under object rotation for rotation invariant recognition.

The analysis is done on feature readouts from both ViT-B/16 and ResNet-50 pre-trained on ImageNet-1k. Input images are generated by rendering 15 objects with 360 rotation around the vertical axis (objects taken from ShapeNet). The paper analyzes the resulting feature embeddings with statistical methods.

In general, I find the experimental setup and presentation of results not sufficient. (see critical weaknesses)

Critical strengths

Investigating the geometry of features under object rotation is an interesting and relevant topic for rotation invariant object classification.

Critical weaknesses
  1. Weak experimental setup: The paper analyzes only two different architectures (ViT-B/16 and ResNet-50) on 15 objects in front of a uniform gray background (lines 164 - 168), which is a very small experimental setup and makes it difficult to see results as general applicable. There is also a core issue with reporting results throughout this paper where there are very often specific objects picked as examples. In Section 4 "Recognition under viewpoint Variation" (airplane 12, chair 02, lamp 11, etc.), or Table 1 (car 03, table 08, etc.) are picked without ever showing how these objects actually look like. Making "airplane 12" not more informative than just saying "an airplane object", and with the low object count one can not generalize that this would be the case for airplanes in general or if it is just this specific object instance. (Even the conclusion lists an example of a single airplane (lines 454 - 455)). Furthermore, the paper reports that the ViT only recognized 2, and the ResNet only 5 objects of 15 correctly (lines 224-226). These pre-trained models have been trained on real images of ImageNet, therefore there is also a factor of this synthetic dataset being out of distribution, which can also result / explain lower accuracy. This was not addressed in the paper, making statements about recognition less impactful.
  2. Presentation of Results: The main presentation of results is in Figures which are too illustrative with cluttered text. Both Figure 1 and 2 are very cluttered with a lot of text and lines that overlap with the plots. There are additional headline sentences on top of the figures, which is not following the ICLR guidelines. Figure 3 is simpler and therefore does not have these overlapping text issues, has no grid lines, making these plots again more illustrative than actually displaying clear readable / comparable data (This also applies to Figure 1 and 2). As there are no additional tables showing raw numerical results, it is difficult to actually interpret / compare these results and one has to rely on numbers provided in the paper's text. Table 1 is also not showing tabular numeric results but rather summarization texts. Also in Figure 1 (d), the purple surface does not illustrate the margin = 0, as it is not intersecting with the dot plots at any point (no occlusion or anything like that), currently it is just a very abstract rectangle in the background.
Questions for authors
  1. Why was such as small object set chosen, even though ShapeNet would have significantly more objects? The analysis only involves model inference, I would therefore assume that compute would not be an issue here?
  2. Out of Distribution: As mentioned in Critical Weaknesses, both ViT-B/16 and ResNet-50 are trained on ImageNet and therefore classifying synthetic renders of objects in front of gray backgrounds could result in wrong classificication due to out of distribution. As detection rates are quite low (lines 224-226), it would be important to check if this is simply caused by the synth to real gap, and not the rotation of the object. For example, shrinking the synth to real gap with more realistic backgrounds, lighting, rendering methods. Or, maybe even real 360 degree images of objects.
  3. Novelty: What exactly sets this work apart from previous related work / what is the main novelty? Currently the related work section / introduction is mainly listing existing work but not describing clearly what this work is doing differently. (lines 045 - 046: "However, previous studies rarely evaluate the network’s predictions for different views of the same object", but this still means they did this already?)
  4. Line 80: "our pretrained ViT", what does "our" refer to here? Based on lines 170 - 172 the models used are pre-trained ImageNet-1k model weights from torchvision, is this a different version?

Minor Notes:

  • It would be good to have some plots or example images which show the actual dataset that has been generated and how these images map into latent space. For example for the rotations, there is currently the description in lines 164 - 168 and no visual examples on how these 360 degree renders looks like.
  • Lines 175 - 179 are a bit confusing. First it introduces the representations taken of final layers of the models as "final representation" but in the sentence directly afterwards the representations are address per index of their relative depth.
Reviewer_stEUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how deep vision networks achieve viewpoint-invariant object recognition and why recognition succeeds on some objects while failing on others. Rather than evaluating representational geometry using label-free metrics or post-hoc probes, the authors evaluate object "view manifolds" directly against the networks' native, fixed linear classification heads z(h)=Wh+bz(h) = Wh + b. The authors formalize three potential geometric routes for a readout to maintain correct classification across views: 1) relative manifold collapse or spread contraction, 2) turning view variation away from readout directions via directional collapse, or 3) shifting the entire manifold location away from the decision boundary.

To evaluate these routes, the authors render 15 ShapeNet objects across 40 azimuth angles on a grey background and pass them through pretrained ViT-B/16 and ResNet-50 models. Overall baseline category recognition is low (ViT recognizes 2 objects; ResNet recognizes 5). Across models, relative manifold collapse is neither necessary nor sufficient for recognition, and final-layer compactness correlates with label stability rather than correctness. Contrary to expectations, view variation is actually more aligned with key readout directions than a random isotropic baseline. Instead, overall recognition correctness is almost entirely determined by the centroid margin mlocm_{\text{loc}}, with a causal ablation showing that shifting a misclassified view set back to its intact centroid restores full recognition. Finally, the paper notes that distance-only geometric summaries cannot certify recognition because rigid translations and rotations alter decision margins without changing internal pairwise distances.

Critical strengths
  • The three proposed routes map cleanly onto how an affine readout succeeds or fails, and the rotation and translation tests show concretely that distance-only summaries miss critical information regarding manifold location and orientation.

  • The finding that viewpoint variation aligns more strongly along decision directions than an isotropic random baseline directly challenges the assumption that networks achieve invariance by rotating viewpoint variance away from readout axes.

Critical weaknesses
  • The empirical study relies on a tiny dataset of 15 synthetic objects from 5 categories, rendered at a single camera elevation on a uniform grey background, and evaluated on just two ImageNet-1k models. Mean category correctness is extremely low (0.390.39 for ViT, 0.460.46 for ResNet), with only 2 objects recognized by ViT and 5 by ResNet. Most classification failures likely stem from the severe domain gap between synthetic grey-background renders and natural ImageNet photos rather than viewpoint variation itself, leaving very little true invariance to study. Furthermore, with n=15n = 15, correlation confidence intervals span approximately ±0.60\pm 0.60 (as acknowledged in L444–448), rendering the statistical claims weak. The causal re-centering ablation is also demonstrated on only a single object in one network.

  • The random-orientation baseline (κnull\kappa_{\text{null}}) assumes an isotropic distribution. However, deep neural feature spaces are highly anisotropic. Observing κ/κnull>1\kappa / \kappa_{\text{null}} > 1 may simply reflect the general anisotropy of deep feature spaces rather than a mechanism specific to viewpoint variation.

  • For an affine classification head, a view's margin between the top two classes decomposes exactly as m(hv)=m(hˉ)+u⊤(hv−hˉ)m(h_v) = m(\bar{h}) + u^\top (h_v - \bar{h}). Consequently, "location" (mlocm_{\text{loc}}) is essentially the mean logit margin, and the other two routes describe its spread. Correlating the mean logit margin with the fraction of views having a positive margin is nearly guaranteed mathematically. Likewise, correlating compactness with label stability merely restates that nearby representations receive the same label. Re-centering restoring recognition is computed exactly through the affine head. Furthermore, the overarching conclusion—that separability does not require collapse and that manifold location matters—is already well-established within manifold capacity theory.

Questions for authors
  • Do the directional alignment findings (κ/κnull>1\kappa / \kappa_{\text{null}} > 1) and the re-centering recovery hold when evaluated at scale (e.g., ≥100\ge 100 objects, multiple camera elevations) and across modern vision backbones (e.g., CLIP, DINOv2, ImageNet-21k ViTs)?

  • Can you provide a formal proof for Proposition 1 and demonstrate that translated or rotated view manifolds can be synthesized by real, valid image inputs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.