acceptodds
Under review as a conference paper at ICLR 2027

SEIS: Subspace-based Equivariance and Invariance Scores for Neural Representations

Abstract

Understanding how neural representations respond to geometric transformations is important for diagnosing robustness, comparing architectures, and selecting models for spatial tasks. Existing measures typically examine output predictions, aggregate individual units independently, or fit an explicit transformation between representations. We introduce SEIS (Subspace-based Equivariance and Invariance Scores), a label-free subspace measure that disentangles equivariance and invariance directly from paired activations of original and transformed inputs, without knowledge of the transformation. SEIS organizes activations into spatial positions and observations, applies a continuous spectral filter that softly downweights weak components, then factorizes the comparison into an observation-space alignment for equivariance and a stricter spatial alignment for invariance, with both scores chance-corrected against null expectations. Controlled experiments show that SEIS cleanly separates the two properties and is stable given sufficient observations. Applied to trained networks, SEIS reveals how representations evolve during training, how augmentation strengthens deeper layers, how decoder architecture and multi-task supervision shape transformation geometry, and how positional encodings affect translation structure in vision transformers. Across a diverse pool of ImageNet-pretrained classifiers, SEIS predicts accuracy retention under affine transformation better than validation accuracy, without labels.

Primary area:
interpretability and explainable AI
Keywords:
Equivariance, Invariance, Neural Representations, Subspace Analysis, Geometric Transformations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_abmSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper the authors introduce SEIS, a novel subspace-based metric for estimating invariance and equivariance of neural representations without external labels. The metric separates the neural activations into "spatial" and "observational" axes and assembles them in matrix-shaped form. An SVD decomposition is then used to provide directions weighting and compute both invariance and equivariant agreement scores. The author then compute the final metric by a final chance-correction step that accounts for the expected random representation overlap. The authors then first validate the metric on a synthetic benchmark of known transformations and later deploy it to study the neural representations in different architecture and learning regimes: they show that both invariance and equivariance build-up during training, that augmentation mainly operates on deeper layers and that both loss and architecture decision influence the final score. Finally, they measure the correlation between the label-free estimation of representation equivariance and model performance on affine-transformed inputs.

Critical strengths

The authors introduce a novel metric to estimate invariance and equivariance of neural representations which is easy to understand and efficient to compute, while requiring no external labels and across the networks' depth. This could be a valuable addition to the toolkit available to study network representations, especially as they evolve during training and architecture choices. The authors then offer several use-cases for the application of the novel metric to the study of representation in various networks and tasks to illustrate this point. Lastly, the paper is generally very well written and easy to follow, with clear illustrations.

Critical weaknesses

One important limitation of the current work is a lack of comparisons between the authors' proposed metric and the alternatives beyond the synthetic benchmark of Fig.1. The authors should report, when feasible, the equivalent scores obtained with the alternative technique to assess whether they agree of not, and in case discuss these differences and the relative advantages. A second important aspect that is missing is a more careful characterisation of the metric as a function of the transformation severity: does the metric report Sinv∼0\mathcal{S}_\mathrm{inv} \sim 0 even for rotations of 1∘1^\circ? Or does it offer more gradual responses? The current synthetic benchmark conflates this distinction and only reports the overall score distribution across transformation type.

Questions for authors

Below are some relevant questions for the authors. I would be willing to increase my score past acceptance level even if only part of the following list is adequately addressed.

  1. In Fig. 1a, what is the expected ground-truth score/distribution?
  2. In Fig.1a authors tests several transformations, however it remains unclear whether their metric is capable of tracking the severity of the transformation, as the chosen visualisation only shows the overall spread. However it would have been interesting to see whether the metric was capable of assigning calibrated graded scores for invariance separating near-identity transformations (e.g. scaling ~0.95, rotation ~5°) from more severe changes. Having a smaller variability than Lenc-Vedaldi does not automatically mean a better metric in this regard. (Fig. 8 partially reports the scores for different rotations [90°, 180°, 270°], but critically misses the ~identity range)
  3. In Fig.1b, can the author report the convergence curves for the other transformations?
  4. Related to Fig. 2, can the author report the per-layer curves for both Sequiv\mathcal{S}_\mathrm{equiv} and Sinv\mathcal{S}_\mathrm{inv}? This would directly support the authors claim that "[...] deep representations converge progressively from lower to higher layers during training".
  5. Why did the author chose to report ∣ΔS∣| \Delta \mathcal{S} | instead of directly S\mathcal{S}? This would have allowed for example in Fig.9 to directly compare whether the from-scratch models achieved comparable level of invariance/equivariance relative to their pre-trained counterparts.
  6. In Fig. 5 (but also elsewhere), can the author report the error-bars (std/sem) over the explored 5 seeds in their visualisations to assess whether the observed difference between positional encoding schemes is significant.
  7. In Fig. 5, can the authors discuss why they think the NoPE configuration seems to perform worse despite the fact that no positional information is provided, thus forcing the transformer to use local image context similarly to CoPE?
  8. In Fig. 6b,d there seems to be a visual split between two groups of "good performer" (high AtransA_\mathrm{trans} or Atrans−AcleanA_\mathrm{trans} - A_\mathrm{clean} with relatively lower AvalA_\mathrm{val}) and "bad performer" (the opposite), which lowers the overall correlation between the validation score and the model performance. Do the authors observed some model characteristics that might explain this fact? (And can the authors also report the correlation between Sequiv\mathcal{S}_\mathrm{equiv} and AvalA_\mathrm{val}?)
  9. Can the authors report what the alternative metrics (L-V, G-EED) measure for their use-case experiments (e.g Fig. 3, Fig.4, Fig.5)? Do the metrics agree in the reported invariances?
  10. In Fig.3, the fact that both invariance and equivariance decrease in the initial layers and then increase in deeper ones is reminiscent to the fact identified in previous works (Ansuini et al. NIPS 2019; Muratore et al. NIPS 2022; Valeriani et al. NIPS 2023) that the intrinsic dimension of the data manifold first increases and then decreases, which was directly related to the initial suppression of "unwanted" regularities (e.g. luminance/contrast) followed by a subsequent buildup of more abstract and task-relevant invariances (e.g. cat/dog). It would be perhaps interesting for the authors to discuss this potential connection in their work.
4 comments

We thank the reviewer for their thorough review and valuable feedback. Below, we address each question point by point.

Q1. Ground-truth interpretation for Fig. 1a

The synthetic construction provides the following expectations: (1) Identity: both scores should be 1; (2) Small noise: both scores should remain near 1; (3) Spatial transformations: Sequiv=1\mathcal{S}_\mathrm{equiv}=1, there is no universal scalar ground truth for Sinv\mathcal{S}_\mathrm{inv}, near-identity transformations should score close to 1, while stronger transformations should produce lower Sinv\mathcal{S}_\mathrm{inv} ; (4) Independent activations: both scores should be 0.

Q2. Transformation severity for Fig. 1a

Thank you for pointing it out. For the new experiment, we varied rotation from 0 to 180, scaling by up to 20%, and translation by up to 15% of image width, using 50 trials per severity. SEIS invariance decreases consistently with transformation severity, with Spearman correlations of ρ\rho=-0.98 for rotation and ρ\rho=-1 for both translation and scaling. These results confirm that SEIS provides a graded response, including near identity.

Lenc–Vedaldi also exhibits a graded response but requires learning a transformation-specific latent map, whereas G-EED is less sensitive to increasing severity and requires prior knowledge of the transformation action to construct the expected equivariant target. SEIS requires neither.

We will add these results into the revised paper.

Q3. Convergence across transformations for Fig. 1b

We will add separate convergence curves for translation, scaling, and rotation. The same convergence pattern holds for all three transformations. Our practical recommendations remain unchanged.

Q4. Per-layer training curves for Fig. 2

Thank you for this suggestion. We initially omitted the per-layer curves because plotting every layer would introduce too many overlapping lines and reduce readability. In the revised paper, we will group layers by network blocks to provide a clear view of how the representations evolve from shallower to deeper blocks during training.

Q5. Why report ∣ΔS∣|\Delta\mathcal{S}| instead of S\mathcal{S} in Fig. 2

We report ∣ΔS∣|\Delta\mathcal{S}| because this section asks how much training changes each representation relative to its own initialisation. This is appropriate when comparing pertained and scratch models, whose initial scores may differ substantially. Comparing raw S\mathcal{S} would mix the model’s initial scores and the change caused by training.

Q6. Error bars over seeds in Fig. 5

Thank you for this suggestion. We will redraw Fig. 5 with mean and SD across five seeds. The resulting error bars are generally small and preserve the qualitative ordering across transformer blocks. Conditional PE yields the highest Sequiv\mathcal{S}_\mathrm{equiv} and Sinv\mathcal{S}_\mathrm{inv}, Relative PE is intermediate, and Absolute/No PE are lower.

Q7. Why can NoPE perform worse than CPE in Fig. 5?

The premise that removing positional encoding forces the transformer to use local context is not correct. Without positional encoding, self-attention remains global, so it cannot distinguish neighbouring from distant patches. CPE instead uses a shared depth-wise convolution to encode local spatial information while preserving translation structure.

Q8. Two model groups in Fig. 6 and Sequiv\mathcal{S}_\mathrm{equiv}–AvalA_{\mathrm{val}} correlation

Thank you for this observation. We analysed the split separately in both Fig. 6b and Fig. 6d. In both figures, a two-component Gaussian mixture is preferred to a single component. However, we do not find that the split can be explained by architecture type. Instead, we found out that models with similar validation accuracy can differ greatly in transformation equivariance. The higher- and lower-robustness components have mean Sequiv\mathcal{S}_\mathrm{equiv} of 0.78 and 0.57, respectively.

The Spearman between Sequiv\mathcal{S}_\mathrm{equiv} and AvalA_{\mathrm{val}} is 0.63, this moderate relationship confirms that the indicators share some signal but are not redundant. SEIS is not merely reproducing validation accuracy, it can be regarded as a complementary information when both indicators are available, and enables label-free robustness when validation labels are unavailable.

Q9. Lenc–Vedaldi and G-EED on real-network use cases

Thank you for your suggestion. We applied Lenc–Vedaldi and G-EED to the same models and transformations for comparison.

For Fig. 3, both baselines confirm that augmentation strengthens deep-layer equivariance and invariance.

For Fig. 4 & Tab. 1: Within each architecture, both baseline rankings correlate positively with SEIS. Across architectures, SEIS favours UNet for both scores, Lenc–Vedaldi favours FCN, while G-EED favours UNet for equivariance but FCN for invariance. We additionally correlated every score with mIoU as shown in table below. SEIS is the only method for which both equivariance and invariance are positively associated with mIoU. Its advantage here is the consistent behaviour of both disentangled scores across architectures. This connection is consistent with prior segmentation work showing that equivariance/invariance can provide useful pixel-level inductive biases.

Methodsrrρ\rho
SEISinv\mathrm{SEIS}_{\mathrm{inv}}0.920.88
SEISequiv\mathrm{SEIS}_{\mathrm{equiv}}0.930.87
Lenc–Vedaldiinv\mathrm{Lenc\text{–}Vedaldi}_{\mathrm{inv}}-0.81-0.59
Lenc–Vedaldiequiv\mathrm{Lenc\text{–}Vedaldi}_{\mathrm{equiv}}-0.58-0.34
G-EEDinv\mathrm{G\text{-}EED}_{\mathrm{inv}}-0.83-0.66
G-EEDequiv\mathrm{G\text{-}EED}_{\mathrm{equiv}}0.940.87

For Fig. 5, Lenc–Vedaldi reproduces exactly the SEIS ordering for both scores. G-EED also strongly agrees.

For Fig. 6, we evaluated Lenc–Vedaldi and G-EED using the same 50 models, same ImageNet validation images, final spatial representations, and fixed affine transformation as Sec. 4.5.

MethodsAtransA_{\mathrm{trans}}AtransA_{\mathrm{trans}}-AcleanA_{\mathrm{clean}}
AvalA_{\mathrm{val}}0.90/0.920.62/0.77
SEISequiv\mathrm{SEIS}_{\mathrm{equiv}}0.74/0.800.77/0.82
Lenc–Vedaldiequiv\mathrm{Lenc\text{–}Vedaldi}_{\mathrm{equiv}}0.60/0.660.61/0.70
G-EEDequiv\mathrm{G\text{-}EED}_{\mathrm{equiv}}0.14/0.230.05/0.17

Each cell reports Pearson (rr)/Spearman (ρ\rho). AvalA_\mathrm{val} best predicts absolute transformed accuracy, while SEIS is the strongest representation-based indicator for AtransA_\mathrm{trans} and the strongest overall indicator for accuracy retention. We therefore interpret SEIS as providing complementary label-free robustness information.

We will add these comparisons in the revised paper with clear visualisations.

Q10. Connection to intrinsic dimension

Thank you for this interesting connection. However, we believe they measure different properties. Intrinsic dimension quantifies manifold complexity, while SEIS measures alignment between the paired manifolds. Therefore, low intrinsic dimension does not imply high equivariance or invariance, or vice versa. For example, a high-dimensional feature map can be perfectly equivariant if it simply shifts with the input.

I thank the authors for the detailed rebuttal and the new experiments that have addressed most of my concerns (e.g. per-layer dynamics, baseline comparisons etc...), although some of the mentioned results like the Lenc-Vedaldi/G-EED comparisons mentioned in the rebuttal (Fig.3 and 5) are not in the revised paper (or I couldn't locate them). I have raised my score to 3 to reflect the improved manuscript.

Thank you for taking the time to review our rebuttal and revised manuscript. The comparisons have now been incorporated into the latest revision. The augmentation comparisons appear in Fig. 3, and the positional-encoding comparisons corresponding to Fig. 5 appear in Appendix H.

Reviewer_EfWfOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a measure for invariance and equivariance of latent representations of 2D domain inputs to spatial transformations. This measure is based on the responses to an input and its transformed version, without additional knowledge on the label or the transformation, by comparing the alignment of spectral filterations of the activations of the two inputs. Empirically, the paper provides an analysis of equivariance and invariance properties of ConvNet and ViT architectures, including their evolution during training and variation across model depth. The equivariance measurement is shown to be correlated with accuracy degradation of ImageNet-pretrained models under an affine transformation.

Critical strengths
  1. The proposed method separately measures invariance and equivariance of internal representations without requiring labels, transformation details, or fitting a transformation-specific mapping. This makes it convenient to apply when the corresponding transformation of the activations is unknown.

  2. The empirical evaluation covers several architectures and training settings, including classification and segmentation models, data augmentation and positional encoding.

  3. The correlation of the equivariance score with accuracy degradation suggests a possible practical application, e.g. on unlabeled data.

Critical weaknesses
  1. The scope beyond 2D spatial representations is not entirely clear. The abstract refers to “geometric transformations,” which can be read as covering a more general setting, while the formulation assumes a spatial action and the evaluation focuses on 2D grids. The applicability to other modalities or more general latent transformation actions should be clarified.

  2. While the empirical analysis covers a wide range of settings, it is not sufficiently clear which findings provide new insights beyond confirming established observations. For example, augmentation improving learned equivariance was already demonstrated in [1], and maintaining translation equivariance is one of the motivations of the original CPE paper [2]. The authors should distinguish between experiments confirming such observations with their proposed measure and experiments that use it to provide new insights. Comparisons with existing measurements in these experiments would also help assess the added value of SEIS.

  3. I find several technical issues with the experiments and overall evaluation of the method.

    1. Section 4.1 says Fig. 1 shows that geometric transformations reduce invariance "according to transformation severity." The notion of severity is not clearly shown in Fig. 1, which shows distributions of the invariance score for each transformation type, rather than a controlled variation in severity.
    2. Section 4.5 shows that the equivariance score is correlated with accuracy degradation under transformation, but it has several limitations:
      1. The paper does not provide any comparison with other equivariance measurements in their correlation with accuracy degradation. As this correlation is mentioned as one of the main contributions of the paper, such a comparison would be helpful to assess the proposed method compared to other similar measurements.
      2. The only compared baseline is the validation accuracy (AvalA_{val}). While it is numerically less correlated with accuracy degradation, there seems to be a separation into two clusters in Fig. 6(d), where a linear curve might fit each subset better separately. This raises the question of whether some property explains this separation (e.g. model size, architecture, augmentations, etc.) and whether that property together with AvalA_{val} would constitute a better predictor of accuracy degradation.

[1] What Affects Learned Equivariance in Deep Image Recognition Models, Bruintjes et al. 2023

[2] Conditional Positional Encodings for Vision Transformers, Chu et al. 2023.

Questions for authors
  1. Is the method limited to spatial transformations of 2D grid representations, or can it also capture transformation structure in other modalities and more general latent transformation actions?
  2. Can the authors compare the correlation with accuracy degradation against other equivariance measurements?
  3. Can the authors investigate the apparent clusters in Fig. 6(d), below and above the dashed curve, and make the list of the evaluated models and their measurements available?
  4. As a possible extension to the analysis in Section 4.5, can SequivS_{equiv} measured before or during training (before completion) predict accuracy degradation at the final checkpoint?
2 comments

W1 & Q1. Scope beyond 2D spatial grids

Thank you for pointing it out. We confirm that SEIS is not restricted to 2D spatial grids and we will clarify this at the beginning of Sec. 3.1. The current example was chosen to make it easier for readers to understand. The same construction applies when a transformation acts on any number of axes. We combine all transformed axes into the positional dimension and all remaining axes into the observation dimension. For example, for 3D activations of shape B×C×D×H×WB\times C\times D\times H\times W, we use p=DHWp=DHW positions and n=BCn=BC observations. We can report the 3D results in the revised paper if helpful. The G-CNN experiment in Appendix D provides another example, where the group and image coordinates are jointly treated as the positional dimension.

W2. Novelty relative to prior observations

We agree that our experiments serve two different purposes: validation against established behaviour and new analysis enabled by SEIS. We separate them explicitly below and will make this distinction clear in the revised paper.

  • Methodological novelty. Our main contribution is SEIS itself (Sec. 3). To our knowledge, it is the first subspace-based framework that separately quantifies invariance and equivariance of internal representations without labels, a specified transformation action, or a fitted transformation-specific map. This contribution is not merely a conventional similarity statistic. SEIS preserves the spatial axis through its matricization (Sec. 3.1), continuously weights reliable spectral components (Sec. 3.2), factorises observation-space and spatial alignment to distinguish equivariance from invariance (Sec. 3.3), and corrects both scores for chance alignment (Sec. 3.4; Appendix C).

  • Sanity checks. We do not intend to claim these underlying phenomena as new.

    • The controlled transformations (Sec. 4.1, Fig. 1a) and D4D_4-equivariant G-CNN (Appendix D) verify that SEIS distinguishes identity, invariance, structured spatial reorganisation, and unrelated representations.
    • The CIFAR-100 classification experiment (Fig. 3a) confirms the established observation that affine augmentation strengthens invariance and equivariance.
    • The positional-encoding experiment (Sec. 4.4, Fig. 5) confirms the intended translation-preserving behaviour of CPE relative to absolute PE.
  • New insights. SEIS additionally reveals behaviour not provided by those prior observations.

    • In segmentation, both scores decrease through the encoder and recover in the decoder, while augmentation has little influence on the earliest encoder layers but consistently increases both scores across the decoder (Fig. 3b). To our knowledge, prior internal-representation measures have been studied primarily in classification and have not characterised how invariance and equivariance evolve through dense-prediction encoders and decoders.
    • Training from scratch produces substantially larger representation changes than fine-tuning a pretrained encoder (Fig. 2b; Appendix E).
    • The effect of task weighting depends on decoder architecture: stronger relative segmentation supervision increases both scores for UNet but has little effect on FCN (Fig. 4). Both Sinv\mathcal{S}_\mathrm{inv} and Sequiv\mathcal{S}_\mathrm{equiv} are strongly associated with mIoU (Table 1).
    • Within the CPE experiment, new findings beyond [2]. [2] motivates CPE through translation equivariance by design and downstream accuracy, but does not measure layer-wise representation structure. SEIS shows that (i) CPE's advantage appears mainly in invariance and grows with depth, (ii) RPE is the second strongest, and (iii) absolute PE is nearly indistinguishable from no PE.
    • Across 50 ImageNet-pretrained classifiers, Sequiv\mathcal{S}_\mathrm{equiv} correlates more strongly with affine accuracy retention than validation accuracy (Sec. 4.5, Fig. 6d), despite requiring no class labels.
  • New comparison with existing measures. We agree that real-network comparisons are important and have now evaluated Lenc–Vedaldi and G-EED on the same models and transformations. Both reproduce the augmentation trend in Fig. 3 and the positional-encoding ordering in Fig. 5, supporting the validity of SEIS. In the extension of Table 1, SEIS is the only method for which both equivariance and invariance are strongly and positively associated with mIoU across architectures.

    Methodsr(Sinv,mIoU)r(\mathcal{S}_{\mathrm{inv}},\mathrm{mIoU})r(Sequiv,mIoU)r(\mathcal{S}_{\mathrm{equiv}},\mathrm{mIoU})ρ(Sinv,mIoU)\rho(\mathcal{S}_{\mathrm{inv}},\mathrm{mIoU})ρ(Sequiv,mIoU)\rho(\mathcal{S}_{\mathrm{equiv}},\mathrm{mIoU})
    SEIS0.920.930.880.87
    Lenc–Vedaldi-0.81-0.58-0.59-0.34
    G-EED-0.830.94-0.660.87

We will add these comparisons and revise the paper to distinguish clearly between validation experiments and findings that demonstrate SEIS’s added value.

W3.1. Transformation severity for Fig. 1a

Thank you for pointing it out. For the new experiment, we varied rotation from 0 to 180, scaling by up to 20%, and translation by up to 15% of image width, using 50 trials per severity. SEIS invariance decreases consistently with transformation severity, with Spearman correlations of ρ=−0.98\rho=-0.98 for rotation and ρ=−1\rho=-1 for both translation and scaling. These results confirm that SEIS provides a graded response, including near identity.

Lenc–Vedaldi also exhibits a graded response but requires learning a transformation-specific latent map, whereas G-EED is less sensitive to increasing severity and requires prior knowledge of the transformation action to construct the expected equivariant target. SEIS requires neither. We will add these results into the revised paper.

W3.2.1 & Q2. Comparison with alternative measures for prediction of affine robustness

Thank you for your suggestion. We evaluated Lenc–Vedaldi and G-EED using the same 50 models, same ImageNet validation images, final spatial representations, and fixed affine transformation as Sec. 4.5 and Fig. 6.

MethodsAtransA_{\mathrm{trans}}AtransA_{\mathrm{trans}}-AcleanA_{\mathrm{clean}}
AvalA_{\mathrm{val}}0.90/0.920.62/0.77
SEISequiv\mathrm{SEIS}_{\mathrm{equiv}}0.74/0.800.77/0.82
Lenc–Vedaldiequiv\mathrm{Lenc\text{–}Vedaldi}_{\mathrm{equiv}}0.60/0.660.61/0.70
G-EEDequiv\mathrm{G\text{-}EED}_{\mathrm{equiv}}0.14/0.230.05/0.17

Each cell reports Pearson (rr)/Spearman (ρ\rho). AvalA_\mathrm{val} best predicts absolute transformed accuracy, while SEIS is the strongest representation-based indicator for AtransA_\mathrm{trans} and the strongest overall indicator for accuracy retention. We therefore interpret SEIS as providing complementary label-free robustness information. We will add this comparison as an extension of Fig. 6.

W.3.2.2 & Q3. Two model groups in Fig. 6

Thank you for this observation. We analysed the split separately in both Fig. 6b and Fig. 6d. In both figures, a two-component Gaussian mixture is preferred to a single component. However, we do not find that the split can be explained by architecture type. Instead, we found out that models with similar validation accuracy can differ greatly in transformation equivariance. The higher- and lower-robustness components have mean Sequiv\mathcal{S}_\mathrm{equiv} of 0.78 and 0.57, respectively.

Q4. Early prediction of final robustness

This is an interesting extension. Our training-dynamics experiment (Sec. 4.2, Fig. 2) shows that SEIS changes rapidly early in optimisation and then stabilises, suggesting that early prediction may be feasible. However those plots characterise representation evolution within individual runs, and they do not establish that an early score predicts final robustness across models. We therefore limit our current claim to label-free comparison of completed models and leave early robustness prediction to a controlled study with matched training trajectories. We agree that this is a valuable direction for future work.

We thank the reviewer again for all the constructive feedback, which helped us clarify the scope and strengthen the empirical evaluation.

Reviewer_dpWdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes two scores intended to measure equivariance and invariance in internal neural representations without requiring labels or fitting a transformation-specific mapping. The method arranges paired activations into matrices of spatial positions and observations, applies centering and spectral filtering, and compares their observation structure and spatial alignment. Both scores include corrections based on expected chance overlap.

The paper validates the scores using synthetic transformations, experiments varying the number of observations, and an untrained group-equivariant CNN. It then applies SEIS to study training dynamics, data augmentation, segmentation architectures and supervision, and positional encodings in vision transformers. The final experiment evaluates 50 pretrained classifiers. The equivariance score, computed using ImageNet representations, correlates with accuracy change under a fixed affine transformation on ImageNetV2 at r=0.77, compared with r=0.62 for ImageNet validation accuracy. The proposed contribution is a practical diagnostic for examining transformation responses inside networks and potentially comparing robustness when labels are unavailable.

Critical strengths
  1. The method addresses a useful practical problem: examining how representations respond to transformations without requiring labels or learning a separate mapping for each transformation. The controlled experiments provide evidence that the scores behave as intended in several known cases. Independent representations receive scores near zero, while the untrained group-equivariant CNN receives high equivariance and low invariance scores. The chance-correction ablation is particularly useful because it shows how unrelated representations can otherwise receive high scores when observations are limited. These experiments give the proposed diagnostic a reasonable empirical foundation, although the interpretation concerns discussed below still need to be addressed.

  2. The experiment across 50 pretrained classifiers is the strongest evidence of potential practical value. A score computed from representations on ImageNet is associated with accuracy preservation on ImageNetV2, suggesting that it may capture information relevant to robustness across models. Evaluating the performance relationship on a separate dataset strengthens this result. If the association remains convincing after comparison with other indicators that require no labels and after controlling for architecture and clean accuracy, SEIS could be useful for model comparison when labeled evaluation data are unavailable.

Critical weaknesses
  1. I have concerns about how the scores should be interpreted, particularly the invariance score. Eq. 8 squares the spatial inner product, so it appears to assign maximal invariance to representations that are negatives of each other after centering. If Y_centered = -X_centered, then Z_Y = -Z_X and S_inv = 1, although the representation has changed. A simple example uses nonnegative activations: X = [[2, 0, 1], [0, 2, 1]] and Y = [[0, 2, 1], [2, 0, 1]]. Here Y is a spatial row swap of X, and position-wise centering makes the two matrices negatives of each other. This seems inconsistent with the stated interpretation of invariance. The paper does not examine how often negative spatial alignment occurs or whether it affects the experimental conclusions.

The scope of the equivariance score also needs clarification. The method assumes a shared spatial action while keeping channel correspondence fixed. It is unclear how the score should be interpreted when a transformation also permutes or mixes channels. Appendix D includes the known group axis in the positional dimension, suggesting that the choice of matricization matters for recognizing the intended action.

The relationship to CKA should also be explained more directly. The normalized observation statistic has the same form as linear CKA applied to the transposed filtered representations. This does not remove the potential contribution of SEIS, but it makes comparison with CKA using the same matricization important for understanding what the filtering and chance correction add.

  1. The experimental evidence is promising, but the comparisons and reporting are not yet sufficient to establish the practical advantage of SEIS. Section 4.5 compares the score only with validation accuracy. Comparisons with effective invariance, prediction agreement, feature cosine similarity, and unfiltered CKA would more directly test its value as a diagnostic that requires no labels. The reported correlations of 0.77 and 0.62 also need a joint uncertainty estimate because they use the same models and outcome. Analyses within architecture families and controls for clean accuracy would help establish whether the association extends beyond differences between model families.

Several evaluation details are missing or unclear. Section 4.1 recommends matching pooling policy and measured resolution across models, but Section 4.5 does not explain how this was handled. Scoring batch sizes and observation-to-position ratios are also needed to assess comparability. In Sections 4.2 and 4.3, it is unclear whether all images contributing to a scored matrix receive the same affine transformation. If each image receives a different transformation, the shared spatial action assumed by the method may no longer apply to the combined matrix.

Finally, the small differences in Section 4.3 are difficult to assess without numerical uncertainty across seeds. The correlation with mIoU in Table 1 pools architectures and supervision settings, so it would be useful to know whether the relationship also holds within each architecture. These omissions leave me uncertain about how much of the reported behavior reflects representation properties and how much depends on the evaluation setup.

Questions for authors
  1. Is S_inv = 1 for representations that become negatives of each other after centering intended? Please explain how this fits the definition of invariance, report how often S_sp < 0 occurs in the real-network experiments, and show whether a sign-preserving alternative changes the main conclusions. If the score is modified, please also explain any implications for its chance correction.

  2. What kinds of latent transformation actions should SEIS reliably recognize? In particular, how should the score be interpreted when the action includes channel permutations or mixing? A controlled example with a known spatial action and channel permutation would help. Please also explain the relationship between the normalized observation statistic and linear CKA, and compare SEIS with unfiltered CKA using the same matricization on representative real-network experiments.

  3. In Sections 4.2 and 4.3, is one affine transformation shared across all images contributing to a scored matrix? If transformations are sampled independently per image, please explain how this relates to the shared spatial action assumed in Section 3.1 and report whether using a shared transformation changes the results.

  4. Does the Section 4.5 result hold against other indicators that require no labels, such as effective invariance, prediction agreement, feature cosine similarity, and unfiltered CKA? Please provide an uncertainty estimate for the difference in correlations and check whether the association persists within architecture families and after controlling for clean accuracy. Please also report the model list, scoring batch sizes, spatial resolutions, pooling policy, and n/p ratios so that the fairness of the comparison can be assessed.

  5. Are the smaller differences in Sections 4.3 and 4.4 consistent across seeds? Please provide numerical uncertainty for the comparisons discussed in the text. Section 4.4 states that standard deviations are reported, but their magnitude is difficult to assess from Figure 5. For Table 1, do the correlations with mIoU also hold within each architecture?

2 comments

We thank the reviewer for the careful assessment. Below, we clarify the questions and report new experiments addressing the reviewer's concerns.

Q1. Sign ambiguity of Sinv\mathcal{S}_\mathrm{inv}

Thank you for identifying this edge case. The reviewer is correct because SEIS measures unoriented subspace alignment, Y=−XY=-X spans the same subspace and receives Sinv=1\mathcal{S}_\mathrm{inv}=1. We will clarify that the sign equivalence is a property of the subspace definition.

This case does not affect our empirical conclusions. Across all real-network activation pairs, we observed no negative spatial alignment (Ssp<0\mathcal{S}_\mathrm{sp} < 0). We also evaluated a sign-preserving variant, the maximum score change was 2.27×10−72.27\times10^{-7}, with all rankings and conclusions unchanged.

Q2. Recognized actions and the relationship to CKA

SEIS takes X,Y∈Rp×nX, Y \in \mathbb{R}^{p \times n} with spatial positions as rows and observations as columns. The latent action SEIS tests is a transformation of the row axis, Y≈TgXY \approx T_g X with an unknown Tg∈Rp×pT_g \in \mathbb{R}^{p \times p}.

In the paper’s spatial analysis, p=HWp=HW and n=BCn=BC. To test the reviewer’s channel-permutation case, we instead ask how channels transform and matricize the same tensor as p=Cp=C, n=BHWn=BHW. Here, pp should be understood as the axis on which the transformation acts (not necessarily physical position), and nn contains repeated observations of that action. SEIS gives Sinv=0.01\mathcal{S}_\mathrm{inv}=0.01 and Sequiv=1.0\mathcal{S}_\mathrm{equiv}=1.0. The representation is strongly equivariant but not invariant, exactly as expected when its channel coordinates are consistently reorganised.

Relationship to CKA. The reviewer is correct that the normalized observation-alignment term has the form of linear CKA applied to the transposed filtered representations. SEIS additionally adds:

  • Spectral reweighting. The kernel eigenvalues are wj2∈(0,1)w_j^2 \in (0,1) instead of sj2s_j^2 (CKA) or 11 (CCA). This reduces CKA's dominance by top components while suppressing noisy ones, with no chosen rank or threshold.
  • Chance correction. E[Sobs/κ]=τ/(dκ)\mathbb{E}[\mathcal{S}_{\mathrm{obs}}/\kappa] = \tau/(d\kappa) is an analytic null that accounts for the spectrum, plus the margin δd\delta_d. Raw CKA is positively biased when nn is small. Appendix C shows the uncorrected Sequiv\mathcal{S}_{\mathrm{equiv}} reaches 0.96 for independent data at n/p=0.05n/p=0.05.
  • A spatial term. CKA is invariant to orthogonal transformations of the feature axis, here space, so on its own it cannot separate invariance from equivariance like SEIS. This is why Sobs\mathcal{S}_{\mathrm{obs}} serves as the equivariance signal. Ssp\mathcal{S}_{\mathrm{sp}} instead provides the invariance score.

For 50-model experiment in Sec. 4.5, we used identical images, transformations, representations, and matricization across all 50 models. CKA correlates with Atrans−AcleanA_\mathrm{trans}-A_\mathrm{clean} at r/ρ=0.74/0.83r/\rho=0.74/0.83, while Sequiv\mathcal{S}_\mathrm{equiv} gives 0.77/0.820.77/0.82. Thus, CKA and SEIS have comparable predictive association.

We will state the identity and ablation in the revised paper.

Q3. Shared transformation across images

In our experiments, one affine transformation is sampled per evaluation repeat and applied identically to every evaluated image. Transformations are not sampled separately for individual images. Activations from all inference batches are combined into one matrix before computing SEIS. The same paired inputs are reused across layers and seeds. Therefore, our experiments already satisfy the shared spatial transformation.

Q4. Label-free baselines, controls, and evaluation details

We evaluated the requested label-free baselines using the same 50 models, images, affine transformation, and evaluation protocol as Sec. 4.5. Their correlations with accuracy retention (Atrans−AcleanA_{\mathrm{trans}}-A_{\mathrm{clean}}) are:

MethodsPearson (rr) / Spearman (ρ\rho)
Prediction agreement0.95/0.95
SEISequiv\mathrm{SEIS}_\mathrm{equiv}0.77/0.82
CKA0.74/0.83
Feature cosine similarity0.71/0.78
Effective invariance0.63/0.60

Prediction agreement is the strongest label-free predictor in this experiment. However, prediction agreement and SEIS answer different questions and SEIS remains useful:

  1. Prediction agreement compares a model’s final outputs. For dense prediction, those outputs must also be aligned to the same spatial frame. SEIS can instead analyze any internal layer directly, without requiring classifier outputs or a known alignment map (Secs. 4.2–4.4).
  2. Prediction agreement cannot identify where transformation structure is gained or lost within a network. SEIS provides layer-wise analyses of how this structure changes during training and with augmentation, decoder architecture, and positional encoding.
  3. Prediction agreement cannot distinguish between invariance and equivariance properties, while SEIS can.

Following the reviewer's request, we quantified uncertainty in the difference between SEIS and validation accuracy correlation:

ComparisonΔr\Delta r [95% CI]Δρ\Delta \rho [95% CI]
Sequiv\mathcal{S}_{\mathrm{equiv}} vs. AvalA_{\mathrm{val}}0.15[−0.03, 0.36]0.15 [-0.03,\,0.36]0.06[−0.10, 0.22]0.06 [-0.10,\,0.22]

Because both intervals include 0, we interpret SEIS as providing complementary information rather than claiming statistically superiority over AvalA_{\mathrm{val}}. SEIS remains associated with accuracy retention after controlling for clean ImageNetV2 accuracy r(Sequiv, Atrans−Aclean∣Aclean)=0.63r(\mathcal{S}_{\mathrm{equiv}},\ A_{\mathrm{trans}} - A_{\mathrm{clean}} \mid A_{\mathrm{clean}}) = 0.63.

The association also remains positive within broad architecture groups: r/ρ=0.76/0.83r/\rho=0.76/0.83 for CNNs, 0.91/0.930.91/0.93 for ViT models, and 0.95/0.930.95/0.93 for mixer-like models.

We will provide the model list. Inference uses batch size 32, but all activations are concatenated before scoring. We evaluate native final pre-pooling feature maps using 1,000 images, giving p=49–196p=49\text{–}196 spatial positions, n=192,000–2,048,000n=192{,}000\text{–}2{,}048{,}000 observations, and n/p=980–41,796n/p=980\text{–}41{,}796.

Q5. Numerical uncertainty and within-architecture results

For the final CIFAR-100 layer in Sec. 4.3, no-augmentation versus augmentation gives 0.26±0.020.26\pm0.02 vs. 0.60±0.010.60\pm0.01 for invariance and 0.37±0.030.37\pm0.03 vs. 0.71±0.010.71\pm0.01 for equivariance (mean ±\pm SD). The augmentation increase occurs in all seeds. We will redraw figures to report SD.

For the final ViT block in Sec. 4.4, CPE has the highest invariance and equivariance in every seed. We will redraw this figure to report the SD.

The Table 1 association also remains positive within each architecture. For FCN, r/ρ=0.70/0.60r/\rho=0.70/0.60 for invariance and 0.73/0.490.73/0.49 for equivariance. For UNet, the corresponding values are 0.50/0.430.50/0.43 and 0.63/0.430.63/0.43.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Changes

We thank all reviewers for their constructive feedback. The main changes are summarised below and highlighted in blue in the revised paper.

  • Sec. 3.1: Generalised the matricization beyond 2D grids to shared actions on spatial positions, tokens, or voxels.
  • Sec. 3.2 and Sec. 3.4: Clarified the relationship to linear CKA, and distinguished SEIS through continuous spectral weighting, spatial alignment, and chance correction.
  • Sec. 4.1, Fig. 1, and Appendix E: Added controlled transformation-magnitude results showing that SEIS provides graded invariance responses to translation, scaling, and rotation. Report convergence for transformations beyond translation in Fig. 1b.
  • Sec. 4.2, Figs. 3, and Appendix F: Added per-layer training trajectories and comparisons with Lenc–Vedaldi and G-EED for classification and segmentation under affine augmentation. SD across evaluations is visualized.
  • Sec. 4.3 and Table 1: Added comparisons with Lenc–Vedaldi and G-EED, supporting the suitability of SEIS for cross-architecture comparison of invariance and equivariance.
  • Sec. 4.4, Fig. 5, Appendix H, and Fig. 12: Added matched positional-encoding comparisons with Lenc–Vedaldi and G-EED. SD across evaluations is visualised.
  • Sec. 4.5, Fig. 6, and Table 2: Expanded the comparison of label-free indicators for predicting accuracy retention under affine transformation.
  • Appendix I and Table 3: Added the complete list of the 50 pretrained models used in the affine-robustness evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.