acceptodds
Under review as a conference paper at ICLR 2027

Measuring Learned Equivariance: An Information Theoretic Approach

Abstract

Learning symmetries through data augmentation offers an alternative to architectural equivariance, but does not guarantee that a network respects those symmetries. Measuring learned equivariance is especially difficult for intermediate representations, whose group actions are not specified. In a finite discrete setting, we show that equivariance requires representations of original and transformed inputs to determine one another exactly, yielding a test based on normalized mutual information (NMI). For high-dimensional activations, we use centered kernel alignment (CKA) as a scalable test, with an exact guarantee for orthogonal feature transformations. Across architectures and tasks, augmentation usually increases transformation consistency, showing that learned equivariance can be probed directly from representations without recovering the hidden transformation rules that relate them.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Equivariance, Geometric Machine Learning, Information Theory

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YrSkOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a method for measuring learned equivariance when group actions are not previously known. For low dimension they use Normalized Mutual Information, and for higher dimension they use Centered Kernel Alignment.

Critical strengths

Strength 1: The problem they adress is important and has multiple direct applications so their work is useful for many other researchers and scientists.

Strength 2: The mathematical foundation they provided is very clear and was easy to follow.

Strength 3: The experiments they provided tested their method on different models and tasks.

Critical weaknesses

Weakness 1: I think that the mathematical gain from this paper is limited as it doesn't introduce any new technique except from using known properties. So for me it looks more like an application of existing KPIs rather than a fundamentally novel mathematical method.

Weakness 2: MIME and CKALE are testing the invertibility property of equivariance but (maybe I missed this) it doesn't appear to measure the homomorphism property of equivariance (ρ(fg)=ρ(f)ρ(g)\rho(fg)=\rho(f)\rho(g) and ρ(e)=I\rho(e)=I for all f,g∈Gf,g \in G). Even if the authors admit in figure 2 that their condition is necessary not sufficient I think it is interesting to look at how we can interpret their results if we don't know the group composition law.

Weakness 3: From my understanding CKA measure the similarity of geometry between 2 sets of representation, but I am not convinced that comparing 2 geometries is the same as comparing equivariance. For example, we could have a constant function, and see as equivariant what is in reality invariant (potentially?).

Other remark:

I am not sure that the figure 2 line 179, page 4 is aligned with ICLR format as traditionally double column is not the norm. But as it is a figure and not text I leave this to the AC to decide/clarify.

Questions for authors

I gave a 2 but I am open to increase my score if the authors can provide a test for the homomorphism property of equivariance (or be convincing that such a test is not needed).

Also I am open for the author to confirm (or refute) if my understanding of weaknesses 1 & 3 is wrong.

I have some questions for the authors out of curiosity:

Q1: Figure 1 shows that CKA changes based on the rotation angle, can we somehow use these patterns to identify the learned symmetry group?

Q2: Do you think measuring the CKA across layers could be a way to see where the learned equivariance appears in a NN?

2 comments

W1: You are correct that we don’t introduce a new mathematical method of measuring similarity but rather apply an existing one in a novel way. However, our paper still contains novelty in the extensive experimentation to identify that mutual information works, that CKA can function as a good replacement in real world applications, and that in real world scenarios it can detect learned equivariance.

W2: We do not test for the homomorphism property of equivariance. However, our method could be used to test for that property. Consider the C4C_4 group. If we test all 90 degree rotation pairs [(x,rx),(rx,r2x),(r2x,r3x),(r3x,x)][(x, rx), (rx, r^2x), (r^2x, r^3x), (r^3x, x)] using CKALE, we will find that if CKA is one, then the group homomorphism property holds as there exists a consistent transform across any number of group actions that transforms one into the rest. We inadvertently did this in the NMI experiment when testing all 2162^16 possible input images (so all images and their rotated forms exist in the dataset) and showed that we still recovered the C4C_4 equivariance of the laplacian filter and the C2C_2 equivariance of the Sobel-X filter. Thank you for this question, this greatly improves our work.

W3: A constant function would produce a reading of equivariant rather than invariant, but it is important to remember that invariance is a type of equivariance, so that reading is still correct. Testing for invariance is a rather simple check of seeing if the outputs are the same no matter the transformation applied to the input, so it is not something difficult to test like equivariance is.

Q1: Figure 1 is showcasing a case of NMI, but the same principle applies to CKA. By splitting up the measurements by rotation angle, we can identify which angles are exactly equivariant and which are not. In the case of figure 1, Sobel-X is C2C_2 equivariant which is indicated by the score of 1 at the 180 degrees. If a network learning equivariance was measured at every 45 degree rotation and only scored 1 at every 90 degree rotation, we would know it was C4C_4, but not C8C_8 equivariant.

Q2: Figure 1 is showcasing a case of NMI, but the same principle applies to CKA. By splitting up the measurements by rotation angle, we can identify which angles are exactly equivariant and which are not. In the case of figure 1, Sobel-X is C2C_2 equivariant which is indicated by the score of 1 at the 180 degrees. If a network learning equivariance was measured at every 45 degree rotation and only scored 1 at every 90 degree rotation, we would know it was C4C_4, but not C8C_8 equivariant.

Thank you for your response and clarifications. While I appreciate the authors' efforts to address my concerns, I remain unconvinced by some of the arguments.

The pdf has also not been updated.

I will therefore maintain my original score.

Reviewer_BNwFOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the challenge of measuring the degree of (group) equivariance in deep neural networks. This is especially difficult for intermediate layers, where a specific group representation may not be known. The authors show the shortcomings of measuring this via normalized mutual information (NMI), and propose to use centered kernel alignment (CKA) as a training-free scalable alternative, showing its advantage over other methods in synthetic setups.

Critical strengths
  1. The paper is well motivated and fairly easy to follow (although some parts need further clarification)
  2. The method is clear and understandable
  3. I appreciated the systematic analysis of failure cases of existing methodologies via synthetic setups
Critical weaknesses
  1. Overclaims. The first contribution point, that "learned equivariance can be tested without specifying how hidden features should transform" seems like an overstatement. In particular, for this contribution point, there is substantial overlap with Möller and Polani (Emergence of common concepts, symmetries and conformity in agent groups—an information-theoretic model), with obvious links to mutual information. I understand that the two manuscripts broadly deal with different questions, but from my understanding the first bullet point shares a lot in common with, at least, Möller and Polani.

  2. Clarity and lack of important details Although the paper is generally fairly easy to follow, a few things need, in my opinion, further specifications. For example,

    • (2.1) The authors could better explain how they construct the 'null' pairs. These are nominated for the first time in the related work (!) and then immediately used in the Experiments section. The authors should comment on the significance of the null pairs and exactly what kind of control they allow.
    • (2.2) Similarly, the authors never define or explain the significance of each of LieGG's subvalues in Tables 6 and 7, making those results poorly contextualized. The same applies for ∣∣∇∣∣||\nabla|| and mm in the same tables.
    • (2.3) The same goes for the SEIS results: what are spatial and channel re-encodings? Why are they relevant? And why is CKA missing from table 8?
    • (2.4) All details needed to reproduce Figure 3's results are missing: how are the datasets generated? What hyperparameters does each model use?
  3. CKA failure modes. The authors acknowledge and expose some of CKA's failure modes: it can output a high degree of equivariance when the representations are not equivariant (Section B.2) or a low degree of equivariance when in fact the representations are equivariant (Section 5.2). Therefore, the authors should argue why CKA is a sound measure of equivariance despite these counterexamples. Some of these problems are resolved if the group action is orthogonal, but this is a condition that should be verified. In the paper, the authors say "We assume that this [that ρ\rho can be taken to be orthogonal] holds for networks trained with augmentations" (lines 301,302), but I don't think that this is a legitimate assumption without verification.

  4. Limited experiments. Almost all experiments are conducted on (i) synthetic datasets and (ii) with geometric actions (rotations). However, to demonstrate CKA's wide applicability, the authors should demonstrate CKA on more realistic datasets and a more diverse set of groups (including non-geometric actions potentially, such as font swapping).

Other points:

Consistency. Why do the authors switch from CKA to 1-CKA in the tables? And why do they have up and down arrows only in table 7?

I found the naming of "MIME" and "CKALE" slightly misleading. The authors are applying well-known quantities (NMI and CKA) for the purpose of measuring equivariance in deep neural networks. Therefore, giving new specific names makes it seem methodologically novel, while it isn't. This does not necessarily detract from the paper, as a thorough analysis of the applicability of such methods is still valuable, but it gives a slightly misleading impression of the paper's contributions.

The title of section 5.4 is also quite misleading, since CKALE is not "assumption free", as immediately stated by the authors themselves in the first line of the section.

Figure 2 is imprecise, since [CKA<1 => non-equivariance] holds only for orthogonal representations.

Questions for authors

Can the authors address the limitations and questions above?

Also, I think that the following setup would be particularly interesting. Maybe the authors could train a neural network with an additional loss term that promotes equivariance to a specified group action/representation$^1$. Then, the authors could plot the equivariance error against their CKA metric to see whether these two measurements agree. This would be particularly interesting and convincing if the authors explored different choices for these group actions: orthogonal, non-orthogonal, non-linear, etc, and the group could act geometrically or non-geometrically.

Finally, can the authors comment on whether CKA is designed to measure the degree of equivariance for groups specifically, and not equivariance to other algebraic objects (e.g. monoids)? If so, why present it as a measure of group equivariance?

1^1: Example papers that use this equivariance penalty score: - Equivariant Neural Rendering - Learning Group Actions on Latent Representations (NeurIPS 2024) - Algebraic Priors for Approximately Equivariant Networks

4 comments

W1: Having read this work, it is definitely a related work on using mutual information to measure similarity in representations that share some symmetry of each other. We would be happy to add this to our related work section. However, we believe the problem we tackle, measuring learned equivariance, is substantially different. Rather than showing that different agents produce the same representation under different interpretations of the same task, we are showing that the same model produces equivariant representations of an input given different rotations. The key difference here being that we are focusing on one model’s internal representations compared to them who are looking at several models with similar representations.

W2.1: Null pairs allow for us to define a noise floor. Essentially between two pieces of data, some of it will relate to some degree. For example, two datasets with two classes randomly assigned might see an agreement noise of 25% just because of how the random assortment comes together. However, if they are more aligned, we should see a score higher than 25% alignment. These null pairs create this noise floor and help us see whether the alignment is real or just an artifact of random chance. We will provide further clarification when updating the paper

W2.2: LieGG is a measure of equivariance error that uses a polarization matrix and SVD. sym_var is the smallest normalized singular value. null_dim is the number of normalized singular values below NULL_TOL, i.e. the dimension of the symmetry algebra LieGG found. SO(2) is one-dimensional, so the ideal answer is 1. align checks whether the recovered null space actually contains the true rotation generator A_true. The code projects the normalized A_true onto the null-space basis and takes the norm: (null @ A).norm(). This is the |cos| between A_true and the recovered subspace. It projects onto the whole subspace, not just the last singular vector, because if null_dim > 1 any single basis vector is arbitrary. A_true/rand basically says that the closer you are to zero the closer you are to picking the real A_true, while the closer you are to one, you aren’t doing any better than a random generator. Again, we can update this in the paper to make it more clear.

W2.3: Table 8 is to point out a failure of SEIS to capture channel re-encodings. Spatial re-encodings means that the points in space such as where pixels are located on an image are re-arranged. Channel re-encodings is what happens to the RGB values of those pixels when they are permuted. CKALE is able to see when both spatial and channel re-encodings occur, but SEIS is not. We will add further clarification in the edited form of the paper.

W2.4: We expand on how the datasets are generated in section 5.5, but we will provide more details and specifics about hyperparameters about the models and datasets made.

W3: Section B.2 is an experiment where equivariance exists but is not perfect as information is lost during aliasing. The equivariance score is a bit complicated to measure. We see a small drop as expected by the information lost in aliasing, but the high score indicates the model is still very equivariant as expected of a C16C_16 model. Section 5.2 shows the failures of CKA which is any form of equivariance that has a non-orthogonal mapping. However, while this is a limitation, most equivariant networks are built to use orthogonal transformations as their means of preserving equivariance. So, we assumed from that logic

W4: It would be interesting to test on further interesting orthogonal actions such as mirroring, but we don’t believe CKA will hold for something such as font-swapping. It may hold in the sense that the latent space between two different fonts of the number one may be very similar, but inherently the images will be far apart. However, we are open to trying this experiment and will provide it when given a chance to update our work.

OP1: We will update the draft to consistently use arrows across all tables. The switch between CKA and 1-CKA is to make it so that CKA follows the pattern where 0 is no equivariance error and 1 is the most possible equivariant error, but we see how this can be confusing.

OP2: NMI and CKA are metrics, whereas CKALE is a method of applying CKA to latent spaces of neural networks after transformations have occurred on the input and MIME is similar but utilizing NMI and is only applied in the discrete case. We did not intend to claim undue credit by renaming common methods but instead to draw a useful distinction. We will clarify that in the revision. If you have any suggestion, we are happy to adopt it.

OP3: We will update the title to clarify that it is not assumption free, but rather has weaker assumptions on identifying equivariance than previous methods.

OP4: We will update the figure to highlight this conditionality for better clarity.

Q1: This would be a good experiment to further test CKALE, however non-orthogonal, non-linear, and non-geometric transform may not work with CKA. However, a non-geometric transform (like say a font change) may be an interesting idea where latent spaces may be similar across different fonts in classification.

Q2: We focus on groups as that is the most relevant to the field at this time. However, any action that is equivariant with invertibility can be used by our method. Because we don’t necessitate the group homomorphism property, it is actually possible for any system that requires an inverse to be equivariant to use CKALE to measure equivariance.

I thank the authors for their reply.

However, some concerns appear to remain unaddressed. My biggest concern is the following:

The authors' response to my Q1 is: *This would be a good experiment to further test CKALE, however non-orthogonal, non-linear, and non-geometric transform may not work with CKA. * This makes me more concerned about the paper as a whole. The authors are proposing a measure for group equivariance, but it turns out that it may only work for group equivariance with respect to orthogonal, linear and geometric group actions. This is a much narrower claim, and contradicts the current claims of the paper that CKALE is a scalable method to measure learned equivariance.

Secondly, I believe that the authors should discuss the work of Mölloer and Polani and what exactly makes their methodology different in greater depth.

Finally, the authors promise revisions (e.g. to the related work) and to the experiments (e.g. font-swapping), but the available manuscript does not contain any of these.

Q1: We did not mean to mislead, it is correct to say the CKALE is limited to orthogonal and geometric group actions because it acts only on the numeric representation of the data. Non-geometric actions such as font-swapping would change the numeric representation of numbers, but not the information provided in classification (the number 1 is the same no matter its font). In the sense of information, both images portray the same information (the number 1) even if they are different fonts, but would emit different vectors to CKA for its process that do score less than 1. However, running a CNN over font-swapped digits shows that it can still learn equivariance and that CKA approaches one the further in the network you get running over two images of the same number with different fonts. So, while it does not work directly for font-swapping, it can still be used in neural networks, but the results underestimate the similarity. Other methods like the Lie derivative from Gruver et al. or LieGG have the same limitations.

Q2: Our work proposes a method for measuring equivariance using similarity measures such as NMI and CKA. It begins from a known symmetry group that a system is supposed to learn or be adapted to, then tests whether trained or constructed representations carry that symmetry and whether established equivariance measures such as the Lie derivative detect it. Möller and Polani instead use information theory as an objective: an information-bottleneck formulation extracts a common conceptualization shared across a group of agents, and symmetries of the environment emerge from that optimization rather than being specified in advance.

Reviewer_krg7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In many applications, desired invariances/equivariances are induced during training using data augmentation. Yet, measuring whether a network actually satisfies those symmetries remains challenging, particularly for intermediate representations where the group action is unknown. This manuscript addresses this issue by introducing two methods for measuring equivariance. The first, MIME, is based on normalized mutual information and the second, CKALE, is based on centered kernel alignment (CKA). Experiments on a broad variety of cases suggest that CKALE performs well compared to existing baselines in measuring equivariance.

Critical strengths

The problem tackled by the paper is important and the methods it proposes are theoretically justified. The experimental section is comprehensive, covering a wide range architectures, datasets, and symmetry groups.

Critical weaknesses
  • The proposed metric cannot identify equivariance given that they are necessary, but not sufficient (Propositions 1 and 2). Hence, they are not measures of equivariance and cannot certify that an architecture is equivariant. This is in contrast to deterministic metrics based on, e.g., operator norms of Lie derivatives [Gruver et al., 2023] or [Otto et al., A unified framework to enforce, discover, and promote symmetry in machine learning, 2024]. The manuscript provides no relation between the value of the metric and, e.g., perturbations of the output.
  • This is particularly problematic for the manuscript's experimental conclusion that CKALE can detect some known equivariances. Indeed, given that CKA cannot certify equivariance (as per Prop. 2), an increase in CKA cannot be used to conclude that data augmentation has led to an improvement in learnt equivariance.
  • Even these guarantees only hold for orthogonal group action representations, which precludes, e.g., sheering. Though the manuscript acknowledges this limitation, it does not discuss their impact in practice or the extent to which they apply its experiments. It is worth noting that previous metrics do not suffer from this issue.
  • The manuscript is not clear about which are the advantages of these approaches over previous ones. In particular, there does not seem to be computational gains (both NMI and CKA are not straightforward computations for large datasets). Given the previously mentioned disadvantages, it's unclear when these metrics should be used. This is particularly crucial for groups with infinite orbits, where NMI and CKA must be computed for every g∈Gg \in G, a challenge overcome in [Gruver et al., 2023] and [Otto et al., A unified framework to enforce, discover, and promote symmetry in machine learning, 2024] by showing that it is enough to check the group's generators.
  • The proposed metric is not deterministic and, as such, can only be computed over distributions of points. It is not possible to ascertain whether a model is equivariant to the group actions at a specific point (a point mass has vanishing entropy).
Questions for authors
  1. What are the advantages of these metrics over previous ones, such as [Gruver et al., 2023] and [Otto et al., A unified framework to enforce, discover, and promote symmetry in machine learning, 2024], in terms of computation or guarantees? How can they be computed for infinite (continuous) groups (e.g., rotations)?
  2. To what extent should we interpret differences in CKA between models as different levels of equivariance? Which observations or results from the manuscript warrant those comparison? To what extent does a higher CKA value actually indicate a lower equivariance error?
  3. Equivariant is a deterministic property (or, in a probabilistic context, an almost sure property). How can these distributional metrics be used to measure equivariance at any given point?
2 comments

W1: You are correct, our method only detects non-equivariance and cannot prove equivariance, but Gruver et al. and Otto et al. can only verify equivariance if you correctly guess the group action. Our methods are thus complimentary. First use our method to indicate if there isn’t equivariance. If there may be equivariance, then you can use Gruver et al to search over the space of group actions to verify equivariance. However, our method is deterministic as it will produce the same results for the same data and network.

W2: This is also true, we can only see that the possible equivariance error has decreased, not necessarily guarantee that the model is equivariant. While a learned equivariant model will never be actually equivariant, our method still poses as a good way to seeing if it is getting closer to equivariance.

W3: The orthogonal case is the most common case in equivariant models. The advantage and key difference of our paper is that while it does require the group action to be orthogonal, it does not require knowledge of the group action on the latent space to be an effective measure. The tradeoff of the limitation from previous work is that we require that the transformation on the latent space be a class of possible actions rather than a single action we specify.

W4:The main advantage about our approach is that we don’t require knowing any information about the latent space transformation that occurs whereas previous methods require strict knowledge about the latent space transformation. We are tackling this problem from the other side where rather than trying to prove equivariance in a specific way, we are trying to rule out equivariance in a more general form.

W5: Both CKALE and MIME are deterministic measures. While miniBatch CKALE has a variance of about 0.0017 (computed from 8196 data points and batch size of 64 which is what we used in training with 20 shuffles to compute CKA. We will add this experiment and details to our paper) it is not inherently non-deterministic. Having perfect knowledge of the probability distribution allows us to compute MIME and having a large enough system allows us to compute CKALE perfectly with no problems.

Q1: The biggest advantage is that you are able to rule out equivariance without necessarily testing every possible group action like you would for the Lie derivative method or LieGG. If you knew the group action in advance, you would be able to test for equivariance, but since we don’t our method helps rule it out. For continuous groups we do subsampling which is practical, but not perfect. As shown in Figure 1, the more you subsample, the closer you can measure actual equivariance, and the closer the sub group (C16C_16) becomes to being SO(2)SO(2) equivariant.

Q2: CKA cannot be compared between models, but only between the same model on different datasets or against a shuffled null as described in the paper. Having the comparison against either tells us if a model is more or less equivariant against itself in different conditions, but a higher CKA only indicates lower equivariance error against its own model. There is a chance it could be used for the outputs since they are the same dimensional space, but it is more designed for analyzing the latent space.

Q3: Detecting equivariance without knowing the group action requires that you look at multiple points. You cannot determine if a map is invertible by only looking at one pair of points.

I appreciate the authors' response. However, the limitations of the work are not sufficiently addressed in their response and many of my original concerns remain. As such, I cannot recommend the acceptance of this manuscript in its current form.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.