- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces a method for measuring learned equivariance when group actions are not previously known. For low dimension they use Normalized Mutual Information, and for higher dimension they use Centered Kernel Alignment.
Critical strengths
Strength 1: The problem they adress is important and has multiple direct applications so their work is useful for many other researchers and scientists.
Strength 2: The mathematical foundation they provided is very clear and was easy to follow.
Strength 3: The experiments they provided tested their method on different models and tasks.
Critical weaknesses
Weakness 1: I think that the mathematical gain from this paper is limited as it doesn't introduce any new technique except from using known properties. So for me it looks more like an application of existing KPIs rather than a fundamentally novel mathematical method.
Weakness 2: MIME and CKALE are testing the invertibility property of equivariance but (maybe I missed this) it doesn't appear to measure the homomorphism property of equivariance ( and for all ). Even if the authors admit in figure 2 that their condition is necessary not sufficient I think it is interesting to look at how we can interpret their results if we don't know the group composition law.
Weakness 3: From my understanding CKA measure the similarity of geometry between 2 sets of representation, but I am not convinced that comparing 2 geometries is the same as comparing equivariance. For example, we could have a constant function, and see as equivariant what is in reality invariant (potentially?).
Other remark:
I am not sure that the figure 2 line 179, page 4 is aligned with ICLR format as traditionally double column is not the norm. But as it is a figure and not text I leave this to the AC to decide/clarify.
Questions for authors
I gave a 2 but I am open to increase my score if the authors can provide a test for the homomorphism property of equivariance (or be convincing that such a test is not needed).
Also I am open for the author to confirm (or refute) if my understanding of weaknesses 1 & 3 is wrong.
I have some questions for the authors out of curiosity:
Q1: Figure 1 shows that CKA changes based on the rotation angle, can we somehow use these patterns to identify the learned symmetry group?
Q2: Do you think measuring the CKA across layers could be a way to see where the learned equivariance appears in a NN?
2 comments
Thank you for your response and clarifications. While I appreciate the authors' efforts to address my concerns, I remain unconvinced by some of the arguments.
The pdf has also not been updated.
I will therefore maintain my original score.
W1: You are correct that we don’t introduce a new mathematical method of measuring similarity but rather apply an existing one in a novel way. However, our paper still contains novelty in the extensive experimentation to identify that mutual information works, that CKA can function as a good replacement in real world applications, and that in real world scenarios it can detect learned equivariance.
W2: We do not test for the homomorphism property of equivariance. However, our method could be used to test for that property. Consider the C4 group. If we test all 90 degree rotation pairs [(x,rx),(rx,r2x),(r2x,r3x),(r3x,x)] using CKALE, we will find that if CKA is one, then the group homomorphism property holds as there exists a consistent transform across any number of group actions that transforms one into the rest. We inadvertently did this in the NMI experiment when testing all 216 possible input images (so all images and their rotated forms exist in the dataset) and showed that we still recovered the C4 equivariance of the laplacian filter and the C2 equivariance of the Sobel-X filter. Thank you for this question, this greatly improves our work.
W3: A constant function would produce a reading of equivariant rather than invariant, but it is important to remember that invariance is a type of equivariance, so that reading is still correct. Testing for invariance is a rather simple check of seeing if the outputs are the same no matter the transformation applied to the input, so it is not something difficult to test like equivariance is.
Q1: Figure 1 is showcasing a case of NMI, but the same principle applies to CKA. By splitting up the measurements by rotation angle, we can identify which angles are exactly equivariant and which are not. In the case of figure 1, Sobel-X is C2 equivariant which is indicated by the score of 1 at the 180 degrees. If a network learning equivariance was measured at every 45 degree rotation and only scored 1 at every 90 degree rotation, we would know it was C4, but not C8 equivariant.
Q2: Figure 1 is showcasing a case of NMI, but the same principle applies to CKA. By splitting up the measurements by rotation angle, we can identify which angles are exactly equivariant and which are not. In the case of figure 1, Sobel-X is C2 equivariant which is indicated by the score of 1 at the 180 degrees. If a network learning equivariance was measured at every 45 degree rotation and only scored 1 at every 90 degree rotation, we would know it was C4, but not C8 equivariant.