- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a novel method to learn relaxed equivariant networks. In particular, this is done throuhg a single parameter which interpolates between an equivariant backbone and unconstrained residual branches. The interpolation parameter itself is optimized with the model under a soft-equivariant constraint via Lagrangian-inspired method.
Critical strengths
The paper presents a neat solution, boiling down the problem of learning relaxed equivariance to single controllable and interpretable scalar parameter . The use of the primal-dual update rule to train under bounded target equivariance-breaking threshold is also interesting and, as far as I know, novel.
Finally, the empirical results show promising gains from using this solution over existing methods.
Critical weaknesses
Unfortunately, I found the presentation quite unclear in different points.
In particular, I have some doubts about the main method, as presented in Eq. 4. As , you get a very trivial equivariant network with just identity operations, am I understanding this correctly? is it desirable? Along this line, it seems this Eq. 4 heavily under-parameterizes the equivariant backbone, no? A complete equivariant network would have at least 3 parameters (a different learnable scalar for each ) there but, most importantly, it also supports multiple copies of each irrep, with each input-output pair having a separate learnable parameter, as well as independent parameters per layer. Is this supported in your construction? This is unclear in the main paper. What I currently understand is that the model is a unconstrained model plus an identity-constrained equivariant residual path (with attenuated high-frequencies), is this correct? If this is true, this seems a very strong limitation in the expressiveness of the model and I wonder if it is justified and desirable.
Sec 3.4 and 3.5 are also not fully clear to me. I think generally the construction could be presented in a different way, as it currently looks like a post-hoc justification of Eq. 4 without clear mathematical rigor. Different components in Sec 3.4 and 3.5 are relaxed to achieve Eq. 4, which still looks fairly limited (see comment above). This makes me wonder if the theoretical construction adopted is the right tool to approach the problem considered.
Questions for authors
Eq. 4 seems a bit ambiguous: you seem to overload the notation, using to identity the input/output features rather than the representations. Can you clarify?
Eq. 5: what is ? What is the difference between and ? what is ? I understand some of these terms might get clearer later, but using them here without any context makes this paragraph much harder to follow.
Sec 3.3, Alg. 2: it says is updated via gradient ascent on but doesn't contain . How does this work? Do you mean on instead?
Eq. 8 what is M? It has been mentioned a few times so far, but never really defined and explained. Soon later, what is the Euclidean hub? Is equivariant? I'm not sure Figure 1 helps understanding Eq. 5 much.
It seems that Sec 3.4 mixes two separate arguments/implications: one is the justification of the decomposition in Eq.4 and 5 to introduce equivariance breaking components, and the other is the justification of the use of only . These two arguments are a bit entangled and I wonder if they can instead be separated. Is a strict limitation of the proposed method?
Is there any justification for Assumption 2? As far as I know, these spectral biases in neural networks arise from fairly different contexts (frequencies in the spectral basis for function on the full Eucidean input space ).
How is Assumption 2 justifying Proposition 2? If indeed there is a spectral bias towards lower frequencies, is it necessary to manually enforce the attenuation of the higher frequency components?
Fig.3: are results aggregated over multiple runs or are the shown results specific of a single run? The difference between the curves seems often within the transparent error bars, but it's unclear what those bars indicate.
Fig 4.: why is the delta decreasing over training? Isn't this a rotated dataset? or is this only using planar rotations along the vertical axis?
RQ5 / Fig 6: is this a surprising result? If I understand correctly, the single layer considered is just a linear layer right? Isn't this simply the statement of theorem 1? Or is Fig 6 communicating anything more?
5 comments
Response to Reviewer TTmx
Q1: Notation in Eq. 4
We use and only for representations: the group actions on a layer's input and output spaces. The features are and , and is the feature block that transforms under the Wigner-D matrix . The submitted text blurred the two by writing that "corresponds to" . We removed that sentence in the revised version. The revised Sec. 3.1 defines features and representations separately before Eq. 4, and Eq. 4 is now written with the message-passing output (see W1).
Q2: Terms in Eq. 5
We agree that Eq. 5 used these symbols before they were defined. In the revised version, it sits at the end of Sec. 3.4, after all of its symbols are defined, as Reviewer yXhG also requested. The symbols are:
- : a matrix. It is the operator of Eq. 8.
- : a matrix. It is the degree- part of at (Eq. 12).
- : a feature vector, not a matrix. It is the degree- block of the features of the layer.
- : a matrix. It is the sum of all terms of that contain (Eq. 9). It is the non-equivariant part.
Difference between and . belongs to the operator , and belongs to the network. Eq. 5 is a relation between matrices. The layer of Eq. 4 has the same form with features: the feature block takes the place of the matrix , and the learned branch takes the place of . (The revised version writes Eq. 5 between matrices only, without on the left-hand side, and writes the feature block as ).
Q3: Dual update in Alg. 2
Yes. is the multiplier of the constraint in the Lagrangian , and the step is a dual ascent on , not on . Alg. 2 abbreviated it as "ascent on " in L8, because the step direction is the quantity that defines . With the projection onto , the update is
taken with Adam in practice. rises while the constraint is violated, falls once it is met, and stays non-negative. In each training step, and first descend with the current , and is then updated. Sec. 3.3 and Alg. 2 now write the update explicitly in the revised version.
response to reviewer TTmx
Q4: What is ? The Euclidean hub? Is equivariant?
What is ? is the torus–hub space drawn in Fig. 1: three tori joined at a central hub. Each torus is one toroidal direction and carries one rotation generator, , or . Formally, (App. A.5). The submission used before introducing it. The revised version introduces it with the operator of Eq. 8.
What is the Euclidean hub? It is the central part of Fig. 1, where the three tori meet. In Eq. 8, it is the term .
Is equivariant? is the commutator of the generators and sits in the hub of Fig. 1. It is a matrix of degree , like the generators. It belongs to the equivariant part (Eq. 9). Only is unconstrained.
Does Fig. 1 help to understand Eq. 5? We agree that the link was not explicit. Fig. 1 does not show Eq. 5 itself. It shows the operator of Eq. 8, from which Eq. 5 is derived. The three tori are . The hub is , and its shade shows . The sum of the three tori and the hub is . The revised Eq. 8 is annotated with these parts of Fig. 1.
Eq. 5 follows from in three steps. We exponentiate . Its Taylor expansion is a sum of matrix products of , and . First, we split this sum (Eq. 9). The products that contain carry the factor , and their sum is . The products without contain only and , and their sum is . Second, we decompose by degree (Eqs. 10–11). Each of its products, e.g. or , is a matrix. A matrix has a part of degree (its trace), (its antisymmetric part) and (its symmetric traceless part). We collect the parts of degree from all products into one block . So is the sum of three blocks, . Third, Proposition 2 (Eq. 12) replaces each block by , where is the block at . The three blocks and together are Eq. 5.
Q5: Two arguments in Sec. 3.4; is a limitation?
Two arguments. We have separated the two arguments in the revised version. To summarize Sec. 3.4: we first define the theoretical operator in Eq. 8.
First argument (Eq. 9). We exponentiate the operator and split its Taylor expansion into two parts, and . contains only and , which are matrices of degree . We call it the equivariant part. contains every term with the unconstrained dense matrix . It is the non-equivariant part and carries the weight . This step does not involve .
Second argument (Eqs. 10–12). The terms of are matrix products of and . We decompose them by degree (Clebsch–Gordan) and group them into the blocks . We then apply Proposition 2 to each block, which gives the factor .
Combining the two gives Eq. 5. We made these steps explicit in the revised manuscript.
Is a strict limitation? No. It comes from the example used in Sec. 3.4, not from the method. In Sec. 3.4, and of Eq. 8 are matrices. A matrix has only three parts: its trace (), its antisymmetric part (), and its symmetric traceless part (). So only these degrees appear in Eq. 10. With larger matrices, higher degrees appear and follow the same law (shown in App. A.3). The network has no such restriction. It scales every irrep block of degree by , for any the backbone uses. Our ModelNet40 network uses features up to .
Response to Reviewer TTmx
We thank the reviewer for the detailed reading, and for recognizing the single interpretable δ and the primal-dual training of δ. We answer each question and weakness below.
W1: Is the equivariant branch the identity? Is it under-parameterized?
No. The equivariant branch is a complete equivariant layer, not an identity path. The confusion came from our notation. In the submitted Eq. 4, ψin(ℓ) already denoted the output of the layer's equivariant message passing (Algorithm 1, L4–L5). The text did not say so, and the sentence "Leq(x)≡⨁ℓδℓψin(ℓ)" read as a bare scaling of the input.
The revised Sec. 3.1 makes this explicit. Each layer first runs its full equivariant message passing: tensor products of the input features with spherical harmonics, weighted by learned radial functions and aggregated over neighbors. We write u=⨁ℓu(ℓ) for its output. The revised Eq. 4 is
ψout=⨁ℓδℓu(ℓ)+(1−δ)W(u).
So δℓ scales the output blocks of a learned equivariant layer. It is not the only equivariant parameter. On the reviewer's three points:
At δ=1 the layer returns ψout=u. The model is then exactly the strict equivariant backbone with all of its parameters: SEGNN on N-body and QM9, and our tensor-product message-passing network on ModelNet40 and Oxford RobotCar. So the model is not an unconstrained model plus an identity-constrained residual path. It is a full equivariant network whose output blocks are graded by δℓ, plus an unconstrained branch weighted by (1−δ). The only global quantity is δ.
W2: Sec. 3.4–3.5 read as a post-hoc justification
Is it post hoc? No. Eq. 4 is obtained from the operator A(δ) shown in Fig. 1 and Eq. 8. Sec. 3.4 defines A(δ) and derives Eq. 5 from it. Sec. 3.5 then obtains the layer of Eq. 4 from Eq. 5. We stated Eq. 4 first only for readability.
The construction, step by step.
Which components are relaxed? Two, and the revised version states both. In step 3, δℓψ~ℓ is a surrogate for the block. It is exact at δ=1, and we present it as a design choice. In step 4, the network uses features and tensor products in place of the matrices of the operator. All other steps are exact. Neither relaxation affects the equivariance of the layer: the scaling by δℓ keeps equivariance, and all equivariance error comes from (1−δ)W (Theorem 1).
Is it the right tool? The construction gives the form of the layer: one δ for both equivariant and non-equivariant branches, and an attenuation δℓ that grows with the degree. Table R1 (general response; Table 4 in the revised version) tests this attenuation on ModelNet40.