acceptodds
Under review as a conference paper at ICLR 2027

Tunable Equivariance via Learnable Order-Dependent Scaling

Abstract

Equivariance is a powerful inductive bias in geometric deep learning, but real-world data often only approximately respects known symmetries. Strictly equivariant models can therefore be overly restrictive, while unconstrained models fail to exploit useful geometric structure. Existing relaxed-equivariance approaches mainly interpolate between equivariant and non-equivariant components, without modulating the internal structure of the equivariant features. We introduce a **tunable-equivariance** framework in which a single learnable parameter controls both the unconstrained branch and the relative strength of irrep components, weighting order by . This yields a structured, continuous modulation of equivariance, motivated by a torus–hub geometric interpretation within an **associative operator algebra**. We learn jointly with the task objective under an empirical equivariance constraint. We instantiate the framework on -equivariant architectures for N-body dynamics, molecular property prediction, 3D shape classification, and LiDAR pose estimation. Across these benchmarks, the learned degree of equivariance is task-dependent. It generally improves performance over strict-equivariant baselines and outperforms learned relaxed-equivariance baselines where directly compared. Higher-order irrep components strengthen with , and the **single-layer equivariance error** is consistent with the predicted bound. Code will be released upon acceptance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Equivariance, tunable equivariance, geometric deep learning, shape classification, pose estimation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TTmxOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a novel method to learn relaxed equivariant networks. In particular, this is done throuhg a single parameter δ∈[0,1]\delta \in [0, 1] which interpolates between an equivariant backbone and unconstrained residual branches. The interpolation parameter δ\delta itself is optimized with the model under a soft-equivariant constraint via Lagrangian-inspired method.

Critical strengths

The paper presents a neat solution, boiling down the problem of learning relaxed equivariance to single controllable and interpretable scalar parameter δ\delta. The use of the primal-dual update rule to train δ\delta under bounded target equivariance-breaking threshold is also interesting and, as far as I know, novel.

Finally, the empirical results show promising gains from using this solution over existing methods.

Critical weaknesses

Unfortunately, I found the presentation quite unclear in different points.

In particular, I have some doubts about the main method, as presented in Eq. 4. As δ→1\delta \rightarrow 1, you get a very trivial equivariant network with just identity operations, am I understanding this correctly? is it desirable? Along this line, it seems this Eq. 4 heavily under-parameterizes the equivariant backbone, no? A complete equivariant network would have at least 3 parameters (a different learnable scalar for each ll) there but, most importantly, it also supports multiple copies of each irrep, with each input-output pair having a separate learnable parameter, as well as independent parameters per layer. Is this supported in your construction? This is unclear in the main paper. What I currently understand is that the model is a unconstrained model plus an identity-constrained equivariant residual path (with attenuated high-frequencies), is this correct? If this is true, this seems a very strong limitation in the expressiveness of the model and I wonder if it is justified and desirable.

Sec 3.4 and 3.5 are also not fully clear to me. I think generally the construction could be presented in a different way, as it currently looks like a post-hoc justification of Eq. 4 without clear mathematical rigor. Different components in Sec 3.4 and 3.5 are relaxed to achieve Eq. 4, which still looks fairly limited (see comment above). This makes me wonder if the theoretical construction adopted is the right tool to approach the problem considered.

Questions for authors

Eq. 4 seems a bit ambiguous: you seem to overload the notation, using ψin/out\psi_{in/out} to identity the input/output features rather than the representations. Can you clarify?

Eq. 5: what is AA? What is the difference between ψl~\tilde{\psi_l} and ψl\psi_l? what is ψflat\psi_{flat}? I understand some of these terms might get clearer later, but using them here without any context makes this paragraph much harder to follow.

Sec 3.3, Alg. 2: it says λ\lambda is updated via gradient ascent on ss but ss doesn't contain λ\lambda. How does this work? Do you mean on LL instead?

Eq. 8 what is M? It has been mentioned a few times so far, but never really defined and explained. Soon later, what is the Euclidean hub? Is CC equivariant? I'm not sure Figure 1 helps understanding Eq. 5 much.

It seems that Sec 3.4 mixes two separate arguments/implications: one is the justification of the decomposition in Eq.4 and 5 to introduce equivariance breaking components, and the other is the justification of the use of only l∈{0,1,2}l \in \{0, 1, 2\}. These two arguments are a bit entangled and I wonder if they can instead be separated. Is l∈{0,1,2}l \in \{0, 1, 2\} a strict limitation of the proposed method?

Is there any justification for Assumption 2? As far as I know, these spectral biases in neural networks arise from fairly different contexts (frequencies in the spectral basis for function on the full Eucidean input space Rn\mathbb{R}^n).

How is Assumption 2 justifying Proposition 2? If indeed there is a spectral bias towards lower frequencies, is it necessary to manually enforce the attenuation of the higher frequency components?

Fig.3: are results aggregated over multiple runs or are the shown results specific of a single run? The difference between the curves seems often within the transparent error bars, but it's unclear what those bars indicate.

Fig 4.: why is the delta decreasing over training? Isn't this a rotated dataset? or is this only using planar rotations along the vertical axis?

RQ5 / Fig 6: is this a surprising result? If I understand correctly, the single layer considered is just a linear layer right? Isn't this simply the statement of theorem 1? Or is Fig 6 communicating anything more?

5 comments
AuthorsOpenReview ↗

Response to Reviewer TTmx

We thank the reviewer for the detailed reading, and for recognizing the single interpretable δ\delta and the primal-dual training of δ\delta. We answer each question and weakness below.

W1: Is the equivariant branch the identity? Is it under-parameterized?

No. The equivariant branch is a complete equivariant layer, not an identity path. The confusion came from our notation. In the submitted Eq. 4, ψin(ℓ)\psi_{\mathrm{in}}^{(\ell)} already denoted the output of the layer's equivariant message passing (Algorithm 1, L4–L5). The text did not say so, and the sentence "Leq(x)≡⨁ℓδℓψin(ℓ)L_{\mathrm{eq}}(x) \equiv \bigoplus_\ell \delta^{\ell}\psi_{\mathrm{in}}^{(\ell)}" read as a bare scaling of the input.

The revised Sec. 3.1 makes this explicit. Each layer first runs its full equivariant message passing: tensor products of the input features with spherical harmonics, weighted by learned radial functions and aggregated over neighbors. We write u=⨁ℓu(ℓ)u=\bigoplus_\ell u^{(\ell)} for its output. The revised Eq. 4 is

ψout=⨁ℓδℓu(ℓ)+(1−δ)W(u).\psi_{\mathrm{out}} = \bigoplus_{\ell} \delta^{\ell} u^{(\ell)} + (1-\delta) \mathbf W(u).

So δℓ\delta^{\ell} scales the output blocks of a learned equivariant layer. It is not the only equivariant parameter. On the reviewer's three points:

  • A learnable scalar per ℓ\ell. The message passing has learnable weights for every tensor-product path (ℓin⊗ℓfilter→ℓout\ell_{\mathrm{in}} \otimes \ell_{\mathrm{filter}} \to \ell_{\mathrm{out}}), far more than one scalar per degree.
  • Multiple copies of each irrep. Yes. Each irrep has several channels, and every input–output channel pair has its own weight.
  • Independent parameters per layer. Yes. Every layer has its own message-passing weights and its own unconstrained branch W\mathbf W.

At δ=1\delta = 1 the layer returns ψout=u\psi_{\mathrm{out}} = u. The model is then exactly the strict equivariant backbone with all of its parameters: SEGNN on N-body and QM9, and our tensor-product message-passing network on ModelNet40 and Oxford RobotCar. So the model is not an unconstrained model plus an identity-constrained residual path. It is a full equivariant network whose output blocks are graded by δℓ\delta^{\ell}, plus an unconstrained branch weighted by (1−δ)(1-\delta). The only global quantity is δ\delta.

W2: Sec. 3.4–3.5 read as a post-hoc justification

Is it post hoc? No. Eq. 4 is obtained from the operator A(δ)\mathbf A(\delta) shown in Fig. 1 and Eq. 8. Sec. 3.4 defines A(δ)\mathbf A(\delta) and derives Eq. 5 from it. Sec. 3.5 then obtains the layer of Eq. 4 from Eq. 5. We stated Eq. 4 first only for readability.

The construction, step by step.

  1. Operator (Eq. 8, Fig. 1). A(δ)=A0+δC+(1−δ)W\mathbf A(\delta)=\mathbf A_0+\delta\mathbf C+(1-\delta)\mathbf W. In Fig. 1, the three tori are A0\mathbf A_0 and the hub is δC+(1−δ)W\delta\mathbf C+(1-\delta)\mathbf W.
  2. Split (Eq. 9). We exponentiate A(δ)\mathbf A(\delta). The Taylor series terms that contain W\mathbf W carry the factor (1−δ)(1-\delta). They form the non-equivariant part (1−δ)ψflat(δ)(1-\delta)\psi_{\text{flat}}(\delta). The other terms form the equivariant part ψeq(δ)\psi_{\text{eq}}(\delta). This gives the weight (1−δ)(1-\delta) of the unconstrained branch in Eq. 4.
  3. Degrees (Eqs. 10–12). ψeq(δ)\psi_{\text{eq}}(\delta) is split into three blocks, one per degree ℓ∈{0,1,2}\ell\in\lbrace 0,1,2\rbrace. Proposition 2 replaces each block by δℓψ~ℓ\delta^{\ell}\tilde\psi_\ell. This gives the factor δℓ\delta^{\ell} in Eq. 4. Steps 2 and 3 together are Eq. 5.
  4. Layer (Eq. 4, Sec. 3.5). Eq. 4 has the same form as Eq. 5, with features in place of matrices: the feature block ψin(ℓ)\psi_{\mathrm{in}}^{(\ell)} (u(ℓ)u^{(\ell)} in the revised version) in place of ψ~ℓ\tilde\psi_\ell, and the learned branch W\mathbf W in place of ψflat(δ)\psi_{\text{flat}}(\delta).

Which components are relaxed? Two, and the revised version states both. In step 3, δℓψ~ℓ\delta^{\ell}\tilde\psi_\ell is a surrogate for the block. It is exact at δ=1\delta=1, and we present it as a design choice. In step 4, the network uses features and tensor products in place of the matrices of the operator. All other steps are exact. Neither relaxation affects the equivariance of the layer: the scaling by δℓ\delta^{\ell} keeps equivariance, and all equivariance error comes from (1−δ)W(1-\delta)\mathbf W (Theorem 1).

Is it the right tool? The construction gives the form of the layer: one δ\delta for both equivariant and non-equivariant branches, and an attenuation δℓ\delta^{\ell} that grows with the degree. Table R1 (general response; Table 4 in the revised version) tests this attenuation on ModelNet40.

AuthorsOpenReview ↗

Response to Reviewer TTmx

Q1: Notation in Eq. 4

We use ρ\rho and ρ^\hat\rho only for representations: the group actions on a layer's input and output spaces. The features are ψin\psi_{\mathrm{in}} and ψout\psi_{\mathrm{out}}, and ψ(ℓ)\psi^{(\ell)} is the feature block that transforms under the Wigner-D matrix Dℓ\mathcal{D}^{\ell}. The submitted text blurred the two by writing that ρ(g)x\rho(g)x "corresponds to" ψin\psi_{\mathrm{in}}. We removed that sentence in the revised version. The revised Sec. 3.1 defines features and representations separately before Eq. 4, and Eq. 4 is now written with the message-passing output uu (see W1).

Q2: Terms in Eq. 5

We agree that Eq. 5 used these symbols before they were defined. In the revised version, it sits at the end of Sec. 3.4, after all of its symbols are defined, as Reviewer yXhG also requested. The symbols are:

  • A(δ)\mathbf A(\delta): a 3×33\times3 matrix. It is the operator A0+δC+(1−δ)W\mathbf A_0+\delta\mathbf C+(1-\delta)\mathbf W of Eq. 8.
  • ψ~ℓ\tilde\psi_\ell: a 3×33\times3 matrix. It is the degree-ℓ\ell part of exp⁡(A0+δC)\exp(\mathbf A_0+\delta\mathbf C) at δ=1\delta=1 (Eq. 12).
  • ψin(ℓ)\psi_{\mathrm{in}}^{(\ell)}: a feature vector, not a matrix. It is the degree-ℓ\ell block of the features of the layer.
  • ψflat(δ)\psi_{\mathrm{flat}}(\delta): a 3×33\times3 matrix. It is the sum of all terms of exp⁡(A(δ))\exp(\mathbf A(\delta)) that contain W\mathbf W (Eq. 9). It is the non-equivariant part.

Difference between ψ~ℓ\tilde\psi_\ell and ψin(ℓ)\psi_{\mathrm{in}}^{(\ell)}. ψ~ℓ\tilde\psi_\ell belongs to the operator A(δ)\mathbf A(\delta), and ψin(ℓ)\psi_{\mathrm{in}}^{(\ell)} belongs to the network. Eq. 5 is a relation between matrices. The layer of Eq. 4 has the same form with features: the feature block ψin(ℓ)\psi_{\mathrm{in}}^{(\ell)} takes the place of the matrix ψ~ℓ\tilde\psi_\ell, and the learned branch W\mathbf W takes the place of ψflat(δ)\psi_{\mathrm{flat}}(\delta). (The revised version writes Eq. 5 between matrices only, without ψin\psi_{\mathrm{in}} on the left-hand side, and writes the feature block as u(ℓ)u^{(\ell)}).

Q3: Dual update in Alg. 2

Yes. λ\lambda is the multiplier of the constraint ϵ(δ)≤τ\epsilon(\delta)\le\tau in the Lagrangian L\mathcal{L}, and the step is a dual ascent on L\mathcal{L}, not on ss. Alg. 2 abbreviated it as "ascent on ss" in L8, because the step direction ϵ(δ)−τ\epsilon(\delta)-\tau is the quantity that defines ss. With the projection onto λ≥0\lambda\ge0, the update is

λ←max⁡(0,λ+ηd(ϵ(δ)−τ)),\lambda \leftarrow \max\big(0, \lambda + \eta_d(\epsilon(\delta) - \tau)\big),

taken with Adam in practice. λ\lambda rises while the constraint is violated, falls once it is met, and stays non-negative. In each training step, θ\theta and δ\delta first descend L\mathcal{L} with the current λ\lambda, and λ\lambda is then updated. Sec. 3.3 and Alg. 2 now write the update explicitly in the revised version.

AuthorsOpenReview ↗

response to reviewer TTmx

Q4: What is M\mathcal{M}? The Euclidean hub? Is C\mathbf C equivariant?

What is M\mathcal{M}? M\mathcal{M} is the torus–hub space drawn in Fig. 1: three tori joined at a central hub. Each torus is one toroidal direction and carries one rotation generator, Wx\mathbf W_x, Wy\mathbf W_y or Wz\mathbf W_z. Formally, M=T3×R2\mathcal{M}=\mathbb{T}^3\times\mathbb{R}^2 (App. A.5). The submission used M\mathcal{M} before introducing it. The revised version introduces it with the operator of Eq. 8.

What is the Euclidean hub? It is the central part of Fig. 1, where the three tori meet. In Eq. 8, it is the term δC+(1−δ)W\delta\mathbf C+(1-\delta)\mathbf W.

Is C\mathbf C equivariant? C=∑i<j[Wi,Wj]\mathbf C=\sum_{i < j}[\mathbf W_i,\mathbf W_j] is the commutator of the generators and sits in the hub of Fig. 1. It is a 3×33\times3 matrix of degree ℓ=1\ell=1, like the generators. It belongs to the equivariant part ψeq\psi_{\text{eq}} (Eq. 9). Only W\mathbf W is unconstrained.

Does Fig. 1 help to understand Eq. 5? We agree that the link was not explicit. Fig. 1 does not show Eq. 5 itself. It shows the operator A(δ)\mathbf A(\delta) of Eq. 8, from which Eq. 5 is derived. The three tori are A0=Wx+Wy+Wz\mathbf A_0=\mathbf W_x+\mathbf W_y+\mathbf W_z. The hub is δC+(1−δ)W\delta\mathbf C+(1-\delta)\mathbf W, and its shade shows δ\delta. The sum of the three tori and the hub is A(δ)\mathbf A(\delta). The revised Eq. 8 is annotated with these parts of Fig. 1.

Eq. 5 follows from A(δ)\mathbf A(\delta) in three steps. We exponentiate A(δ)\mathbf A(\delta). Its Taylor expansion is a sum of matrix products of A0\mathbf A_0, C\mathbf C and W\mathbf W. First, we split this sum (Eq. 9). The products that contain W\mathbf W carry the factor (1−δ)(1-\delta), and their sum is (1−δ)ψflat(δ)(1-\delta)\psi_{\text{flat}}(\delta). The products without W\mathbf W contain only A0\mathbf A_0 and C\mathbf C, and their sum is ψeq(δ)\psi_{\text{eq}}(\delta). Second, we decompose ψeq(δ)\psi_{\text{eq}}(\delta) by degree (Eqs. 10–11). Each of its products, e.g. A02\mathbf A_0^2 or C2\mathbf C^2, is a 3×33\times3 matrix. A 3×33\times3 matrix has a part of degree ℓ=0\ell=0 (its trace), ℓ=1\ell=1 (its antisymmetric part) and ℓ=2\ell=2 (its symmetric traceless part). We collect the parts of degree ℓ\ell from all products into one block ψeq,ℓ(δ)\psi_{\text{eq},\ell}(\delta). So ψeq(δ)\psi_{\text{eq}}(\delta) is the sum of three blocks, ℓ=0,1,2\ell=0,1,2. Third, Proposition 2 (Eq. 12) replaces each block ψeq,ℓ(δ)\psi_{\text{eq},\ell}(\delta) by δℓψ~ℓ\delta^{\ell}\tilde\psi_\ell, where ψ~ℓ\tilde\psi_\ell is the block at δ=1\delta=1. The three blocks δℓψ~ℓ\delta^{\ell}\tilde\psi_\ell and (1−δ)ψflat(δ)(1-\delta)\psi_{\text{flat}}(\delta) together are Eq. 5.

Q5: Two arguments in Sec. 3.4; is ℓ∈{0,1,2}\ell \in \lbrace 0, 1, 2\rbrace a limitation?

Two arguments. We have separated the two arguments in the revised version. To summarize Sec. 3.4: we first define the theoretical operator A(δ)\mathbf A(\delta) in Eq. 8.

First argument (Eq. 9). We exponentiate the operator and split its Taylor expansion into two parts, ψeq(δ)\psi_{\text{eq}}(\delta) and (1−δ)ψflat(δ)(1-\delta)\psi_{\text{flat}}(\delta). ψeq(δ)\psi_{\text{eq}}(\delta) contains only A0\mathbf A_0 and C\mathbf C, which are 3×33\times3 matrices of degree ℓ=1\ell=1. We call it the equivariant part. ψflat(δ)\psi_{\text{flat}}(\delta) contains every term with the unconstrained dense matrix W\mathbf W. It is the non-equivariant part and carries the weight (1−δ)(1-\delta). This step does not involve ℓ\ell.

Second argument (Eqs. 10–12). The terms of ψeq(δ)\psi_{\text{eq}}(\delta) are matrix products of A0\mathbf A_0 and C\mathbf C. We decompose them by degree (Clebsch–Gordan) and group them into the blocks ℓ∈{0,1,2}\ell\in\lbrace 0,1,2\rbrace. We then apply Proposition 2 to each block, which gives the factor δℓ\delta^{\ell}.

Combining the two gives Eq. 5. We made these steps explicit in the revised manuscript.

Is ℓ∈{0,1,2}\ell \in \lbrace 0,1,2\rbrace a strict limitation? No. It comes from the example used in Sec. 3.4, not from the method. In Sec. 3.4, A0\mathbf A_0 and C\mathbf C of Eq. 8 are 3×33\times3 matrices. A 3×33\times3 matrix has only three parts: its trace (ℓ=0\ell=0), its antisymmetric part (ℓ=1\ell=1), and its symmetric traceless part (ℓ=2\ell=2). So only these degrees appear in Eq. 10. With larger matrices, higher degrees appear and follow the same δℓ\delta^{\ell} law (shown in App. A.3). The network has no such restriction. It scales every irrep block of degree ℓ\ell by δℓ\delta^{\ell}, for any ℓ\ell the backbone uses. Our ModelNet40 network uses features up to ℓ=2\ell=2.

Reviewer_2BiVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work provides a novel relaxation of the equivariant constraint in equivariant neural network architectures that is controlled by a single learnable parameter δ\delta and is able to achieve different levels of relaxation for irreducible representations of different orders. The authors propose that higher-order representations should be attenuated more strongly when performing relaxation of the equivariant constraint, which they motivate on the assumption that higher-order features tend to be less reliable and harder to learn, as they correspond to features varying with much higher frequency within the group. Based on the previous work of Manolache et al. (2025) they showcase how this variable attenuation of features of different orders allows them to outperform such previous works in a set of experiment that focus on SO(3) and SE(3) equivariant networks.

Critical strengths
  • The proposed relaxation of equivariant networks is easily implementable and ties really well with previous works, while it also provides a more granular way to control the level of relaxation of irreducible representations of different types.
  • The authors' argument about attenuating higher-order degrees more strongly, and its connection to the general spectral bias in neural network learning, is a novel perspective on the problem of learning relaxation in equivariant neural networks.
Critical weaknesses
  • The authors, while motivating the use of δl\delta^l for attenuating different types of representations with different levels of relaxation they do not provide any ablation study that showcases that this choice is actually useful. Why, for example, is using 11 for the scalar features and δ\delta for all other representations not as effective?
  • The use of a single δ\delta for controlling the relaxation level of the whole network is not motivated enough. Why is the ability to learn different levels of relaxation for different layers of the network something that can create problems during learning?
  • There is limited discussion or comparison with other works that propose learning different levels of relaxation for different irreducible representation orders. Most notably: Lars Veefkind, Gabriele Cesa "A Probabilistic Approach to Learning the Degree of Equivariance in Steerable CNNs"
Questions for authors
  • How do the authors verify that δl\delta^l is actually a better strategy for relaxing the equivariant constraint of different irreducible representations than the simpler approach of learning different levels of relaxation for different irreducible representations as separate parameters?
  • Why is it important to have a single parameter controlling the level of relaxation for all the layers in the network?
  • How does the proposed method behave with different irreducible representations of the same order?
3 comments
AuthorsOpenReview ↗

response to reviewer 2BiV

We thank the reviewer for the thorough review, and for recognizing the granular control over irreps and the spectral-bias perspective.

W1 & Q1: Is δℓ\delta^{\ell} useful? Why not uniform grading, or separate per-degree parameters?

We tested this directly. Table R1 (general response) compares, with identical architecture, constraint, seeds and training, four per-degree laws: uniform grading (c0=1c_0 = 1, cℓ=δc_\ell = \delta for ℓ≥1\ell \geq 1, as the reviewer suggests), the reverse ordering (c1=δ2c_1 = \delta^2, c2=δc_2 = \delta), no attenuation (cℓ=1c_\ell = 1, i.e. a single blend weight), and ours (cℓ=δℓc_\ell = \delta^{\ell}). Relative to ours, uniform grading costs about 2 points of accuracy (class 77.6 vs 79.5; instance 84.3 vs 86.1), although both variants converge to the same δ≈0.87\delta \approx 0.87. No attenuation costs about 1.5 points (77.9 / 84.8), and the reverse ordering about 3 points (76.4 / 83.2). All differences exceed the seed standard deviations (≤0.7\le 0.7 points). So the attenuation, its frequency ordering, and the direction of that ordering all matter, and δℓ\delta^{\ell} is the best of the four.

On separate per-degree parameters: Table R1 covers this case. In our network (ℓ≤2\ell\le2), separate parameters are two factors, c1c_1 for ℓ=1\ell=1 and c2c_2 for ℓ=2\ell=2. The table tests every ordering of the two: c1>c2c_1 > c_2 (ours), c1=c2c_1=c_2 (uniform), c1<c2c_1 < c_2 (reverse), and no factors at all (c1=c2=1c_1=c_2=1). Only c1>c2c_1 > c_2 wins. The other three lose by 1.51.5 to 33 points. So what matters is the ordering of the two factors, not more freedom in choosing them. More freedom does not help along depth either: per-layer δi\delta_i, with three more parameters, do not improve on one δ\delta. Separate parameters would have to rediscover the right ordering from the data. The tied law δℓ\delta^{\ell} has two properties that they lack:

  1. A built-in ordering. δℓ\delta^{\ell} decreases in ℓ\ell for every δ∈(0,1)\delta \in (0, 1), so higher orders are always relaxed more. This ordering is what the ablation shows to be beneficial, and it holds without having to be learned.
  2. One parameter for both roles. The same δ\delta sets how much each order is trusted and how much weight the unconstrained branch receives, so the two move together. This holds at any ℓmax⁡\ell_{\max} and adds no parameters.

W2 & Q2: Why a single δ\delta for all layers?

We do not claim that per-layer relaxation causes problems. Shared and layer-wise relaxation are both established options; Veefkind & Cesa (2024) offer both. We chose a shared δ\delta for three reasons:

  • The constraint is network-level. It acts on the measured equivariance error of the whole network, not on individual layers, so one δ\delta is its natural variable. ACE, which learns one coefficient per layer, uses one constraint per layer.
  • Per-layer allocation still emerges. Each layer has its own branch W\mathbf W. By Theorem 1, layer ii's error is at most (1−δ)C(1-\delta)C, so δ\delta sets a global budget and each W\mathbf W determines its share. Table R2 shows this on ModelNet40: the relative error ranges from 0.0850.085 in the first layer to 0.330.33 in the last.
  • Per-layer δi\delta_i add little. A variant with one learned δi\delta_i per layer (Table R1) ends with δ0>δ1>δ2>δ3\delta_0 > \delta_1 > \delta_2 > \delta_3, but with a small spread (≈0.015\approx 0.015), and accuracy almost matches that of the global δ\delta as in Table R1 (general response and Fig.7b in revised manuscript).

A hand-specified depth split (OursSER\mathrm{Ours_{SER}} on ModelNet40) underperforms the learned global δ\delta (class accuracy 78.77 vs 79.54; instance accuracy 85.03 vs 86.09).

AuthorsOpenReview ↗

response to reviewer 2BiV

W3: Comparison with Veefkind & Cesa (ICML 2024)

Veefkind & Cesa replace the uniform group averaging in the steerable kernel projection with a learnable likelihood over the group, parametrized by Fourier coefficients per irrep frequency, layer-wise or shared. Equivariance is broken inside the kernel, without additional layers. Our method differs in what is learned and how it is controlled:

  • What is learned. Their likelihood specifies which transformations remain symmetries. Our δ\delta specifies how much each irrep degree is trusted relative to unconstrained capacity.
  • Parametrization. Free Fourier coefficients versus a single δ\delta with a tied, frequency-ordered law. Their bandlimiting of high frequencies is close in spirit to our spectral-bias motivation.
  • Control. The equivariance error of our network is measured during training and kept within an explicit budget τ\tau by the dual update, and each layer's error is bounded linearly in (1−δ)(1-\delta) (Theorem 1). Their degree of equivariance is regularized but not budgeted, and they note that no proof links the learned likelihood to the true equivariance (their Sec. 6). The trade-off weight λ\lambda is learned to meet the interpretable target τ\tau, rather than set by hand as a regularization weight such as their alignment and KL terms. (Both methods are exactly equivariant at their strict endpoint; the difference is the control of the relaxed regime.)
  • Integration. Our method wraps an existing irrep-feature network without modifying its operators, at negligible parameter cost. It therefore applies directly to tensor-product networks, which have no kernel projection to modify.

We added this discussion to Related Work.

Q3: Irreps of the same order

They are treated the same. The factor δℓ\delta^{\ell} depends only on the order ℓ\ell. So all irreps of order ℓ\ell receive the same factor: every channel of that order, and both parities (e.g., 1o1o and 1e1e). The reason is Assumption 2. The order ℓ\ell is the angular frequency, and we attenuate by frequency. Irreps of the same order have the same frequency, so they get the same attenuation. Multiplying all of them by the same number δℓ\delta^{\ell} also keeps each of them equivariant.

AuthorsOpenReview ↗

Official comment by the authors

Dear Reviewer 2BiV,

Thank you again for reviewing our paper and for your comments. As the rebuttal period is ongoing, please let us know if you have any follow-up questions. We will clarify them.

Kind regards, the authors

Reviewer_yXhGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper is devoted to the problem of relaxed equivariant approaches, specifically the coefficient that balances between strictly equivariant and non-equivariant parts. The paper proposes to tune (by learning) a single real parameter δ\delta (taking values from 0 to 1) that controls the balance between the unconstrained part and equivariant part. The authors try to justify the approach by referring to the geometric interpretation of the related operator in associative operator algebra. In general, the approach with learnable δ\delta improves performance of the relaxed equivariant models on N-Body, QM9 molecular property prediction, 3D shape classification on ModelNet40, and LiDAR pose estimation on OxfordRobotCar.

Critical strengths

The idea of combining the equivariant and non-equivariant parts in the relaxed equivariant model using a tunable parameter seems to be natural, and I believe that it may lead to outperforming results in experiments.

Critical weaknesses

1)I am quite critical of the significance of the obtained results. The approach seems novel but probably too straightforward. So, I agree that the idea with a tunable coefficient between equivariant and non-equivariant parts is progress in this area, but the proposed method does not seem to be a sort of breakthrough. The idea of such a tunable parameter between two different models in general looks quite straightforward. 2) If I understood correctly, in the proposed approach, every layer shares the same relaxation strength δ\delta. So, the method cannot discover that early layers should be more equivariant and late layers more flexible, etc. This is a bit strange that the whole model shares the same δ\delta.

Comment on the presentation of results: Lines 155–161 look strange and difficult to understand at this point in the reading. I got lost trying to understand what A(δ)A(\delta), what is ψflat\psi_{flat}, and how all this is related to each other, because this notation has not been introduced yet (and it turns out it will be introduced later). I suggest either adding comments on this notation here or moving this part further in the text.

Questions for authors

What makes the proposed approach not straightforward? Is it not natural (and in some sense obvious) that if we want to balance between two models (e.g., an equivariant and a non-equivariant one), the best idea is to put a learnable coefficient between them, and if trained well, it will give better results compared to the counterpart without tuning? I would like to ask the authors to clarify what the difficulty was in introducing such a model (and what was not so straightforward), to make the significance clearer and to prevent the solution from being treated as merely an engineering one. I am sorry if I missed the value, and I would be grateful for comments that would help me understand it.

5 comments
AuthorsOpenReview ↗

response to reviewer yXhG

We thank the reviewer for the careful reading and the constructive questions.

W1 & Q1: What is not straightforward?

What previous methods do. Relaxed-equivariance methods add a non-equivariant branch or weights to an equivariant branch or weights and blend the two either with one scalar: hand-tuned, or multiple scalars per layer (learned or hand tuned). The scalar decides how much to relax. The equivariant branch itself is left untouched, so every irrep order is relaxed by the same amount. (This we already explained in abstract, introduction and related work.)

What we add. We also use one scalar which is learned and it has two roles: how much to relax i.e. blending and where to relax. We achieve this by scaling each irreps features ℓ\ell by δℓ\delta^{\ell} and blend the equivariant and non-equivariant branch by (1−δ)(1-\delta), where δ∈[0,1]\delta \in [0,1]. The scaling of irrep features follows the Rahaman et al. spectral bias for neural network observations. The invariant ℓ=0\ell=0 block is not scaled or attenuated. ℓ=1\ell=1 block attenuated by δ\delta and ℓ=2\ell=2 block is attenuated by δ2\delta^2. So δ\delta decides not only how much to relax but also where: higher orders are attenuated first. For example, at δ=0.88\delta=0.88 the equivariant branch keeps 100% of its ℓ=0\ell=0 content, 88% of ℓ=1\ell=1 and 77% of ℓ=2\ell=2, while the non-equivariant branch enters with weight 0.120.12 in every order. The share removed from a high order is what the non-equivariant branch compensates for.

Why later layers cannot undo this. In each irrep order, the scaled equivariant signal and the non-equivariant signal are added into the same channels before the next layer sees them. The next layer receives only their sum. It can rescale the sum, but it cannot separate the two parts or change their proportion. The ratio δℓ:(1−δ)\delta^{\ell}:(1-\delta) is therefore a property of the layer itself, not something downstream weights can reverse.

Why one δ\delta still gives each layer its own level of relaxation. Every layer has its own non-equivariant branch W\mathbf W. δ\delta sets which orders are attenuated, and all equivariance error comes from (1−δ)W(1-\delta)\mathbf W (Theorem 1). How much of that error a layer actually produces depends on its own W\mathbf W. Measured per layer (Table R2 in general response above and Fig.7 in revised manuscript), the relative equivariance error on ModelNet40 rises from 0.0850.085 in the first layer to 0.330.33 in the last. A single global δ\delta thus already yields depth-dependent relaxation, which is what the reviewer expected only a per-layer parameter could provide.

Why the law is not a tuning choice. Two points were not straightforward:

  1. The exponent comes from analysis. In our operator analysis (Sec. 3.4), the equivariant part is split into blocks of degree ℓ\ell. In the block of degree ℓ\ell, the terms with at least ℓ\ell factors of the commutator C\mathbf C have the common factor δℓ\delta^{\ell}. Proposition 2 keeps these terms, which gives the attenuation law δℓ\delta^{\ell}. The exponent is the irrep degree, not a tuned hyperparameter.
  2. Grading preserves exact equivariance. Scaling an irrep block by a scalar commutes with the rotation matrices, so the grading adds no equivariance error. All error comes from (1−δ)W(1-\delta)\mathbf W, and it is bounded linearly in (1−δ)(1-\delta) (Theorem 1).

Evidence. δ\delta measurably restructures the irrep composition (RQ4; Fig.5). The new ablation (general response, Table R1 and Table 4 in revised manuscript) isolates the second role of δ\delta. With the same architecture, constraint and seeds, removing the frequency ordering (uniform grading) costs about 2 points of accuracy (class 77.6 vs 79.5; instance 84.3 vs 86.1), although both variants converge to the same δ≈0.87\delta \approx 0.87. Removing the attenuation altogether, which leaves a single learned blend weight, costs about 1.5 points (77.9 / 84.8). Inverting the ordering costs about 3 points (76.4 / 83.2). All differences exceed the seed standard deviations (≤0.7\le 0.7 points). The gain therefore comes from where δ\delta relaxes, not from the blending scalar alone.

AuthorsOpenReview ↗

response to reviewer yXhG

W2: Every layer shares the same δ\delta

This is a deliberate choice; shared and layer-wise relaxation are both established options (e.g., Veefkind & Cesa, 2024 or ACE). A single δ\delta does not remove per-layer allocation:

  • Per-layer branches under a global budget. Each layer has its own unconstrained branch Wi\mathbf W_i, and the constraint acts on the network's measured equivariance error. By Theorem 1, layer ii's error is bounded by (1−δ)Ci(1-\delta)C_i, with Ci=2(Li∥xi∥+∥Wi(xi)∥)C_i = 2\big(L_i\Vert x_i\Vert + \Vert\mathbf W_i(x_i)\Vert\big). δ\delta sets a global budget, and each Wi\mathbf W_i determines how much of it its layer uses. Table R2 in general response above measures this on ModelNet40: the relative error ranges from 0.0850.085 in the first layer to 0.330.33 in the last.
  • A hand-specified depth split does not help. OursSER\mathrm{Ours_{SER}} on ModelNet40 keeps early layers strictly equivariant (δ=1\delta = 1) and frees later layers (δ=0\delta = 0). The learned global δ\delta outperforms it (class accuracy 79.54 vs 78.77; instance accuracy 86.09 vs 85.03).
  • Per-layer δi\delta_i add little. We also trained a variant with one learned δi\delta_i per layer (Table R1 in general response above and Fig.7(b) in revised manuscript). The δi\delta_i end in the order δ0>δ1>δ2>δ3\delta_0 > \delta_1 > \delta_2 > \delta_3, i.e., early layers more equivariant and later layers more relaxed, but with a small spread (≈0.015\approx 0.015), and accuracy does not improve over the global δ\delta (class 79.0 vs 79.5; instance 86.1 vs 86.1).

Presentation: lines 155–161

We agree. This paragraph contains Eq. 5, which used A(δ)\mathbf A(\delta), ψ~ℓ\tilde\psi_\ell and ψflat\psi_{\mathrm{flat}} before they were defined. In the revised version, we moved it to the end of Sec. 3.4, after all of its symbols are defined. In short:

  • A(δ)=A0+δC+(1−δ)W\mathbf A(\delta)=\mathbf A_0+\delta\mathbf C+(1-\delta)\mathbf W is a 3×33\times3 matrix, the operator shown in Fig. 1. A0\mathbf A_0 is the sum of the rotation generators, C\mathbf C is their commutator, and W\mathbf W is a learned matrix with no equivariance constraint.
  • ψflat(δ)\psi_{\mathrm{flat}}(\delta) is the sum of all terms of exp⁡(A(δ))\exp(\mathbf A(\delta)) that contain W\mathbf W. It is the non-equivariant part and has the weight (1−δ)(1-\delta).
  • The terms without W\mathbf W form the equivariant part. It splits into three parts, of degree ℓ=0\ell=0, 11 and 22. ψ~ℓ\tilde\psi_\ell is the part of degree ℓ\ell, with δ\delta set to 11. In Eq. 5, it is multiplied by δℓ\delta^{\ell}.

Eq. 5 says that one δ\delta weights both parts, and that the equivariant part is attenuated by degree with the exponent ℓ\ell. The layer of Eq. 4 has the same form, with the features of the network in place of these matrices.

We hope these clarifications address the reviewer's concerns, and we would be grateful if they could be taken into account in the final assessment.

AuthorsOpenReview ↗

official comment by authors

Dear reviewer yXhG,

Thank you again for reviewing our paper and your comments. As the rebuttal period is ongoing, please let us know if you have any follow-up questions. We will clarify them.

Kind regards, the authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers for their careful reading. We first clarify the method, then summarize the new experiments and revisions. Individual responses follow.

What the method is

Each layer first runs the full strict backbone layer (e.g., tensor-product message passing with learned weights). We write uu for its output. δ\delta then plays two roles:

ψout=⨁ℓδℓu(ℓ)+(1−δ)W(u).\psi_{\mathrm{out}} = \bigoplus_{\ell} \delta^{\ell} u^{(\ell)} + (1-\delta) \mathbf W(u).

At δ=1\delta = 1, the model is exactly the strict backbone. For δ<1\delta < 1, each irrep degree ℓ\ell is weighted by δℓ\delta^{\ell}, and the unconstrained branch W\mathbf W by (1−δ)(1-\delta). In degree ℓ\ell, equivariant and unconstrained content therefore enter in the ratio δℓ:(1−δ)\delta^{\ell} : (1-\delta). Most prior methods control only how much to relax. Ours also controls where: per irrep degree, inside the equivariant branch, with the same δ\delta. Veefkind & Cesa also act per frequency, but inside the steerable kernel and with free coefficients.

MethodHow relaxation is controlledWhere it acts
REMULWeight of an added equivariance loss, hand-tuned or adapted (GradNorm)Whole unconstrained model
Pertigkiozoglou et al.Unconstrained term with a fixed schedule, removed at inferencePer layer
ACEOne coefficient per layer, learned by constrained optimizationPer layer
Veefkind & CesaLearned likelihood over the group (Fourier coeff.)Per frequency, inside the kernel
OursOne δ\delta learned under an equivariance-error constraintPer irrep degree: degree ℓ\ell is scaled by δℓ\delta^{\ell}, and the unconstrained branch by (1−δ)(1-\delta)

New experiments

Table R1 — ablations on ModelNet40. Identical architecture, constraint, seeds, and training; each row changes one thing relative to ours. Mean ± std over 3 seeds.

VariantExtra parametersClass acc.Instance acc.
Reverse ordering: c1=δ2c_1 = \delta^{2}, c2=δc_2 = \delta00.7637 ± 0.00340.8323 ± 0.0064
Uniform grading: c0=1c_0 = 1, cℓ=δc_\ell = \delta for ℓ≥1\ell \geq 100.7761 ± 0.00720.8425 ± 0.0054
No attenuation: cℓ=1c_\ell = 100.7789 ± 0.00400.8481 ± 0.0043
Fixed δ=0.87\delta = 0.87 (not learned), cℓ=δℓc_\ell = \delta^{\ell}00.7822 ± 0.00400.8549 ± 0.0031
Per-layer δi\delta_i (one per layer), cℓ=δiℓc_\ell = \delta_i^{\ell}30.7900 ± 0.00460.8605 ± 0.0030
Ours: one learned δ\delta, cℓ=δℓc_\ell = \delta^{\ell}00.7954 ± 0.00160.8609 ± 0.0035

All rows use the layer ψout=⨁ℓcℓu(ℓ)+(1−δ)W(u)\psi_{\mathrm{out}}=\bigoplus_{\ell} c_\ell u^{(\ell)}+(1-\delta) \mathbf W(u) with ℓ∈{0,1,2}\ell\in\lbrace 0,1,2\rbrace and c0=1c_0=1. The unconstrained branch always has the weight (1−δ)(1-\delta). The rows differ only in the factors c1c_1, c2c_2 and in how δ\delta is set:

  • Ours. c1=δc_1=\delta, c2=δ2c_2=\delta^{2}. One learned δ\delta for the whole network. The higher degree is attenuated more.
  • Reverse ordering. c1=δ2c_1=\delta^{2}, c2=δc_2=\delta. The two factors are swapped, so the lower degree is attenuated more.
  • Uniform grading. c1=c2=δc_1=c_2=\delta. Both degrees get the same factor, so there is no ordering.
  • No attenuation. c1=c2=1c_1=c_2=1. The equivariant branch is not scaled, and δ\delta only sets the weight (1−δ)(1-\delta) of the unconstrained branch. This is a branch-level blend only, the layer form of ACE, with one weight shared across layers.
  • Fixed δ\delta. c1=δc_1=\delta, c2=δ2c_2=\delta^{2} as in ours, but δ\delta is frozen at its initial value 0.870.87. This is the final converged value in our learned setting.
  • Per-layer δi\delta_i. Layer ii has its own learned δi\delta_i, with c1=δic_1=\delta_i, c2=δi2c_2=\delta_i^{2} and branch weight (1−δi)(1-\delta_i). This adds 3 parameters (four δi\delta_i in place of one δ\delta).

Results. Relative to ours, the reverse ordering loses about 3 points, the uniform grading about 2 points, and no attenuation about 1.5 points. These gaps are well beyond the seed standard deviations. So the attenuation helps, and its order matters: the higher degree should be attenuated more. Fixing δ\delta loses 1.3 points of class accuracy and 0.6 of instance accuracy, so learning δ\delta helps. Per-layer δi\delta_i reach the same instance accuracy and a class accuracy 0.50.5 points lower, about one seed standard deviation. So one δ\delta suffices.

AuthorsOpenReview ↗

General response to all reviewers

Table R2 — per-layer equivariance error of our trained models with a single global δ\delta (ModelNet40: δ≈0.876\delta \approx 0.876; N-body: δ≈0.964\delta \approx 0.964). The same numbers are shown as bar plots in Fig.7 of the revised manuscript (left panel: relative error; right panel: error vs. the W\mathbf W-term).

LayerModelNet40: rel. errorModelNet40: errorModelNet40: 2∥(1−δ)Wi∥2\Vert(1-\delta)\mathbf W_i\VertN-body: rel. errorN-body: errorN-body: 2∥(1−δ)Wi∥2\Vert(1-\delta)\mathbf W_i\Vert
10.0850.3350.8343.9×10−43.9 \times 10^{-4}1.01×10−21.01 \times 10^{-2}2.87×10−22.87 \times 10^{-2}
20.2852.0066.0103.6×10−43.6 \times 10^{-4}1.17×10−21.17 \times 10^{-2}3.08×10−23.08 \times 10^{-2}
30.1951.8824.0432.7×10−42.7 \times 10^{-4}9.57×10−39.57 \times 10^{-3}3.02×10−23.02 \times 10^{-2}
40.3272.5424.3663.0×10−43.0 \times 10^{-4}1.29×10−21.29 \times 10^{-2}4.51×10−24.51 \times 10^{-2}

How each column is computed.

  • Data. 30 test batches; one random rotation R∈SO(3)R \in \mathrm{SO}(3) per batch. All quantities are means over nodes and batches.
  • Layer input xix_i. Taken from the trained model's forward pass on the unrotated input: the output of layer i−1i-1 (the embedded input for i=1i = 1). Each layer is therefore evaluated on its actual input.
  • Rotated input. xiD(R)⊤x_i D(R)^{\top}, where D(R)D(R) is the representation of the layer's input irreps. Node and edge attributes are recomputed from the rotated positions (and velocities); the graph topology is kept fixed. This isolates layer ii's own error instead of accumulating errors of earlier layers.
  • Error. ∥Fi(ρ(R)xi)−ρ(R)Fi(xi)∥\Vert F_i(\rho(R)x_i) - \rho(R)F_i(x_i)\Vert, with FiF_i the full layer (equivariant branch, δℓ\delta^{\ell} scaling, unconstrained branch, skip).
  • Relative error. The error divided by the mean norm of Fi(xi)F_i(x_i); comparable across layers and datasets.
  • W\mathbf W-term, 2∥(1−δ)Wi∥2\Vert(1-\delta)\mathbf W_i\Vert. Measured directly: the same layer is run twice, with and without its unconstrained branch (branch output set to zero), and the mean norm of the difference equals ∥(1−δ)Wi(ui)∥\Vert(1-\delta)\mathbf W_i(u_i)\Vert with the trained weights and the checkpoint's δ\delta. We report twice this value, the W\mathbf W-term of the bound in Theorem 1. No quantity is computed from δ\delta separately; δ\delta is given for information only.

On ModelNet40, the relative error varies almost fourfold across depth: the first layer stays closest to equivariant, and deeper layers relax more. On N-body, where the learned relaxation is small, the error is tiny and nearly uniform across layers. In both models, every layer's error is below its W\mathbf W-term, consistent with Theorem 1. Depth-dependent allocation therefore emerges from the per-layer branches Wi\mathbf W_i when relaxation is used, without per-layer relaxation parameters.

Revisions

  • resolved notation overloading in Sec.3.1 to represent representation and features using ρ\rho and ψ\psi respectively (as pointed out by reviewer TTmx)
  • Eq. 4 shows the backbone layer uu explicitly. LeqL_{\mathrm{eq}} is now defined as the δℓ\delta^{\ell} scaling applied to uu (to clarify Reviewer TTmx's comments on Eq. 4).
  • Sec. 3.3 role of lambda made explicit as per concern of reviewer TTmx
  • RQ5 is reframed as a consistency check of Theorem 1.
  • Added one Related Work: Veefkind & Cesa (2024) (addressing comment of reviewer 2BiV)
  • New citations added to further support Assumption 2: Basri et al. (NeurIPS 2019), Bietti & Mairal (NeurIPS 2019), Cao et al. (IJCAI 2021).
  • New ablation study added: Table R1 (Table 4 in the revised manuscript) and Fig.7(a) showing Table R2 and Fig.7(b) showing per-layer variation of different delta for 3 seeds on ModelNet40 during training.
  • Presentation: As per the suggestions of reviewer: yXhG, the theoretical operator paragraph in Sec.3.1 is removed and the theoretical operator is moved further down and defined in Sec.3.4. Now it is Eq. 11
  • Equations 7-10 are clarified for better readability in Sec.3.4 (as per comments of reviewer TTmx and yXhG)
  • Fig.1 caption explains how the theoretical operator A(δ)\mathbf{A}(\delta) is tied to the diagram and also provides a chain of equations from the theoretical operator to the implemented layer LδL_{\delta}, explaining how the layer is derived. (as per reviewer TTmx comment on Fig.1)
  • Conclusion: Added the justification of why our method is not straightforward (as per comment of reviewer yXhG)
AuthorsOpenReview ↗

General comment by authors

Dear Reviewers and Area Chair,

Thank you again for your careful reading and constructive comments. We have posted our responses and uploaded the revised manuscript. In the revised manuscript, each change is marked in the right margin with the code of the reviewer it addresses. Please let us know if any concerns remain or if anything needs further clarification. We will be glad to answer.

Kind regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.