acceptodds
Under review as a conference paper at ICLR 2027

Interpreting SO(n) multiplication: deep networks generalize by learning an algorithm

Abstract

We reverse engineer networks trained on SO(n) multiplication (the rotations of the -dimensional sphere) and discover that adding layers results in networks learning clean generalizing algorithms that are not learned by shallow 1- or 2-hidden-layer networks. In deep networks learning SO(3), we discover this task is solved by neurons learning to activate on different axes of the sphere. These `axial'-neurons combine information to compute the answer to the multiplication via Rodrigues' Rotation Formula, which is an efficient algorithm used to compute the rotation of a vector in space, given an axis and angle of rotation. The story is similar in higher dimensions, but neurons learn to rotate about planes instead of an axis. To our knowledge, this is the first Lie Group that has been reverse-engineered and interpreted, whereas prior work has focused on finite groups, e.g., modular addition (cyclic group), alternating groups, etc.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Manifold learning, representations, group theory, Lie Group

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_5HMDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the internal representations and computational mechanisms that emerge when deep neural networks learn multiplication in the rotation group SO(n). The authors train multilayer perceptrons (MLPs) to perform rotation matrix multiplication and analyze their internal representations using neuron activation analysis, linear probing, representation space decomposition, neuron ablation, and Fourier analysis. The results show that, as network depth increases, hidden representations become more linearly related to the target rotation product matrix and exhibit mathematical structures corresponding to the Rodrigues rotation formula. The authors further identify groups of neurons that encode different components of the formula and observe similar patterns in higher-dimensional rotation groups. Based on these findings, they argue that deep networks learn to implement the Rodrigues algorithm for rotation multiplication and propose this as a mechanism underlying their generalization ability.

The observed depth-dependent patterns in internal representations provide valuable insights into how neural networks encode mathematical operations. However, the current results primarily establish mathematical correspondences between internal representations and the Rodrigues formula, rather than demonstrating that the networks actually use this algorithm to perform rotation multiplication. More importantly, the paper does not explain why such mathematical structures emerge during training or why they become more pronounced with increasing network depth. Although the neuron ablation experiments provide some functional evidence, they do not sufficiently reveal the underlying computational process or establish a clear causal relationship between the proposed mechanism and generalization performance. Overall, while the empirical findings are interesting, the central claims regarding the emergence of the Rodrigues algorithm and its role in generalization go beyond what the current evidence can support.

Critical strengths
  1. Understanding how neural networks learn mathematical operations and develop internal computational mechanisms is an important research problem. Extending previous studies, which have primarily focused on finite groups, to continuous Lie groups also introduces a degree of novelty.

  2. The authors conduct a systematic experimental analysis. The study examines internal representations at multiple levels, from individual neurons and groups of neurons to entire hidden layers. The authors also evaluate the robustness of their findings across different network architectures, training settings, and ablation experiments. These results provide useful insights into how network depth influences the organization and representation of information within neural networks.

Critical weaknesses
  1. The work remains largely descriptive and lacks a deeper understanding of how the internal representations emerge. Through systematic experimental analyses, the authors identify mathematical correspondences between neuron responses and the Rodrigues formula. However, they do not explain why such representations emerge during training, or why they become more pronounced as network depth increases. Consequently, the study mainly provides empirical observations and mathematical interpretations of internal representations, without sufficiently revealing the mechanisms underlying their emergence or the specific role of network depth.

  2. The paper does not sufficiently demonstrate that the networks actually learn and use the Rodrigues algorithm. The authors first observe that the responses of neurons in deeper networks can be described by linear functions of the rotation product matrix R=XY, and then interpret these responses using the Rodrigues formula. However, the Rodrigues formula is itself a mathematical identity satisfied by rotation matrices. In particular, the two Rodrigues components analyzed by the authors can be expressed directly as linear or affine functions of R. Therefore, once a network has developed a linear representation of the rotation product, these mathematical structures may naturally emerge without the network actually implementing the Rodrigues algorithm. The current analysis does not adequately distinguish between these possibilities and thus does not sufficiently support the central claim that the networks have learned and implemented the Rodrigues algorithm.

  3. The relationship between Rodrigues-like representations and generalization remains unclear. Figure 3 shows that although deeper networks develop more pronounced representations of the rotation product, their mean rotation prediction error increases from 0.0175 rad for shallow networks to 0.0246 rad for deeper networks. The authors do not adequately explain this observation. Furthermore, shallow networks also contain many neurons classified as Rodrigues-like, with proportions comparable to those in deeper networks. Their hidden layers can also recover different components of the Rodrigues formula reasonably well and exhibit corresponding Fourier structures. These findings suggest that Rodrigues-like representations are not unique to deeper networks, and their relationship with predictive performance remains unclear. Therefore, the current evidence does not sufficiently establish that such internal representations are a key mechanism underlying network generalization.

Questions for authors
  1. Can the authors explain why Rodrigues-like representations emerge spontaneously during training? The correspondence between hidden representations and the Rodrigues formula becomes more pronounced as network depth increases. Can the authors provide an explanation for this phenomenon in terms of network architecture, optimization dynamics, or representation learning?

  2. How can the authors demonstrate that the networks actually implement the Rodrigues algorithm? The two Rodrigues components analyzed in the paper can be expressed directly as linear or affine functions of the rotation product matrix R=XY. Therefore, a network that encodes R may naturally exhibit these mathematical structures without actually implementing the Rodrigues algorithm. Can the authors provide further evidence to distinguish between these two possibilities?

  3. Why do shallow networks also exhibit similar Rodrigues-like structures? The experiments show that shallow networks contain many Rodrigues-like neurons and can recover different components of the Rodrigues formula and the corresponding Fourier structures within their hidden layers. What are the fundamental differences between the internal computational mechanisms of shallow and deep networks? Can these differences be quantitatively demonstrated?

  4. What is the relationship between Rodrigues-like representations and generalization performance? Figure 3 shows that although deeper networks exhibit more pronounced representations of the rotation product, their mean rotation prediction error increases from 0.0175 rad to 0.0246 rad. How do the authors explain this result? In particular, why do shallow networks achieve lower prediction errors despite exhibiting less pronounced Rodrigues-like representations?

2 comments
AuthorsOpenReview ↗

Official comment by authors

Thank you for your long and detailed review.

Response to weaknesses

The work remains largely descriptive and lacks a deeper understanding of how the internal representations emerge. Through systematic experimental analyses, the authors identify mathematical correspondences between neuron responses and the Rodrigues formula. However, they do not explain why such representations emerge during training, or why they become more pronounced as network depth increases. Consequently, the study mainly provides empirical observations and mathematical interpretations of internal representations, without sufficiently revealing the mechanisms underlying their emergence or the specific role of network depth.

You are correct. This work was meant as an exploratory empirical work to lay the foundations for further deeper mechanistic research. We plan to address deeper mathematical questions about how the representations emerge and relate it to learning dynamics in a future work.

The paper does not sufficiently demonstrate that the networks actually learn and use the Rodrigues algorithm. The authors first observe that the responses of neurons in deeper networks can be described by linear functions of the rotation product matrix R=XY, and then interpret these responses using the Rodrigues formula. However, the Rodrigues formula is itself a mathematical identity satisfied by rotation matrices. In particular, the two Rodrigues components analyzed by the authors can be expressed directly as linear or affine functions of R. Therefore, once a network has developed a linear representation of the rotation product, these mathematical structures may naturally emerge without the network actually implementing the Rodrigues algorithm. The current analysis does not adequately distinguish between these possibilities and thus does not sufficiently support the central claim that the networks have learned and implemented the Rodrigues algorithm.

You are correct; To control for this, we have added new linear regression experiments for the terms KK and K2K^2 which are components of Rodrigues (see response to question 1). We furthermore isolate the neurons which contribute the most to the regression for these two terms, and find that they are mostly non-overlapping neural populations. Furthermore, when running linear regression restricted to the activations of these populations, we find that the population that learns KK has a much higher R2R^2 on the antisymmetric Rodrigues term than on the symmetric Rodrigues term, and vice-versa for the population that learns K2K^2. This shows that distinct populations in each layer encode the different terms in the Rodrigues rotation formula. Hence the network is not solely learning the product rotation then retrieving the Rodrigues terms from that.

The relationship between Rodrigues-like representations and generalization remains unclear. Figure 3 shows that although deeper networks develop more pronounced representations of the rotation product, their mean rotation prediction error increases from 0.0175 rad for shallow networks to 0.0246 rad for deeper networks. The authors do not adequately explain this observation. Furthermore, shallow networks also contain many neurons classified as Rodrigues-like, with proportions comparable to those in deeper networks. Their hidden layers can also recover different components of the Rodrigues formula reasonably well and exhibit corresponding Fourier structures. These findings suggest that Rodrigues-like representations are not unique to deeper networks, and their relationship with predictive performance remains unclear. Therefore, the current evidence does not sufficiently establish that such internal representations are a key mechanism underlying network generalization.

We update our previous claim that deep networks generalize better than shallow ones. Upon further analysis, we now believe that deep networks instead learn a quantitatively different solution than shallow networks and we do not make any claim about improvements in generalization. The details about the difference in solution is found in section 4. In a sentence, there emerge distinct populations of neurons in the deep network for encoding the different terms in Rodrigues' formula, but this division-of-labor does not appear in the shallow network.

Response to questions:

Can the authors explain why Rodrigues-like representations emerge spontaneously during training? The correspondence between hidden representations and the Rodrigues formula becomes more pronounced as network depth increases. Can the authors provide an explanation for this phenomenon in terms of network architecture, optimization dynamics, or representation learning?

Thank you for your question. We did not relate training dynamics to the representations learned as it was beyond the scope of the paper. The correspondence between hidden representations and the Rodrigues formula becomes more pronounced as network depth increases: Intuitively, each layer incrementally refines the neurons to either be strong-l1 or strong-l1, but not both. This is seen in Figure 5 where the percentages of l=1 and l=2 neurons increase with depth.

How can the authors demonstrate that the networks actually implement the Rodrigues algorithm? The two Rodrigues components analyzed in the paper can be expressed directly as linear or affine functions of the rotation product matrix R=XY. Therefore, a network that encodes R may naturally exhibit these mathematical structures without actually implementing the Rodrigues algorithm. Can the authors provide further evidence to distinguish between these two possibilities?

See answer to weakness 2 of reviewer EWpV.

Why do shallow networks also exhibit similar Rodrigues-like structures? The experiments show that shallow networks contain many Rodrigues-like neurons and can recover different components of the Rodrigues formula and the corresponding Fourier structures within their hidden layers. What are the fundamental differences between the internal computational mechanisms of shallow and deep networks? Can these differences be quantitatively demonstrated?

We observe that wide-enough shallow networks have layer activations that linearly encode Rodrigues-like structures such as KK and K2K^2 (see Figure 10) Despite these similarities, there are quantifiable differences in what the shallow and deep networks learn. We find that the overlap in the neuronal populations encoding for the KK and K2K^2 terms is statistically significant in the deep network (especially in the later layers), but not in the shallow network. This shows that in the deep network, the late-layer layers experience encoding specialization, where different populations of neurons are responsible for different terms in Rodrigues' formula (see Figures 11, 12)

What is the relationship between Rodrigues-like representations and generalization performance? Figure 3 shows that although deeper networks exhibit more pronounced representations of the rotation product, their mean rotation prediction error increases from 0.0175 rad to 0.0246 rad. How do the authors explain this result? In particular, why do shallow networks achieve lower prediction errors despite exhibiting less pronounced Rodrigues-like representations?

See answer to Question 4 of reviewer fdDo.

Reviewer_WPwbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors train MLPs to multiply pairs of 3D rotations (SO(3)) across a large sweep of 3,240 networks, and study what the hidden neurons learn. They state that deeper networks come to represent the product of the two input rotations directly in later layers, while shallow networks do not. They link these neurons to Rodrigues’ rotation formula, which is a standard way to describe a rotation by its unit axis and angle. They support their claims with neuron-level analyses, ablations, and a connection to earlier work on finite groups. They also provide a short extension to higher-dimensional rotations (SO(4)).

Critical strengths
  1. Earlier work has looked at what neural networks learn when trained on simple tasks with a finite number of possible inputs, for example, modular addition. This paper studies 3D rotations instead, where there are infinitely many possible inputs. I think that this is a useful new direction for this kind of work.
  2. The experiments are extensive. The authors trained 3,240 networks with many different settings and checked their results on held-out data and against several baselines. Because of this, I trust that the main finding about depth is real, even if I’m less sure how to interpret it (see weaknesses).
Critical weaknesses
  1. I had trouble telling how much of the Rodrigues result is a new finding. Appendix A.4 seems to show that the sin θ and 1 − cos θ patterns follow from the mathematics once a neuron responds linearly to the product rotation. If that is right, I would like the authors to clarify what the Rodrigues analysis shows beyond this, and how the earlier layers compute the product.

  2. Some of the wording was hard for me to match with the reported numbers. The title and abstract suggest that depth is what leads to a generalising solution, but the error seems to go up slightly with depth (0.0175 to 0.0246, lines 239–241). The text also says that depth produces ℓ = 1 and ℓ = 2 neurons, but shallow and deep networks seem to have about the same share of these neurons (lines 256–258). I may be misreading these results, so any clarification from the authors would help.

Questions for authors
  1. Following up on my first weakness: what does the Rodrigues analysis show beyond the neurons being linear in the product rotation?
  2. The abstract mentions SO(n), but the higher-dimensional results seem to focus on SO(4). Did you try larger n, and do you expect the same picture to hold there?
2 comments
AuthorsOpenReview ↗

Official comment by authors

Response to weaknesses

I had trouble telling how much of the Rodrigues result is a new finding. Appendix A.4 seems to show that the sin θ and 1 − cos θ patterns follow from the mathematics once a neuron responds linearly to the product rotation. If that is right, I would like the authors to clarify what the Rodrigues analysis shows beyond this,

Thank you for your feedback. We agree that at first glance, it seems like the sinusoidal structure in the axial neuron activations follows immediately once it learns to respond to the product RR. However we go beyond this and show further that the activations of populations of axial neurons can be used to linearly decode the terms KK and K2K^2, which are independent of thetatheta and instead are functions of the entries of the product rotation axes. We also show that in deep networks, disparate populations of neurons encode KK and K2K^2 (section 4.3). The population for KK is more effective at decoding the antisymmetric Rodrigues term than the population for K2K^2, and vice versa.

and how the earlier layers compute the product.

We conjecture that the earlier layers compute the product by Taylor expanding the activation function applied to the preactivation, which naturally produces product terms. With many neurons, the network cancels higher order terms, leaving as much as the raw product as possible. We plan to expound on this argument in a forthcoming paper.

Some of the wording was hard for me to match with the reported numbers. The title and abstract suggest that depth is what leads to a generalising solution, but the error seems to go up slightly with depth (0.0175 to 0.0246, lines 239–241).

We have amended our original hypothesis that deeper networks generalize better into something more nuanced; Now we instead have evidence to show that the deep and shallow networks learn quantitatively different solutions and make no judgement about differences in generalization error (see section 4.3).

The text also says that depth produces ℓ = 1 and ℓ = 2 neurons, but shallow and deep networks seem to have about the same share of these neurons (lines 256–258). I may be misreading these results, so any clarification from the authors would help.

We have run further experiments to confirm that the proportion of neurons that are ℓ=1\ell=1 and ℓ=2\ell=2 increase with depth. We stand by our original plot in Figure 5 that shows the proportions increasing with depth. We hope this allays your concerns.

Response to questions

Following up on my first weakness: what does the Rodrigues analysis show beyond the neurons being linear in the product rotation?

You are correct; To control for this, we have added new linear regression experiments for the terms KK and K2K^2 which are components of Rodrigues (see response to question 1). We furthermore isolate the neurons which contribute the most to the regression for these two terms, and find that they are mostly non-overlapping neural populations. Furthermore, when running linear regression restricted to the activations of these populations, we find that the population that learns KK has a much higher R2R^2 on the antisymmetric Rodrigues term than on the symmetric Rodrigues term, and vice-versa for the population that learns K2K^2. This shows that distinct populations in each layer encode the different terms in the Rodrigues rotation formula. Hence the network is not solely learning the product rotation then retrieving the Rodrigues terms from that.

The abstract mentions SO(n), but the higher-dimensional results seem to focus on SO(4). Did you try larger n, and do you expect the same picture to hold there?

You are correct; We have edited the manuscript to remove mentions to SO(n) for general nn. We only ran preliminary tests on n=4n=4 and have found evidence for the analogue of Rodrigues' formula for SO(4)SO(4) being learned. However further analyses are needed. We do not expect a priori for analogues of Rodrigues' formula to hold for arbitrarily large nn, all other experimental conditions being equal. However, we predict the emergence of analogues of axial neurons in higher dimensions. Preliminary tests have confirmed this for the SO(4)SO(4) case, but for the general case this requires a mathematical proof which is beyond the scope of this paper.

Reviewer_fdDoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how MLPs learn rotation-matrix multiplication in SO(n), focusing on SO(3) with a shorter extension to SO(4). The authors train 3,240 networks across depths, widths, dataset sizes and optimization settings. They analyze the networks with per-neuron linear fits, axis–angle decompositions, neuron classifications, ablations and population-level decoding.

The main finding is that, with depth, hidden neurons become approximately linear functions of the product R = XY. Median final-layer held-out R^2 rises from 0.017 to 0.988, while linear functions of X, Y and shuffled-product controls explain essentially none of the variance. The authors decompose each neurons fitted functional into irreps and classify neurons whose angular dependence matches the sin and 1-cos terms of Rodrigues rotation formula, and report that these terms can be linearly decoded from whole layers. They also relate the representations to Fourier structure on embedded finite cyclic subgroups. They conclude that deep networks, unlike shallow ones, learn Rodrigues formula as an algorithm for multiplying rotations.

Critical strengths

The papers has a large, well-controlled sweep with an interesting finding. Across 3,240 networks, with separate fit and evaluation sets, later-layer neurons become nearly linear in XY. Linear functions of X, Y, (X, Y), shuffled products and independent products all fit at about zero (Figures 3b and 13). The finding holds across width, data size, learning rate and weight decay (Figure 16a).

Moving group-multiplication interpretability from finite groups to a continuous compact group is also an interesting step.

Critical weaknesses

There is a lot of overclaiming. A lot of empirical claims are implied directly by product linearity: population decoding (Figure 7, Appendix B.9); angular profiles (Figure 4); family agreement (Figure 14); finite-subgroup Fourier results (Section 4.3). Although the ablation results establish that certain neurons are important for performance, they do not demonstrate that the network uses the proposed algorithm: the intermediate computations that transform the two input rotations into the proposed axis–angle components are not identified, nor is it shown that these computations are causally necessary. So all in all, these are not separate empirical observations -- they are a consequence of linearity in XY, and act mostly as consistency checks.

Some of the main claims actually do not follow from the data.

  • Depth actually hurts the task, as geodesic error rises from 0.0175 to 0.0246 going from d = 1 to d = 5 (Section 4.1), and no train/test gap is reported. The neuron-family numbers disagree with each other: the text and Figure 16bc give essentially flat fractions: l = 1 goes from 0.303 to 0.323, and l = 2 from 0.569 to 0.574. Figure 5a shows l = 1 at most about 0.025 and l = 2 at most about 0.35, with zero through layer 2.

  • Population decoding saturates before the neuron-level change. Figure 7b shows Rodrigues decoding reaching R^2 \approx 1 by about layer 1, several layers before product-linear neurons appear. The text (lines 320–323) says the two transitions closely track each other. What depth actually changes is whether individual neurons align with R-linear directions, not whether the Rodrigues terms are represented, as the authors partly acknowledge in lines 359–361.

  • The divide-and-conquer claim in the conclusion has no supporting analysis. The paper does not identify a network computation that divides the input multiplication problem into simpler subproblems and combines their solutions.

Questions for authors
  1. Which family numbers are right? The share of l = 1 and l = 2 neurons differs across the paper: the text and Figure 16 give about 0.32 and 0.57, flat with depth; Figure 5a shows under 0.03 and under 0.35, rising with depth; Figure 4 reports 42 and 899 neurons. The text numbers are close to 3/9 and 5/9, which is what a random matrix would give. Are they actually averages of |A_l|^2/|A|^2 rather than the fraction of neurons classified?

  2. Do the neuron families survive a null test? Would they still appear if each neurons matrix A were randomly rotated (A -> gAh)? The task treats all such rotations the same, so if the families survive, they may come from the analysis rather than the network. Also, are the neurons preferred axes spread evenly, or bunched along the coordinate axes?

  3. Does the network compute the axis or angle of XY? Is there evidence for this beyond the network simply representing XY? In SO(4), the individual Rodrigues terms are not linear in R, so decoding them could be a good test. Please report R^2 and compare against decoding from R directly.

  4. Do deeper networks really generalize better -- they have higher test error? Please report train and test error by depth, and explain in what sense deeper networks generalize better.

  5. How were the ablations done? Which network and layer were ablated? How were neurons removed, and how were the random control neurons chosen? Does the l = 2 effect hold when controls are matched on activation size? Are the networks learning features? Or do they stay close to initialization? Please show the main measurements at initialization too. The learning rate isn’t scaled with width, so does the width effect in Figure 16a hold with learning rates tuned per width, or with muP?

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your review. You, as well as the other reviewers all seem to have the same primary concern: we showed that networks are learning the product of the Rodrigues’ rotation formula, but we did not show that the individual terms are learned by disjoint populations of neurons, and thus we didn’t show that the terms were learned, or that an implementation (algorithm) computing the Rodrigues’ rotation formula was learned. All reviewers pointed out the same reason: the terms of Rodrigues’ Rotation formula are linear within the product matrix R, and thus linear regression should always score highly on them both, even if the two terms weren’t disjointly learned. We’ve bridged this gap in our claims by adding new experiments. To summarize, we take the matrices without the sinusoidal coefficients sin⁡θ\sin \theta or (1−cos⁡θ)(1 - \cos \theta) in the first term of Rodrigues’ and the second term, and do a linear regression in each layer onto each of those. Unlike the terms we fit in the submitted paper, which are linear functions of the product matrix RR, these terms are not. Hence in order to find KK and K2K^2 via linear regression, with an R2R^2 greater than if we decode them from RR directly, the network must learn them individually. Indeed, we acquire the desired result here, but go farther. Next we take the top 25% of neurons that contribute to each matrix in each layer, and test whether the neurons are disjoint. We find that deep networks have disjoint neurons, but shallow networks do not. Thus, while both networks learn

We have fixed this by adding new experiments to show there are two disjoint populations of neurons in deep networks, but not in shallow networks, and we also show that on components of the terms making up Rodrigues’ rotation formula, which can’t be pulled out

Response to weaknesses

Depth actually hurts the task, as geodesic error rises from 0.0175 to 0.0246 going from d = 1 to d = 5 (Section 4.1), and no train/test gap is reported. The neuron-family numbers disagree with each other: the text and Figure 16bc give essentially flat fractions: l = 1 goes from 0.303 to 0.323, and l = 2 from 0.569 to 0.574. Figure 5a shows l = 1 at most about 0.025 and l = 2 at most about 0.35, with zero through layer 2.

Thank you for pointing this out. Upon further analysis and tests, we realized that the values in figure 5 (showing an increasing proportion of ℓ=1,ℓ=2\ell=1, \ell=2 neurons with depth) are the correct ones and we have removed Figure 16 which shows an incorrect flat trend. We modify our original claim of “deeper networks generalize better" to "deeper networks learn a solution that is quantitatively different from the one learned by shallow networks".

Population decoding saturates before the neuron-level change. Figure 7b shows Rodrigues decoding reaching R^2 \approx 1 by about layer 1, several layers before product-linear neurons appear. The text (lines 320–323) says the two transitions closely track each other. What depth actually changes is whether individual neurons align with R-linear directions, not whether the Rodrigues terms are represented, as the authors partly acknowledge in lines 359–361.

Thanks for pointing this out. We acknowledge that Rodrigues terms are decodable in earlier layers, but only in a shallow network that is sufficiently wide. In a deeper network, Rodrigues terms are decodable only farther down in the layers. Although it may seem that this contradicts our hypothesis that deep and shallow networks learn different solutions, upon further analysis we found quantitative differences in how the Rodrigues terms were represented in the two types of networks. In short, the deep neural network has distinct populations that learn the symmetric and antisymmetric terms in Rodrigues' rotation formula. These populations are non-overlapping with statistical significance, implying a level of specialization. The shallow network on the other hand does not develop these traits. For further details see Appendix C.3 and Section 4.5.

The divide-and-conquer claim in the conclusion has no supporting analysis. The paper does not identify a network computation that divides the input multiplication problem into simpler subproblems and combines their solutions.

We thank the reviewer for pointing this out. We have added a new experiment showing the emergence of specialized populations in deep neural networks encoding separately the symmetric and antisymmetric terms in Rodrigues' rotation formula. Since the populations are largely non-overlapping, this gives evidence that a divide-and-conquer computation is occurring in the network. We hope this allays your concerns.

AuthorsOpenReview ↗

Official comment by authors (part 2)

Response to questions

Question 1

Which family numbers are right? The share of l = 1 and l = 2 neurons differs across the paper: the text and Figure 16 give about 0.32 and 0.57, flat with depth; Figure 5a shows under 0.03 and under 0.35, rising with depth; Figure 4 reports 42 and 899 neurons. The text numbers are close to 3/9 and 5/9, which is what a random matrix would give. Are they actually averages of |A_l|^2/|A|^2 rather than the fraction of neurons classified?

After reviewing the analyses and running further tests, we have determined that the numbers in Figure 5a are correct (a rising trend in the proportion of ℓ=1,ℓ=2\ell=1,\ell=2 neurons in each layer as we go deeper into the network).

Question 2

Do the neuron families survive a null test? Would they still appear if each neurons matrix A were randomly rotated (A -> gAh)? The task treats all such rotations the same, so if the families survive, they may come from the analysis rather than the network. Also, are the neurons preferred axes spread evenly, or bunched along the coordinate axes?

We run additional analyses to confirm that the neuronal populations are not spurious artifacts of the analyses. We use F-regression to find the neurons in each layer which correlate the most with either the KK (antisymmetric Rodrigues term) or the K2K^2 (symmetric Rodrigues term) terms showing up in Rodrigues' rotation formula. These cannot be written as a linear function in the product rotation RR. We find that the populations encoding for KK and K2K^2 overlap, at rates much higher than expected from chance, with the previously identified sets of strong ℓ=1\ell=1 and strong ℓ=2\ell=2 neurons in each layer. This provides further evidence that the populations are legitimate (see sections 4.5 and appendix C.3). We give examples of the distribution of the preferred axes of neurons in each layer where such neurons exist, for both trained and untrained networks, in Appendix section C.5.

Question 3

Does the network compute the axis or angle of XY? Is there evidence for this beyond the network simply representing XY? In SO(4), the individual Rodrigues terms are not linear in R, so decoding them could be a good test. Please report R^2 and compare against decoding from R directly.

The network is able to linearly represent the matrix KK, which has the same entries as the normalized product rotation axis, in its layers. Hence this is evidence that the network is learning the axis in its layers. Note that the axis's entries cannot be represented as linear functions in the entries of the product rotation matrix. For the SO(4)SO(4) case: We run linear regression on the layer activations versus the SO(4)SO(4) Rodrigues terms and compare the resulting R2R^2 with the R2R^2 if we decode from the product matrix directly. We find that for two terms in the formula, decoding from the layer activations yields a higher R2R^2 (See end of section 4).

Question 4

Do deeper networks really generalize better -- they have higher test error? Please report train and test error by depth, and explain in what sense deeper networks generalize better.

We update our previous claim that deep networks generalize better than shallow ones. Upon further analysis, we now believe that deep networks instead learn a quantitatively different solution than shallow networks and we do not make any claim about improvements in generalization. The details about the difference in solution is found in section 4. In a sentence, there emerge distinct populations of neurons in the deep network for encoding the different terms in Rodrigues' formula, but this division-of-labor does not appear in the shallow network.

AuthorsOpenReview ↗

Official comment by authors (part 3)

Question 5

How were the ablations done? Which network and layer were ablated? How were neurons removed, and how were the random control neurons chosen? Does the l = 2 effect hold when controls are matched on activation size? Are the networks learning features? Or do they stay close to initialization? Please show the main measurements at initialization too. The learning rate isn’t scaled with width, so does the width effect in Figure 16a hold with learning rates tuned per width, or with muP?

The experiment was conducted on eighteen neural networks of varying depth and size. Thirteen networks contained 1,024 neurons in each hidden layer. Of these, three were trained independently at each of one, two, three, and four hidden layers, while one contained six hidden layers. Five additional networks contained two hidden layers. Four of these had 256 neurons per layer and were trained using 20,000 and 200,000 examples, with two different activation functions (ReLU & SiLU). The remaining network contained 2,048 neurons per layer. The importance of particular groups of neurons was examined by removing them from the final hidden layer after training. Their responses were set to zero, without otherwise altering the network. The proportion removed was increased gradually from 0% to 100% and the resulting error was measured using 15,000 pairs of rotations that had been withheld from training. For comparison, an equal number of neurons was selected at random from the same layer. These were chosen to have approximately the same distribution of activity variation as the neurons under examination, with the comparison made separately across ten ranges of variation. Five independent selections were made for each proportion removed. Since the random selections could themselves contain neurons from the group under examination, the comparison tended to understate rather than exaggerate the difference between the two groups. The neurons considered l2 proved substantially more important than their randomly selected counterparts. In networks containing a single hidden layer, removing all such neurons increased the angular error by a median of 7.6 degrees more than removing the corresponding random selection. In the deeper networks, the additional error ranged from 17.7 to 22.2 degrees. The difference therefore persisted even after accounting for the extent to which the neurons varied in their activity. A separate examination established that these representations had developed during training. In a network containing six hidden layers, the similarity between the patterns of activity before and after training was measured on a scale from zero to one, with values near one indicating close agreement. From the first hidden layer to the sixth, the similarities were 0.999, 0.95, 0.71, 0.09, 0.05, and 0.05. Thus, while the earliest layers remained comparatively close to their initial condition, the representations in the later layers underwent considerable change. The changes in the weights of the second through sixth layers were likewise substantial, amounting to between 40 and 49 percent of their original magnitudes. Before training, the responses of neurons in the final hidden layer bore almost no systematic relation to the product of the two input rotations. The median proportion of variation explained by this relationship lay between 0.009 and 0.021, irrespective of depth. No neurons satisfied the criteria for either the l1 or l2. The average angular error was approximately 2.2 radians, corresponding to chance-level performance. After training, the relationship between neuronal activity and the product rotation became much stronger, particularly in the deeper networks. The proportion of variation explained rose as high as 0.988. As many as 36 percent were characterized as l2. At the same time, the angular error fell to between 0.014 and 0.018 radians. These results indicate that the rotational representations identified in the trained networks arose during learning and that the neurons associated with the l2 representation made a measurable contribution to the accuracy of the computation.

Reviewer_EWpVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper trains MLPs to multiply SO(3) rotations. It finds that deeper networks have hidden neurons well described as linear functionals ⟨A, XY⟩ of the product. These neurons are split into ℓ=1 (sin θ) and ℓ=2 (1−cos θ) families matching Rodrigues' formula. Population probes decode the Rodrigues terms, restriction to cyclic subgroups recovers Fourier structure, and SO(4) shows an analogous bivector structure. The authors conclude that depth lets networks learn Rodrigues' formula and support a divide-and-conquer hypothesis for continuous groups.

Critical strengths
  1. Strong theoretical foundation and well-motivated
  2. The analysis is thorough and convincing.
  3. Clear contrast between shallow and deep networks. The finding that shallow networks encode factors while deep ones encode the product is a concrete, testable result.
Critical weaknesses
  1. Background definitions (for example Axial neurons and L=1 vs. L=2) are unclear and difficult to read.
  2. R is itself a combination of the Rodrigues terms, any layer that linearly encodes R will show ℓ=1/ℓ=2 neurons and decode T₁ and T₂. The close correspondence of the independent fit described by equation 10 is expected since the neurons are specifically fit to linearly encode R.
  3. The results convincingly show that the neuron features linearly encode R but do not fully show that Rodrigue’s formula is specifically being used.
Questions for authors

Can you add a control for a different function linear in R to test for the Rodrigues terms?

1 comment
AuthorsOpenReview ↗

Official Comment By Authors

Thank you for your review. All reviewers shared your primary concerns: (Weakness 2. and 3., as well as question 1.) Thus, we have uploaded a global rebuttal to summarize how our new experiments address these concerns.

Weakness 1.

Background definitions (for example Axial neurons and L=1 vs. L=2) are unclear and difficult to read.

We apologize for this oversight and we have rewritten the mathematical background and methods sections to improve clarity.

Weakness 2.

R is itself a combination of the Rodrigues terms, any layer that linearly encodes R will show ℓ=1/ℓ =2 neurons and decode T₁$ and T₂. The close correspondence of the independent fit described by equation 10 is expected since the neurons are specifically fit to linearly encode R.

You are correct; To control for this, we've added new linear regression experiments for the matrix terms KK and K2K^2 which are respectively components of the antisymmetric and symmetric Rodrigues’ terms (see response to question 1).

Recall Rodrigues’ Rotation Formula. Write K=[u].

$ R = I + \sin K + (1 - \cos\theta) K^2 $

Where

\begin{pmatrix} 0 & -u_3 & u_2 \\\\ u_3 & 0 & -u_1 \\\\ -u_2 & u_1 & 0 \\\\ \end{pmatrix}$$ Each Rodrigues’ term is linear in the product matrix R, and thus linear regression cannot distinguish between the network learning $R$ and the network learning the individual terms (as reviewers pointed out). However, upon removing the scalar in front of these matrix terms, the two matrix terms are non-linear in R and thus linear regression can only score highly on them if the two terms are explictly represented linearly in the layer activations (if only the sum R is present, linear regression will not score well on $K$ or $K^2$). For each multiplication sample $((X,Y),XY),X,YSO(3)$, we construct the following quantities: - $S_(X,Y)$, the activation of a collection of neurons S on this sample. We will make heavy use of $L_{i,(X,Y)}$ where $L_i$ consists of all the neurons in layer i. R=XY, the full product matrix - The antisymmetric generator $K = [u]_\times$ - The symmetric generator $K^2 = [u]_\times^2$ We run linear probing experiments on the layer activations on these quantities. Specifically, we run the following: - Baseline experiment: We run linear regression on R versus K and K2 to determine how much of K and K2 can be linearly decoded from R right away. - Regression on activativations: We run linear regression on $L_{i, (X,Y)}$ versus $K$ and $K^2$ to see if the layer activations encode information about these component matrices beyond what encoding only $R = XY$ provides (see first experiment) The plots for these experiments are found in the paper in Figures 18 and 19, appendix C. Additionally, we isolate the neurons which contribute the most to the regression for these two terms, and find that they are mostly non-overlapping neural populations. Furthermore, when running linear regression restricted to the activations of these populations, we find that the population that learns $K$ has a much higher $R^2$ on the antisymmetric Rodrigues term than on the symmetric Rodrigues term, and vice-versa for the population that learns $K^2$. This shows that distinct populations in each layer encode the different terms in the Rodrigues rotation formula. Hence the network is not learning the product rotation and retrieving the Rodrigues terms from that, but as claimed in the main paper (but now supported much more strongly), there are disjoint neurons, where on set learns the antisymetric Rodrigues term, and the other population learns the symmetric Rodrigues term. Naturally, when these two sets of neurons add their computations together, they form the product matrix R that the submitted paper showed was learned in DNNs. Thus the computation of the efficient algorithm for a rotation, which is the Rodrigues Rotation formula is learned in deep networks.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global rebuttal

We thank all reviewers for their highly informative reviews. All reviewers all have the same primary concern: we showed that networks are learning the product of the Rodrigues’ rotation formula, but we did not show that the individual terms are learned by disjoint populations of neurons. Thus, we didn’t show that the terms were learned, or that an implementation of the algorithm computing the Rodrigues’ rotation formula was learned. All reviewers pointed out the same reason we fell short: the terms of Rodrigues’ Rotation formula are linear in the product matrix R, and thus linear regression should always score highly on both terms, even if the two terms weren’t learned individually.

We’ve bridged this gap in our claims by adding new experiments, where we utilize components in the terms of Rodrigues’ rotation formula that are not linear in the product matrix R. Thus, the network can only score highly on these components in the terms if they are disjointly present in the activations. Additionally, we take the population of the top 25% of neurons that contribute to each term and test whether they’re disjoint neurons. We find that deep networks learn the component in each term with largely non-overlapping populations with high statistical significance (low p-value), but shallow networks do not. This evidence supports the main claim in the paper that the Rodrigues’ rotation formula has been implemented by DNNs and that the network contains two populations (one for each term) where the neurons are learning the antisymmetric term or the symmetric term. We go farther however, and take the top 16 neurons in each population and fit a linear regression onto each term of Rodrigues’ rotation formula and find that the neurons encode the corresponding term in Rodrigues' much more strongly than the other term (Figure 19). We believe this evidence greatly strengthens the paper and thank all reviewers for pointing out that unless we added additional arguments, we were overclaiming.

To be short and clear: we now show networks, both deep and shallow, learn the Rodrigues’ product matrix R (the answer to the SO3 multiplication), but only deep networks learn the terms to compute it by specializing neurons to each term.

Additionally, reviewers asked about whether deep networks were generalizing better, and we didn’t intend for this to be how the paper read. Our intention was just to state that deep networks cleanly learn the algorithm, while shallow networks learn something else. Thus, we have amended the paper to ensure it doesn’t read like deep networks “achieve better generalization accuracy”.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.