- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the internal representations and computational mechanisms that emerge when deep neural networks learn multiplication in the rotation group SO(n). The authors train multilayer perceptrons (MLPs) to perform rotation matrix multiplication and analyze their internal representations using neuron activation analysis, linear probing, representation space decomposition, neuron ablation, and Fourier analysis. The results show that, as network depth increases, hidden representations become more linearly related to the target rotation product matrix and exhibit mathematical structures corresponding to the Rodrigues rotation formula. The authors further identify groups of neurons that encode different components of the formula and observe similar patterns in higher-dimensional rotation groups. Based on these findings, they argue that deep networks learn to implement the Rodrigues algorithm for rotation multiplication and propose this as a mechanism underlying their generalization ability.
The observed depth-dependent patterns in internal representations provide valuable insights into how neural networks encode mathematical operations. However, the current results primarily establish mathematical correspondences between internal representations and the Rodrigues formula, rather than demonstrating that the networks actually use this algorithm to perform rotation multiplication. More importantly, the paper does not explain why such mathematical structures emerge during training or why they become more pronounced with increasing network depth. Although the neuron ablation experiments provide some functional evidence, they do not sufficiently reveal the underlying computational process or establish a clear causal relationship between the proposed mechanism and generalization performance. Overall, while the empirical findings are interesting, the central claims regarding the emergence of the Rodrigues algorithm and its role in generalization go beyond what the current evidence can support.
Critical strengths
-
Understanding how neural networks learn mathematical operations and develop internal computational mechanisms is an important research problem. Extending previous studies, which have primarily focused on finite groups, to continuous Lie groups also introduces a degree of novelty.
-
The authors conduct a systematic experimental analysis. The study examines internal representations at multiple levels, from individual neurons and groups of neurons to entire hidden layers. The authors also evaluate the robustness of their findings across different network architectures, training settings, and ablation experiments. These results provide useful insights into how network depth influences the organization and representation of information within neural networks.
Critical weaknesses
-
The work remains largely descriptive and lacks a deeper understanding of how the internal representations emerge. Through systematic experimental analyses, the authors identify mathematical correspondences between neuron responses and the Rodrigues formula. However, they do not explain why such representations emerge during training, or why they become more pronounced as network depth increases. Consequently, the study mainly provides empirical observations and mathematical interpretations of internal representations, without sufficiently revealing the mechanisms underlying their emergence or the specific role of network depth.
-
The paper does not sufficiently demonstrate that the networks actually learn and use the Rodrigues algorithm. The authors first observe that the responses of neurons in deeper networks can be described by linear functions of the rotation product matrix R=XY, and then interpret these responses using the Rodrigues formula. However, the Rodrigues formula is itself a mathematical identity satisfied by rotation matrices. In particular, the two Rodrigues components analyzed by the authors can be expressed directly as linear or affine functions of R. Therefore, once a network has developed a linear representation of the rotation product, these mathematical structures may naturally emerge without the network actually implementing the Rodrigues algorithm. The current analysis does not adequately distinguish between these possibilities and thus does not sufficiently support the central claim that the networks have learned and implemented the Rodrigues algorithm.
-
The relationship between Rodrigues-like representations and generalization remains unclear. Figure 3 shows that although deeper networks develop more pronounced representations of the rotation product, their mean rotation prediction error increases from 0.0175 rad for shallow networks to 0.0246 rad for deeper networks. The authors do not adequately explain this observation. Furthermore, shallow networks also contain many neurons classified as Rodrigues-like, with proportions comparable to those in deeper networks. Their hidden layers can also recover different components of the Rodrigues formula reasonably well and exhibit corresponding Fourier structures. These findings suggest that Rodrigues-like representations are not unique to deeper networks, and their relationship with predictive performance remains unclear. Therefore, the current evidence does not sufficiently establish that such internal representations are a key mechanism underlying network generalization.
Questions for authors
-
Can the authors explain why Rodrigues-like representations emerge spontaneously during training? The correspondence between hidden representations and the Rodrigues formula becomes more pronounced as network depth increases. Can the authors provide an explanation for this phenomenon in terms of network architecture, optimization dynamics, or representation learning?
-
How can the authors demonstrate that the networks actually implement the Rodrigues algorithm? The two Rodrigues components analyzed in the paper can be expressed directly as linear or affine functions of the rotation product matrix R=XY. Therefore, a network that encodes R may naturally exhibit these mathematical structures without actually implementing the Rodrigues algorithm. Can the authors provide further evidence to distinguish between these two possibilities?
-
Why do shallow networks also exhibit similar Rodrigues-like structures? The experiments show that shallow networks contain many Rodrigues-like neurons and can recover different components of the Rodrigues formula and the corresponding Fourier structures within their hidden layers. What are the fundamental differences between the internal computational mechanisms of shallow and deep networks? Can these differences be quantitatively demonstrated?
-
What is the relationship between Rodrigues-like representations and generalization performance? Figure 3 shows that although deeper networks exhibit more pronounced representations of the rotation product, their mean rotation prediction error increases from 0.0175 rad to 0.0246 rad. How do the authors explain this result? In particular, why do shallow networks achieve lower prediction errors despite exhibiting less pronounced Rodrigues-like representations?
2 comments
Response to questions:
Can the authors explain why Rodrigues-like representations emerge spontaneously during training? The correspondence between hidden representations and the Rodrigues formula becomes more pronounced as network depth increases. Can the authors provide an explanation for this phenomenon in terms of network architecture, optimization dynamics, or representation learning?
Thank you for your question. We did not relate training dynamics to the representations learned as it was beyond the scope of the paper. The correspondence between hidden representations and the Rodrigues formula becomes more pronounced as network depth increases: Intuitively, each layer incrementally refines the neurons to either be strong-l1 or strong-l1, but not both. This is seen in Figure 5 where the percentages of l=1 and l=2 neurons increase with depth.
How can the authors demonstrate that the networks actually implement the Rodrigues algorithm? The two Rodrigues components analyzed in the paper can be expressed directly as linear or affine functions of the rotation product matrix R=XY. Therefore, a network that encodes R may naturally exhibit these mathematical structures without actually implementing the Rodrigues algorithm. Can the authors provide further evidence to distinguish between these two possibilities?
See answer to weakness 2 of reviewer EWpV.
Why do shallow networks also exhibit similar Rodrigues-like structures? The experiments show that shallow networks contain many Rodrigues-like neurons and can recover different components of the Rodrigues formula and the corresponding Fourier structures within their hidden layers. What are the fundamental differences between the internal computational mechanisms of shallow and deep networks? Can these differences be quantitatively demonstrated?
We observe that wide-enough shallow networks have layer activations that linearly encode Rodrigues-like structures such as and (see Figure 10) Despite these similarities, there are quantifiable differences in what the shallow and deep networks learn. We find that the overlap in the neuronal populations encoding for the and terms is statistically significant in the deep network (especially in the later layers), but not in the shallow network. This shows that in the deep network, the late-layer layers experience encoding specialization, where different populations of neurons are responsible for different terms in Rodrigues' formula (see Figures 11, 12)
What is the relationship between Rodrigues-like representations and generalization performance? Figure 3 shows that although deeper networks exhibit more pronounced representations of the rotation product, their mean rotation prediction error increases from 0.0175 rad to 0.0246 rad. How do the authors explain this result? In particular, why do shallow networks achieve lower prediction errors despite exhibiting less pronounced Rodrigues-like representations?
See answer to Question 4 of reviewer fdDo.
Official comment by authors
Thank you for your long and detailed review.
Response to weaknesses
You are correct. This work was meant as an exploratory empirical work to lay the foundations for further deeper mechanistic research. We plan to address deeper mathematical questions about how the representations emerge and relate it to learning dynamics in a future work.
You are correct; To control for this, we have added new linear regression experiments for the terms K and K2 which are components of Rodrigues (see response to question 1). We furthermore isolate the neurons which contribute the most to the regression for these two terms, and find that they are mostly non-overlapping neural populations. Furthermore, when running linear regression restricted to the activations of these populations, we find that the population that learns K has a much higher R2 on the antisymmetric Rodrigues term than on the symmetric Rodrigues term, and vice-versa for the population that learns K2. This shows that distinct populations in each layer encode the different terms in the Rodrigues rotation formula. Hence the network is not solely learning the product rotation then retrieving the Rodrigues terms from that.
We update our previous claim that deep networks generalize better than shallow ones. Upon further analysis, we now believe that deep networks instead learn a quantitatively different solution than shallow networks and we do not make any claim about improvements in generalization. The details about the difference in solution is found in section 4. In a sentence, there emerge distinct populations of neurons in the deep network for encoding the different terms in Rodrigues' formula, but this division-of-labor does not appear in the shallow network.