The form of the weights in deep networks trained on modular addition

Our work presents the form of the weights between neurons (and thus between layers) that deep neural networks (DNNs) learn on the popular task of modular addition. This task has been studied by many prior works, but only the activation geometry, which are the manifolds corresponding to the learned neural representations, are understood in the case of depth. The weights that are learned are only understood in 1-hidden layer networks and thus, this work serves to extend our understanding of the weights to multi-layer networks. To our knowledge, this is the first work to describe and validate the form that learned weights in deep networks take between layers across random seeds on a group multiplication task.