The Dynamics of Routing in Deep Linear Networks

Over the course of learning, computation in biological and artificial neural networks is distributed across the underlying network in complex ways, reflecting the interaction of network architecture and the dynamics of learning, but we have limited theoretical understanding of this process. Here, we develop a mathematical formalism to study the routing dynamics exhibited by gradient descent learning in linear networks defined over a directed acyclic graph. We show that the conserved quantities satisfied by gradient descent allow learning to be decomposed into an overall capacity variable $Q$ and a routing variable $s$ living in a finite polytope, measuring the relative distribution of weights across edges in the network. In balanced networks of uniform depth, in the positive scalar setting, we show that learning separates cleanly into scalar growth and an autonomous dynamical flow on the routing polytope, which is controlled by the distribution of computation across paths in the network. We prove that the stable fixed points of these projective dynamics correspond to chain-equivalent subgraphs, allowing us to predict when a network will collapse to a smaller subgraph in the large-target limit; we also show evidence for this effect in non-linear DAG networks. When depth is mixed or conserved charges are non-zero, the geometry yields transient depth races and complex routing movement across the graph, before ultimately converging to the previous homogeneous case in the large-target limit. We also show that the local dynamics at interior fixed points are governed by a kind of path-memory, while the local behavior at boundary fixed points allows for spontaneous circuit formation. Finally, we show that richness in linear DAG networks is governed by an electrical formalism. We demonstrate that several of these phenomena generalize in some form beyond our exact setting, to matrix and non-linear networks. Our results provide an analytical bridge between structural graph topology, conserved quantities, and emergent routing behaviors in the deep linear setting, which we hope can eventually help support further understanding of complex dynamics in more general networks.