- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This is a very beautiful paper with deep mathematics and corresponding empirical validation discussing how over time SGD successively encourages neural networks to become simpler and simpler by merging blocks of identical neurons (identical incoming and outgoing weights) into larger blocks of identical neurons. They identify order parameters (such as variance across runs) and when and why this variance will spike due to merger of blocks and their block size. They connect this to the important idea of delayed generalization and grokking because of the time it takes to merge and simplify the network. They perform empirical validations on both toy network and transformers.
Critical strengths
Beautiful theory about the timing and implications of mergers of blocks of similar neurons. Wonderful and relevant empirical validation of the results (e.g. the empirical observed variance peaks in Fig. 11 and earlier figures are very nice).
Critical weaknesses
They use the phrase topological phase transitions. I am not sure why - what is topological about the transition - it is just more neurons becoming similar to each other (i.e. projecting onto a subspace in weight space). Would be nice to clarify this language.
Questions for authors
Just the question about the use of the word topological in this paper.
2 comments
Thank you for the clarification.
Our response
We thank the reviewer for the generous assessment and the question on terminology.
The term "topological". We use the term for changes in how the subnetworks of a network connect to each other as training moves through parameter space. This motion matters because trained weights are known to keep moving after the training loss has stabilized and to approach an approximate invariant measure instead of a stationary point (Zhang et al., 2022). Which solution is learnt then depends on the path that training takes. Studies of mechanistic interpretability (Nanda et al., 2023) and of low-rank biases (Huh et al., 2023) lead us to postulate that this path consists of discrete events. In our formalism such an event is a symmetric subnetwork collapsing into the set of parameters in which its interchangeable units coincide. Because swapping these units leaves the loss unchanged, gradient updates keep this set invariant while stochastic attractivity (Chen et al., 2023) draws training into it. In the parameters a collapse thus appears as neurons becoming identical, as Chen et al. (2023) observed empirically during training.
The topology enters once we record which subnetworks have collapsed together. We treat each subnetwork as a node and join two nodes by an edge while their parameters lie closer than the link distance of Definition 3.3. A collapse then adds an edge to this network of subnetworks and its reversal removes it. Because an edge forms exactly when the neighborhoods of half the link distance around two nodes overlap in parameter space, the components of the network match the connected parts of the region these neighborhoods cover. This is the construction by which topological data analysis turns point clouds into complexes (Edelsbrunner & Harer, 2008). Adding or removing an edge therefore changes how this region of parameter space is connected, even though the parameters themselves move continuously. Following these changes through training gives the Reeb graph of Section 3. Its branches join at condensations and split at fragmentations, the two topological transitions of Theorem 3.4.
During phases in which training collapses neurons toward a simpler solution, Proposition 4.4 reduces the Reeb graph to a percolation process. In percolation the components of a graph grow as edges are added to it. When each new edge joins two randomly chosen nodes (Erdős–Rényi percolation), the largest component grows continuously with the number of edges. Deep networks add their edges along nested symmetry instead, since units can be permuted within a block and whole blocks can be permuted with each other as the two subtrees of a binary tree can. Two blocks whose units have each become identical can then become identical themselves in a single event that joins all their units at once. The largest component therefore grows in discrete jumps (Proposition 4.5). The revision will state this construction in Section 3 and use the term consistently.
References