Invariant Graph Properties for Neural Computation Graphs

Representing neural networks as graphs allows their architecture and learned structure to be studied using tools from network science, topological data analysis, discrete geometry, and graph representation learning. Yet the graph associated with a trained network is not uniquely determined by the function it computes: exact parameter symmetries can alter edge weights and graph structure without changing the network’s predictions. We study invariance under common symmetries, including hidden permutations, positive rescaling, and general linear changes of basis. For positive rescaling, we identify minimal coordinates and a canonical representative on each fixed support, and show why zero weights preclude a single continuous exact representation across supports. For general linear symmetries, we derive operator representations that remain complete when ranks drop, including an operator graph for multi-head attention. We then connect this symmetry dependence to the reliability of predictions and interventions. Experiments on collections of MLP and ViT checkpoints show that raw graph analyses can differ substantially across equivalent parameterizations, whereas symmetry-aware methods remain stable and preserve predictions and transported interventions. The appropriate graph representation therefore depends on both the parameter symmetries and the target of the analysis.