Tropical Boundary Complexity of Deep ReLU Networks

We derive a recursion for the expected squared Frobenius norm of the zonotope generator matrices that govern the decision boundary of a random Gaussian deep ReLU binary classifier. For a network of depth $L$ with weight variances $({\sigma }_{l=\mathrm{1..}L}^2)$, the boundary complexity is ${\Gamma }_L=8(1+1/\pi ){\prod }_{l=1}^L{\sigma }_l^2/2$. He initialization ${\sigma }_w^2=2$ is the unique uniform weight variance under which ${\Gamma }_L$ is preserved with depth, matching the Poole–Schoenholz mean-field critical value on a different observable. ${\Gamma }_L$ is identified with an observable per-unit-length boundary-crossing density via Rice's formula. The relation is supported by Monte Carlo experiments.