Intrinsic Information Theoretic Analysis of ReLU Nets

Mutual information (MI) is fundamental to representation-learning analysis in deep networks, but in high dimensions MI estimators depend on hyperparameters — bin widths, kernel bandwidths, neighbourhood radii, auxiliary networks — whose bias often dominates the signal. We show that continuous piecewise-linear (CPWL) networks, including ReLU networks, admit an intrinsic information-theoretic analysis that avoids this pitfall: their inherent partition of the input space into convex linear regions assigns to every input a discrete region label $\Pi$ alongside the continuous representation $T$. We decompose $I(Y;T)$ exactly into geometric terms, where the routing information $I(Y;\Pi )$ admits a hyperparameter-free plug-in estimator that approximates a population lower bound on every other MI term in the decomposition. When the partition saturates, a functional-equivalence quotient at relative Frobenius tolerance $\epsilon \in [0,2]$ collapses regions implementing the same linear operator and restores informativeness. Empirically, the routing estimator correlates strongly with three established MI baselines \it without any hyperparameter, and the functional quotient at moderate $\epsilon$ recovers informative estimates in the regime where the raw partition is bounded by ${\log }_{2}N$ rather than $H(Y)$. We provide a parameter-free framework for analysing the internal information geometry of CPWL networks, grounded in their inherent piecewise-linear structure rather than in external discretisation.