When Are Energy Proxies Reliable? Excitation and Curvature in ReLU Equilibria
Abstract
An energy proxy replaces asymmetric recurrent interactions with a symmetric field, but a smaller field mismatch need not imply a more accurate answer. We construct a three-dimensional ReLU family whose original updates are uniformly contractive, yet the metric that minimizes rotational defect produces unbounded proxy error. This metric is fixed, has condition number below , and performs worse than the bounded-error identity proxy. For fixed input distributions on a positive box, the energy penalty grows as and the mean squared state error grows as , where is the proxy curvature margin. We explain these rates with two-sided population bounds for strongly convex proxies, valid across activation changes and at any rotation strength. Inactive-unit slack can screen rotation even when the response second moment is nonsingular, so the lower bound measures excitation through jointly active responses. Complementarity supplies an energy-budget upper bound, and both bounds recover the growth rates on the separating family. Input-coverage and response-balance conditions specify when the lower bound is positive and quantitatively informative. Experiments on trained ReLU networks and a fixed-anchor recurrent Transformer support the directional mechanism: input-excited rotation predicts proxy errors better than rotation magnitude.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.