acceptodds
Under review as a conference paper at ICLR 2027

When Are Energy Proxies Reliable? Excitation and Curvature in ReLU Equilibria

Abstract

An energy proxy replaces asymmetric recurrent interactions with a symmetric field, but a smaller field mismatch need not imply a more accurate answer. We construct a three-dimensional ReLU family whose original updates are uniformly contractive, yet the metric that minimizes rotational defect produces unbounded proxy error. This metric is fixed, has condition number below , and performs worse than the bounded-error identity proxy. For fixed input distributions on a positive box, the energy penalty grows as and the mean squared state error grows as , where is the proxy curvature margin. We explain these rates with two-sided population bounds for strongly convex proxies, valid across activation changes and at any rotation strength. Inactive-unit slack can screen rotation even when the response second moment is nonsingular, so the lower bound measures excitation through jointly active responses. Complementarity supplies an energy-budget upper bound, and both bounds recover the growth rates on the separating family. Input-coverage and response-balance conditions specify when the lower bound is positive and quantitatively informative. Experiments on trained ReLU networks and a fixed-anchor recurrent Transformer support the directional mechanism: input-excited rotation predicts proxy errors better than rotation magnitude.

Primary area:
optimization
Keywords:
equilibrium models, recurrent neural networks, matrix symmetrization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wMuhOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies energy proxies in three-dimensional ReLU networks with uniformly contractive original updates. The central finding of the paper is that the least rotational defect can select an arbitrarily inaccurate energy proxy, even when the original recurrence is uniformly contractive. The experiments provide a comparison of input-excited rotation ranks proxy errors and rotational defect, using ReLU classifiers and a recurrent transformer.

Critical strengths

The paper presents an analysis framework for energy proxies for a specific family of ReLU networks. In the defined scope, it contributes with a failure theorem for energy proxy selection, input and curvature dependent reliability conditions, and controlled and trained network evidence. The paper also presents strong mathematical theory and derivations supporting the claims.

Critical weaknesses
  1. Presentation and clarity could be substantially improved. In general, the paper is difficult to follow. The abstract appears confusing because the sentences are not sufficiently coherent, and the terms are not clearly defined. The paper would benefit greatly from overview illustration.

  2. The paper is not sufficiently self-contained. The introduction should do a better job introducing central terms, such as energy proxies, asymmetric interactions, rotational defects, contractive updates, and equilibria.

  3. The contributions of the paper, albeit valid in the defined scope, appear quite niche, and the authors provide little insight regarding their utility in broader context.

  4. The mathematical formulations appear interesting but are difficult to follow as the paper does not provide sufficient context as to their motivation or method.

Questions for authors
  1. Could the authors provide clear definitions of energy proxies, proxy errors, field approximation, answer fidelity, and rotational defect?

  2. What is “a three-dimensional ReLU family” and how is it defined?

  3. How do the proposed analysis framework and results generalize to dimensions greater than 3?

Reviewer_6MivOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper asks when a symmetric “energy proxy” reproduces the equilibrium of an asymmetric ReLU recurrent network. Its main result is a counterexample: choosing coordinates that minimize the discarded rotational part can give arbitrarily large proxy error, under favorable assumptions, while the identity proxy stays accurate. To explain this, the authors derive upper and lower error bounds showing that what matters is rotation excited by the inputs on jointly active units, combined with the proxy’s curvature, rather than the size of the rotation. Experiments on trained ReLU networks and a recurrent Transformer support this conclusion.

Critical strengths

The theoretical analysis is rigorous and carefully executed, with explicit constants, sharpness examples, and bounds that remain valid across activation patterns. The separating construction is a clean, fully analyzable example that isolates a single failure mechanism.

Critical weaknesses

The motivation given by authors is not convincing: the paper never shows that anyone builds symmetric proxies of trained asymmetric networks, or that proxy error has downstream consequences, so the separation result addresses a problem with no demonstrated use cases. The presentation is also hard to follow, using in asbtract notions like “symmetric fields,” “field mismatch,” and “rotational defect” before explaining why energy proxies are of interest. Overall I beleive the main weakness of this paper is that I came away unconvinced that it was addressing an issue of significant relevance to ML practice.

Questions for authors

Why would one use a proxy at all, given that the true equilibrium is a problem that can be solved directly at comparable cost?

Reviewer_haXUOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper establishes certain results in the domain of recurrent neural networks, specifically whether the resulting fixed point of a recurrent computation can or cannot be approximated by minimizing a surrogate loss. Various conditions for this are investigated and supported by experimental verification.

Critical strengths

The paper provides a) a failure theorem for energy-proxy selection in the context of recurrent Relu-networks, and b) error bounds for special configurations of such networks.

Critical weaknesses

The paper contains no justification for this research at all, nor an explanation of the most basic terms from the abstract , e.g., "energy proxy". All terms from the abstract should be introduced in the introduction section as early as possible to maintain readability. A justification for the interest in eqn. (1) is not provided either, i.e.: in what prominent neural network architectures does this recurrent computation occur? What benefits can be expected from these results? Concerning the experiments: it is not clear what they are supposed to show, partly because the architectures are not explained: the citation is apparently just for the dataset but does not describe or justify the networks used here. How can such recurrent architectures perform classification, and why can we expect them to work? What are "determinant-one-metrics" and why are they used?

Questions for authors

As stated above:

  • what is the wider interest, for the community, of eqn. (1)? In what relevant application does it occur?
  • do any practical advantages follow from your results? Where could they be beneficially applied?
  • What is the justification of the architectures chosen for experiments? Why are they relevant?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.