acceptodds
Under review as a conference paper at ICLR 2027

Beyond Estimator Moments: Minimal Information for Predicting Zeroth-Order Adam's Response

Abstract

Zeroth-order (ZO) query laws are conventionally compared through estimator mean, covariance, and mean-squared error. We show that these statistics need not determine Adam's response: in a local separable exponential-loss model with an annealed stabilizer, equal-cost laws can match the zero-radius mean, full covariance, and MSE yet induce distinct mean normalized updates at their respective slow equilibria. The mechanism is root adjustment: equilibrium motion absorbs part of the direct query-law effect, leaving a generally oblique-projected response. For a sphere-supported class with prescribed data-contracted moments, we characterize exactly how much additional information is required for uniform next-order prediction. The minimum number of exact scalar expectation readings equals the rank of the surviving response operator and, under nondegeneracy and full realizability, is , where is the parameter dimension and the active-row rank. Indistinguishable admissible laws prove necessity, while explicit polynomial readings attain the bound. On a matched angular-mixture family, a response-aligned reading estimated from only 16 i.i.d. directions reduces held-out prediction MSE to about 1.4% of the reference-only baseline across two disjoint phases, whereas a same-count orthogonal reading misses the response. Finite-radius bounds and a local initialized weak-limit result further delimit the regime in which the response characterization applies.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.