Gradient Estimation View of Zeroth-Order Optimizers

Zeroth-order (derivative-free) methods—including evolution strategies, finite differences, and sphere-sampling estimators—are often viewed as approximating gradients of smoothed objectives. But is this a universal property of ZO methods, or do only specific algorithms admit such a view? We answer this with an update-first reconstruction principle: given a sampler $q_{\theta }$ and update weights $\psi$, we reconstruct an implicit smoothing measure $p_{\theta }$ such that the expected update equals the derivative of $J(\theta )=\mathbb{E}\_x\sim p_{\theta }[f(x)]$. The reconstruction equation ${\partial }_{{\theta }_j}{p}_{\theta }=q_{\theta }{\psi }_{{\theta }_j}$ cleanly separates where the algorithm samples ($q_{\theta }$) from which objective it is smoothing ($p_{\theta }$). It covers classical likelihood-ratio (REINFORCE) estimators, finite-difference and sphere rules whose weak derivatives live on boundaries, and previously-unexplained schemes such as random-denominator differences—for which we give the first explicit smoothed objective. Worked examples (Gaussian ES, forward and central FD, sphere sampling, randomized Kiefer–Wolfowitz, and an Epanechnikov-kernel uniform ES) yield closed-form kernels that make each method's induced smoothing visible, comparable, and interpretable.