acceptodds
Under review as a conference paper at ICLR 2027

Shrinkage in Zeroth-Order Optimization: Estimation Risk and Expected Descent

Abstract

Reducing the estimation error of a zeroth-order (ZO) update need not improve optimization after its step size is chosen. We study this distinction for anchored ridge shrinkage under exact finite batches of random directions. Optimally rescaled Euclidean risk and optimized isotropic quadratic descent rank updates identically. For iid sphere updates, an explicit Stein kernel gives risk bounds at a specified nonzero shrinkage strength. With the direction count fixed at and dimension sufficiently large, the same zero-anchor rule has certified risk reduction and strictly worse optimized descent. We quantify this relative loss and give explicit finite-dimensional enclosures. The nonlinear coefficient remains random in the limit, making effective steps vary across updates. In a specified curvature family with bounded anchors, the rule can beat Raw while remaining asymptotically inferior to the fixed mixture . Independent simulations test these predictions. At frozen OPT-125M checkpoints, expanded step searches yield interior fitting-loss minima in all forty selections, yet the nonlinear rule has no consistent advantage. Independent profiling predicts some coefficient effects at small steps, with reduced predictive agreement at the full selected step. A local audit obtains informative risk and optimization bounds but no final endpoint confirmation in six budget–curvature comparisons. The results separate estimation risk, expected descent after choosing a step size, and statistical confirmation.

Primary area:
optimization
Keywords:
zeroth-order optimization, James-Stein

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SyyjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper examines whether shrinkage-induced reductions in gradient estimation error lead to better zeroth-order optimization. It derives an explicit Stein risk bound for finite batches of sphere-based ZO estimates and shows that a specified nonlinear shrinkage rule can reduce MSE while worsening expected descent after separate step-size optimization. It further studies anchor and curvature effects, supported by synthetic simulations, frozen OPT-125M checkpoint experiments, and a finite-sample confirmation procedure.

Critical strengths
  1. The explicit sphere-law analysis establishes a technically substantive separation between estimation MSE and optimized expected descent for the same shrinkage rule and strength.

  2. The use of fixed rescaling, shared-anchor controls, separate step-size selection, and independent evaluation helps distinguish the effects of scaling, anchor information, and nonlinear mixing.

Critical weaknesses
  1. The main argument is fragmented. Sections 3–5 distribute the formulation, shrinkage rule, and theoretical results across several sections without sufficiently explaining their connections. The motivation and implications of the results deserve more attention, particularly the distinction between established identities and the central new contribution. A clearer separation between the methodological setup and theoretical analysis would substantially improve readability.

  2. The experimental presentation is insufficiently self-contained. Understanding Sections 6–7 requires repeatedly consulting Appendices H and I for implementation details and distinctions between experimental stages. A unified experiments section should consolidate the final protocol and clearly explain which theoretical claim each experiment tests, where the assumptions differ, and what the results establish.

  3. Operational records obscure the scientific narrative. Appendix I.2 includes runner and vendor filenames, configuration-recovery history, reconstructed records, and missing execution logs. Much of this material belongs in a reproducibility README or accompanying code documentation. Its prominence, together with repeated descriptions of earlier and later experiments, makes the manuscript read like a record of the research process rather than a sufficiently synthesized research argument.

Questions for authors
  1. What is the principal new insight beyond the established distinction between estimation MSE and step-size-optimized descent? Please clarify how the main theoretical results jointly establish this contribution and distinguish it from supporting identities and extensions.

  2. Could the authors consolidate the final experimental protocol and explicitly map the reported results to the theoretical claims? In particular, what conclusions about practical shrinkage are supported by the network experiments, given the limited confirmation at the selected full step?

Reviewer_ruhYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is concerned with estimation methods for zero order optimization.

Critical strengths

None

Critical weaknesses

l011 "need not improve optimization" : this is not a rigorous statement
l012 "this distinction" : distinction from what? l012 "Optimally rescaled Euclidean risk and optimized isotropic quadratic descent rank updates identically." random statement without link with previous sentences

l013 - l028 this goes on, there are no logical link between sentences and random notions are introduced without explanation

l045 "Shrinkage" : what does it mean here, why is it introduced ? the rest of the paragraph is incomprehensible

l053 what is a "direction" . "

l053 "the same shrinkage rule yields lower expected descent than Raw" : what is "Raw" ?

l057 "coefficient" what coefficient ?

l063 What is a "Stein Risk" ?

l128 "Euclidean space of dimension r" : standard is to say R^d (using d in place of r)

l130 F_h is not introduced

l132 "A two-sided difference is exactly g^T u, independently of H_h" . This does not mean anything. What is H_h

l133 "The estimators" : of what ?

Questions for authors

The authors should define the mathematical objects they use in thir paper so that the reader may understand their point.

Reviewer_CWUFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper argues that lower estimation risk does not imply better optimization for zeroth-order (ZO) shrinkage updates. In an exact quadratic model with iid sphere directions, the authors derive a Stein risk bound and prove that, for fixed m>=3 and sufficiently large dimension, a nonlinear shrinkage rule can reduce risk while worsening expected descent even after separate step optimization. They analyze anchor information and transverse curvature via four moments, compare fixed mixtures, and validate with simulations, OPT-125M checkpoint searches, and a finite-sample local audit. Simulations support the limits; network evidence is mixed, and the audit fails endpoint confirmation.

Critical strengths

The central theoretical separation is clear and consequential: it formalizes why risk certificates and optimization comparisons can disagree, with explicit bounds and matched controls—fixed rescaling, fixed anchor mixtures, and separate step optimization. The four-moment curvature criterion and finite-sample audit protocol are useful methodological contributions. Honest reporting of mixed network results and failed endpoint confirmation also increases credibility. These support acceptance because the paper provides a principled evaluation framework beyond “lower MSE” and carefully separates the value of anchor information from that of a nonlinear mixing coefficient.

Critical weaknesses

Empirical support for practical impact is weak. Checkpoint experiments show no consistent nonlinear advantage, and the local audit fails endpoint confirmation; the main theorem is asymptotic/fixed-m under an exact quadratic sphere law, not trained networks. The four-moment diagnostic requires paid gradients/HVPs and known curvature coordinates, limiting deployability. Without stronger evidence that the separation matters in realistic ZO fine-tuning or that the diagnostics are actionable at feasible cost, practical significance is uncertain. This is material if acceptance requires validated optimization benefit, not only a theoretical caution.

Questions for authors
  1. Is the main intended claim a theoretical warning about risk-based evaluation, or a practical method for improving ZO updates?

  2. What finite-dimensional regimes beyond the large-r, fixed-m limit exhibit both certified risk reduction and worse optimized descent, and how relevant are they to OPT-125M-scale D?

  3. How would the four-moment criterion be estimated in a function-value-only setting, and what is the total query cost relative to direct endpoint selection?

  4. Why does the expanded checkpoint study show no consistent nonlinear advantage—finite-step effects, approximation error, or the chosen anchor/ridge?

  5. Can the authors report any setting where nonlinear shrinkage yields a statistically confirmed improvement in actual loss after step selection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.