- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper examines whether shrinkage-induced reductions in gradient estimation error lead to better zeroth-order optimization. It derives an explicit Stein risk bound for finite batches of sphere-based ZO estimates and shows that a specified nonlinear shrinkage rule can reduce MSE while worsening expected descent after separate step-size optimization. It further studies anchor and curvature effects, supported by synthetic simulations, frozen OPT-125M checkpoint experiments, and a finite-sample confirmation procedure.
Critical strengths
-
The explicit sphere-law analysis establishes a technically substantive separation between estimation MSE and optimized expected descent for the same shrinkage rule and strength.
-
The use of fixed rescaling, shared-anchor controls, separate step-size selection, and independent evaluation helps distinguish the effects of scaling, anchor information, and nonlinear mixing.
Critical weaknesses
-
The main argument is fragmented. Sections 3–5 distribute the formulation, shrinkage rule, and theoretical results across several sections without sufficiently explaining their connections. The motivation and implications of the results deserve more attention, particularly the distinction between established identities and the central new contribution. A clearer separation between the methodological setup and theoretical analysis would substantially improve readability.
-
The experimental presentation is insufficiently self-contained. Understanding Sections 6–7 requires repeatedly consulting Appendices H and I for implementation details and distinctions between experimental stages. A unified experiments section should consolidate the final protocol and clearly explain which theoretical claim each experiment tests, where the assumptions differ, and what the results establish.
-
Operational records obscure the scientific narrative. Appendix I.2 includes runner and vendor filenames, configuration-recovery history, reconstructed records, and missing execution logs. Much of this material belongs in a reproducibility README or accompanying code documentation. Its prominence, together with repeated descriptions of earlier and later experiments, makes the manuscript read like a record of the research process rather than a sufficiently synthesized research argument.
Questions for authors
-
What is the principal new insight beyond the established distinction between estimation MSE and step-size-optimized descent? Please clarify how the main theoretical results jointly establish this contribution and distinguish it from supporting identities and extensions.
-
Could the authors consolidate the final experimental protocol and explicitly map the reported results to the theoretical claims? In particular, what conclusions about practical shrinkage are supported by the network experiments, given the limited confirmation at the selected full step?