- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper focuses on giving an asymptotic analysis of Stochastic Newton method when Hessian are not available. The authors then suggest to consider the case where the curvature is learned via inference with perturbed gradient calls (also called "score secants"). The authors give an introduction to related work before defining their method, which involves a projection of the Hessian estimation via score secants onto the set of symmetric positive definite matrices with bounded eigenvalues. Them, authors describe their hypotheses and introduce their main theoretical results in section 3 and validate them experimentally in section 4.
Critical strengths
The paper extends the results from Am & Huo in the case of an indirect access to curvature estimation via "score secant", aka finite differences of gradient calls with perturbed model.
Critical weaknesses
- The readability of the paper is not good. Many notations are not nicely introduced, the jargon used makes understanding complicated and certain sections like Section 2 hard to understand. The related work section is long and unclear. The problem is not introduced unlike in Leluc & Portier 2023, Godichon-Baggioni & Werge 2025 or An & Huo 2026. The algorithm is not clearly stated : not in the main paper and in a too verbose version in the appendix.
- A clear comparison and discussion of the hypotheses and results from this paper and previous works is needed in the main paper. It would be better to highlight what is in common with An and Huo (2026) and more explicitly explain what is the added value of this work.
- Assumption (T1) seems to be taken from an unpublished work An and Huo (2026) (cf their Assumption 5), yet it seems very restrictive and the scope of it's "applicability" should be discussed in the main paper, rather than in the last page of the appendix.
Questions for authors
- Can the Proposition C.42 be applied to all the regression cases considered, including quantile regression which score function is not smooth? In other words, is it reasonable to assume (T1) for the method of this paper
- Could you please highlight more explicitly the motivation of the paper? When does it happen one cannot directly access the Hessian? What are the other concurrent methods to solve these situations (first or second order)?
3 comments
Thank you again for your careful review and constructive feedback. We hope that our responses and revisions have addressed your concerns and questions. If you find that they resolve your main concerns, we would be grateful if you would reconsider your assessment and score in light of these changes. We would be happy to clarify any outstanding points.
Concern about assumption T1
I thank the authors for their answers and their efforts.
I checked Prop C.42 and still do not understand why the assumption that the gradient bias shrinks to zero with the step size sequence. Can the authors justify this simply and clearly?
This proposition is the equivalent of Prop. 2.1 of An and Huo 2026, but in their case, authors do not need to impose their gradient bias to shrink, as they assume to unbiased gradient estimate.
Thank you for your time and valuable comments.
W1. Section 2 now places the update order beside the defining equations. Starting from the past-measurable θt−1 and Bt−1, the method solves Bt−1st=Gt(θt−1) and updates θt=θt−1−ηtst. Auxiliary score pairs are centered at θt−1, and their projected recursive average forms Bt, first used at the next primary update. The lagged iterate average and score outer-product average are updated online. Detailed pseudocode remains in the appendix. The introduction states the quantile-regression motivation directly, and the related-work discussion separates online inference, curvature estimation and nonsmooth quantile methods.
W2. The comparison now separates shared assumptions from the secant calculation. The mean-square iterate bound (T1) corresponds to Assumption 5 of An and Huo; their decomposition and stabilization threshold supply the transfer argument. Their Proposition 5.1 also supplies an L2-moment route to first-moment remainder control. Theorem 3.3 here verifies stabilization for the projected score-secant recursion using the exponent q, radius δt and batch size Rt. Quantile endpoint coupling gives the different sufficient restrictions for q=1 and q=2. The resulting limit shares the usual averaged covariance with stabilized Newton and SGD procedures, while the learned sensitivity is available for sandwich inference.
W3. (T1) is explicitly retained as a model-specific stability input in the main theorem. Secant moment bounds verify (T2), separately from the parameter-trajectory rate in (T1). The discussion next to (Q5) states both the deterministic-time localization and the mean-square rate required for the actual quantile recursion. Proposition C.42 provides a sufficient risk-descent route on a valid stability region. The general multivariate quantile result remains conditional on establishing that region and rate, or directly verifying (Q5).
Q1. Proposition C.42 applies through the population risk, so a discontinuous sample score is admissible. Its hypotheses are a convex region containing the actual trajectory and target, uniform strong convexity and smoothness of the population risk there, a predictable preconditioner with eigenvalues bounded away from zero and infinity, conditional score variance bounded by C1+∣g(θ)∣2, and approximation bias bounded by Cηt1/2. For a direct unbiased score the bias is zero. For a sufficiently small initial stepsize, these conditions give
\le(1-c\eta_t)[R(\theta_{t-1})-R(\theta_\star)]+C\eta_t^2,$$ and hence the mean-square rate for $\eta_t\asymp t^{-\alpha}$, $1/2 < \alpha < 1$. Least squares satisfies a global quadratic-risk version under its stated moments. Logistic, Huber and quantile models require their stated trajectory conditions or a separate stability argument. The scalar compact-support quantile example verifies an invariant interval and the rate by its own drift calculation. Local positive population curvature therefore provides the local risk conditions, with trajectory stability handled separately. **Q2.** Quantile regression gives a concrete setting: the sample check loss is piecewise linear, with zero Hessian wherever that Hessian exists, whereas the population sensitivity is $A_\tau=\mathbb E\\{f_{Y\mid X}(X^T\theta_\tau\mid X)XX^T\\}$ and may be positive definite. Direct score SGD is a first-order alternative; sample-Hessian Newton is available for smooth losses; and stochastic score-difference or quasi-Newton methods use other curvature approximations. For inference, sandwich estimation, batching and iterate-based self-normalization are distinct choices. Our method uses paired score evaluations to learn and apply curvature, then reuses it in the sandwich estimator when the clipping set contains the population sensitivity. Its intended scope is fixed, moderate dimension with available auxiliary scores. Function-value primary updates retain that auxiliary-score requirement, and the additional observations, score calls and dense-matrix operations are part of the computational tradeoff. Thank you again for your valuable feedback. We hope our responses address your concerns and welcome any further questions or discussion.