Finite-Sample Performance of Gradient Descent in Logistic Regression with Gaussian Design

We consider the parameter estimation problem in logistic regression with Gaussian design: the estimation of a fixed unknown parameter ${\theta }^{\ast }\in {\mathbb{R}}^d$ ($\Vert {\theta }^{\ast }{\Vert }_2\ge$ 1) from $n$ i.i.d. samples $\{(x_i,y_i){\}}_{i=1}^n$, where $x_i\sim N(0,I_d)$ and $y_i\mid x_i\sim \mathrm{Bernoulli}$ (1/ (1+exp($-x_i^T$ ${\theta }^{\ast }$ ))). Our main aim is to characterize the finite-sample estimation performance and convergence behavior of gradient descent (GD) on the maximum likelihood objective (i.e., the logistic loss). Under small $O(1)$ stepsize and $0$ initialization, it is shown that GD linearly converges to a small neighborhood of the true parameter and achieves ${\ell }_2$ error of order $O(\sqrt{\Vert \theta {\Vert }_2^{5}d/n}).$ This substantially goes beyond existing theoretical results that lack non-asymptotic estimation error rate and exhibit much slower parameter convergence. We also establish a faster local linear convergence to the same statistical error under a large stepsize. The main technical component is to show that the gradient of the logistic loss satisfies a certain approximate invertibility condition (AIC). To that end, we bound the concentration term via covering and peeling arguments, and show that the bias term is a contraction by a delicate eigenvalue analysis of the population Hessian matrices. Finally, we build upon the recent work Matsumoto and Mazumdar (2025) and devise a novel efficient estimator that attains a sharper rate in high dimensions. This indicates that the existing non-asymptotic guarantees exhibit sub-optimal dependence on $\Vert \theta {\Vert }_2$, and that in many regimes $\Theta (\sqrt{\Vert \theta {\Vert }_{2}d/n})$ is the tight estimation error rate. Numerical examples are provided to corroborate our theoretical results for GD.