Principal-Subspace Newton–Gradient Methods with Curvature Tracking and Reuse

A few large Hessian eigenvalues can restrict the stepsize of gradient descent and slow progress in flatter directions, while full Newton steps can be too costly in high dimensions. We analyze principal-subspace Newton-gradient (PSNG) methods, which combine Newton steps in an approximate leading Hessian eigenspace with gradient steps in its orthogonal complement. On strongly convex quadratics, we show that a fixed, sufficiently accurate subspace preserves spectral acceleration: the iteration bound depends on the condition number of the remaining spectrum rather than that of the full Hessian. For objectives with a varying Hessian, we first consider updating the subspace and projected Hessian at every iteration. Under suitable local conditions, warm starts enable linear convergence with a fixed Hessian-vector product (HVP) budget per iteration after initialization. We then propose lazy PSNG, which reuses both quantities until accumulated step length reaches a prescribed threshold. Under corresponding local conditions, lazy PSNG retains linear convergence with a finite total HVP bound independent of the target accuracy, for fixed initialization and parameters. With additional global strong convexity and smoothness, an Armijo phase and a prescribed local restart extend this guarantee to arbitrary starting points. Experiments on quadratics and strongly convex logistic regression illustrate spectral acceleration and reduced HVP costs.