Affine-invariant Cubic Newton with Weak Learners

Modern gradient boosting algorithms are based on Newton's method, which excels empirically but lacks global convergence guarantees. While recent work on Gradient-Regularized Newton (GRN) boosting achieved a global $\mathcal{O}(1/{\gamma }^{12}{k}^2)$ rate, its analysis was not able to exploit the natural Hessian geometry and suffers from a ${\gamma }^{12}$ dependence on a worst-case `weak gradient edge' $\gamma$. In this paper, we tackle these theoretical drawbacks by generalizing the Affine-Invariant Cubic Newton (AICN) scheme to convex optimization with weak learners. Our analysis is based on average cosine values ${\overline{\Theta }}_k$ along the optimization path and directional improvements ${\delta }_k$ in the Hessian-induced norm. We show that the weak-AICN update reduces to a damped weak-Newton step with an explicit stepsize determined by ${\delta }_k$. Under semi-strong self-concordance, we establish a global $\mathcal{O}(1/{\overline{\Theta }}_k^4{k}^2)$ rate, significantly improving upon the weak-GRN bound. We further prove a local linear rate, global linear convergence for Hessian-dominated losses, and cosine angle lower bounds. Finally, we demonstrate the broader applicability of this framework beyond gradient boosting by showing that Newton coordinate descent emerges naturally as a special case of optimization with weak learners.