Blockwise Stabilized Adaptive Cubic Regularization with Trust-Controlled ${\phi }_1$ Subsolver

Overparameterized networks often have ill-conditioned, saddle-rich loss landscapes. They are usually trained with first-order methods that struggle on those landscapes, while many second-order methods stay infeasible at scale. Dense Hessian variants in the cubic Newton family stop scaling near $15$k parameters. Some scalable cubic methods replace the Hessian by a cheap approximation and drop the exact block curvature. This paper introduces ARC-block, a blockwise adaptive regularization with cubics (ARC) optimizer. For each parameter tensor it minimizes a cubic model over the true block Hessian, with a per-block adaptive cubic constant and a monotone guard on the full loss. Large tensors are handled matrix-free in a Lanczos-built Krylov subspace. ARC-block's CubicKrylov subsolver's descent is monotone with a per-block cubic decrease bound. For exact steps, and for Krylov steps that meet the Cartis-Gould-Toint residual test, the proof gives an $\mathcal{O}(\sqrt{B}{\epsilon }^{-3/2})$ sweep bound above a cross-block coupling floor. A second subsolver, ARC-${\phi }_1$, uses a trust-controlled exponential relaxation step: Newton on high-curvature modes, gradient descent on flat modes, and a clamped exponential escape on negative modes, at the same per-trial cost. Among the cubic Newton methods evaluated here, these are the only ones whose steps stay exact on every parameter block (within the Krylov subspace), including on a $91.4$M-parameter INR. Stopped after $400$ sweeps without a $0.05$ dB gain on FINER 2D image-fitting INR network, ARC-${\phi }_1$ reaches $133.5$ dB peak signal-to-noise ratio (PSNR), with no post peak collapse. Tuned Adam peaks at $78.2$ dB at $2,629$ s (step $\sim 625$k, about $44$ minutes), further steps degrade its PSNR. ARC-${\phi }_1$ reaches that $78.2$ dB sooner, at $1,927$ s, and by Adam's peak time is already at $84.2$ dB. By Adam's $10^6$-step budget (about $70$ minutes) ARC-${\phi }_1$ reaches $95.6$ dB. On the same benchmark, SOAP, L-BFGS, and the scalable diagonal cubic method AdaCubic stay at or below Adam's $78.2$ dB peak. Over the benchmark's twenty Earth System Model images, run to a plateau rule, ARC-${\phi }_1$'s per-image PSNR has median $132.4$ and mean $131.5\pm 3.3$ dB against tuned Adam's $72.8\pm 4.5$ dB. The paper provides the FINER network loss landscape analysis explaining the behavior of these optimizers. The paper provides the FINER network loss landscape analysis explaining the behavior of these optimizers.