acceptodds
Under review as a conference paper at ICLR 2027

Blockwise Stabilized Adaptive Cubic Regularization with Trust-Controlled Subsolver

Abstract

Overparameterized networks often have ill-conditioned, saddle-rich loss landscapes. They are usually trained with first-order methods that struggle on those landscapes, while many second-order methods stay infeasible at scale. Dense Hessian variants in the cubic Newton family stop scaling near k parameters. Some scalable cubic methods replace the Hessian by a cheap approximation and drop the exact block curvature. This paper introduces ARC-block, a blockwise adaptive regularization with cubics (ARC) optimizer. For each parameter tensor it minimizes a cubic model over the true block Hessian, with a per-block adaptive cubic constant and a monotone guard on the full loss. Large tensors are handled matrix-free in a Lanczos-built Krylov subspace. ARC-block's CubicKrylov subsolver's descent is monotone with a per-block cubic decrease bound. For exact steps, and for Krylov steps that meet the Cartis-Gould-Toint residual test, the proof gives an sweep bound above a cross-block coupling floor. A second subsolver, ARC-, uses a trust-controlled exponential relaxation step: Newton on high-curvature modes, gradient descent on flat modes, and a clamped exponential escape on negative modes, at the same per-trial cost. Among the cubic Newton methods evaluated here, these are the only ones whose steps stay exact on every parameter block (within the Krylov subspace), including on a M-parameter INR. Stopped after sweeps without a dB gain on FINER 2D image-fitting INR network, ARC- reaches dB peak signal-to-noise ratio (PSNR), with no post peak collapse. Tuned Adam peaks at dB at s (step k, about minutes), further steps degrade its PSNR. ARC- reaches that dB sooner, at s, and by Adam's peak time is already at dB. By Adam's -step budget (about minutes) ARC- reaches dB. On the same benchmark, SOAP, L-BFGS, and the scalable diagonal cubic method AdaCubic stay at or below Adam's dB peak. Over the benchmark's twenty Earth System Model images, run to a plateau rule, ARC-'s per-image PSNR has median and mean dB against tuned Adam's dB. The paper provides the FINER network loss landscape analysis explaining the behavior of these optimizers. The paper provides the FINER network loss landscape analysis explaining the behavior of these optimizers.

Primary area:
optimization
Keywords:
cubic regularization, second-order optimization, blockwise optimization, Krylov methods, implicit neural representations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ovBhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper develops ARC-block, a tensor-wise adaptive cubic regularization method with per-block regularization and loss-based acceptance. Large-block cubic subproblems are solved in Lanczos spaces using true block-Hessian products. A second variant, ARC-phi1, uses a trust-controlled exponential relaxation of the local quadratic model. The analysis gives conditional cubic sweep complexity above a cross-block coupling threshold and separate, weaker guarantees for the exponential step. Experiments on implicit neural representations include a 91.4M-parameter model and high-accuracy FINER fitting, with ablations, wall-clock comparisons, and curvature diagnostics.

Critical strengths
  1. Substantial reported gains on difficult INR fitting problems. On FINER image 0, ARC-phi1 reaches 95.6 dB at approximately Adam's million-step wall-clock budget, compared with Adam's 78.2 dB best value (Section 6.3 and Table 6). Table 6 identifies the ARC-phi1 configuration as Lanczos depth 163. The 20-image study broadens the evidence beyond one image, although it uses one seed and different overall budgets. The large-model experiments and explicit wall-clock measurements make this a consequential engineering contribution.

  2. Useful ablations and candid implementation diagnostics. The paper examines per-block acceptance, fresh large-block gradients, regularization floors, Krylov depth, and exponential versus cubic steps. Appendix S reports how often the cubic steps satisfy the theorem's residual condition and explicitly distinguishes truncated steps from certified ones. These diagnostics help explain the practical algorithm and identify the limits of its theoretical coverage.

Critical weaknesses
  1. The second-order clause has a skipped-block gap, and the cubic rate has limited coverage of the experiments. Algorithm 1 sets the step to zero or skips a block when its gradient is zero. For the smooth, lower-bounded one-block objective f(x)=1+cos⁡(x)f(x)=1+\cos(x) at x=0x=0, this leaves the iterate unchanged even though f′′(0)=−1f''(0)=-1 and the coupling is zero. Theorem B.10 imposes its additional curvature condition on accepted steps; its full-space curvature conclusion also needs a condition or check on skipped blocks. This objection concerns the second-order conclusion, not the first-order gradient bound. Separately, Table 20 reports that only 10% of accepted cubic trials in the gradient-form runs meet the required step-form residual test. The coupling diagnostics in Appendix I.5 also put the reported counting thresholds far above the achieved gradient norms. These are acknowledged limits on the cubic theorem's relevance. The paper correctly gives ARC-phi1 a separate, weaker analysis; the cubic rate should not be used to explain that variant's high-accuracy behavior.

  2. The long-run comparisons do not establish an intrinsic accuracy ceiling for the baselines. In Appendix I.5, Adam uses the selected learning rate 10−410^{-4}, SOAP inherits Adam's learning rate, and AdaCubic uses its authors' settings. Only ARC-phi1 receives the extended plateau runs, with three images taking approximately 20.0–27.5 hours. The reported late deterioration of the baseline runs does not establish what learning-rate reduction or independently tuned SOAP would attain. The image-0 equal-time advantage remains meaningful, but the much larger plateau gap and its attribution to saddle escape need stronger controls, particularly with one seed per image in the long-run study.

Questions for authors
  1. Soundness clarification. How should Algorithm 1 and the second-order clause of Theorem B.10 handle a zero-gradient block with negative curvature, such as f(x)=1+cos⁡(x)f(x)=1+\cos(x) at zero? Is a curvature check performed before skipping, or should the theorem add a suitable condition on skipped blocks?

  2. Evidence for the baseline ceiling. What FINER accuracy is attained by Adam with a decaying learning rate or plateau-triggered reduction, and by independently tuned SOAP, under matched wall-clock budgets and stopping criteria? This would help distinguish the limitations of the reported configurations from limitations of the methods themselves.

  3. Scope clarification. For the cubic configurations, which complete runs satisfy all rate assumptions, including the residual test, fresh gradients, and rejection handling? How does the positive acceptance tolerance affect the claim? Proposition A.10's discussion retains a subtractive term proportional to that tolerance, and Table 20 records accepted trials that do not meet the unadjusted decrease bound. Please distinguish these guarantees from the separate ARC-phi1 analysis.

Reviewer_6pe8OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The authors develop ARC-block, an adaptive cubic-regularization method that works on parameter-tensor blocks rather than a full Hessian. Large blocks are treated with Lanczos/Hessian–vector products, while an alternative ARC-φ1 subsolver uses a trust-controlled exponential-relaxation step. The paper combines convergence analyses with striking image-fitting results, particularly on FINER and the 20 Earth System Model images.

Critical strengths

The paper contains a lot of careful optimization engineering. Per-block regularization, fresh gradients for large blocks, and acceptance against the actual loss are sensible responses to problems that arise when cubic Newton methods are scaled up. I was particularly interested in the 91.4M-parameter ViSIR experiment: working with curvature information at that size is not trivial. The FINER results also deserve attention. The paper makes an effort to compare wall-clock time, not just iterations, and provides extensive supple- mentary diagnostics and ablations. It is also appropriately explicit that the O(ε−3/2) cubic guarantee does not automatically extend to ARC-φ1. Reference list is comprehensive.

Critical weaknesses

I had trouble connecting the strongest theoretical statement to the optimizer that produces the headline numbers. The sweep theorem (Theorem B.10) uses fresh block gradients and a certified subproblem residual; Phase A of Algorithm 1 instead reuses a sweep-start gradient, and Table 20 in Appendix S reports about 10% step-form certification for the gradient-form stopping configuration at Lmax = 163. More importantly, Appendix I.5 reports coupling floors around 7.7×103–3.8×105, far above the attained gradient norms. So the stated cubic sweep rate does not explain the experimentally interesting low-gradient regime, and the best FINER numbers come from ARC-φ1, which has a different, weaker conditional guarantee.

I appreciate that the paper includes these caveats, but they should be much easier to see in the main text. The reported PSNR gains are unusually large, which makes replication particularly important. The 20- image experiment uses only one initialization seed, and the clearest wins come from a relatively specialized class of INR tasks. Multiple seeds and an equally careful time-matched comparison on another architecture would make the empirical case much more convincing

Questions for authors
  1. Please give a compact table distinguishing the practical algorithm variants from the variants actually covered by each theorem, including gradient reuse and stopping criteria?

  2. Do the FINER gains persist across multiple initializations, with wall-clock variability reported as well as best PSNR?

Reviewer_dCguOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a second-order block optimization method based on cubic regularization. For each block of parameters, the method constructs a cubic model of the objective function using the exact block Hessian. In the case of large blocks, the cubic subproblem is solved approximately within a Krylov subspace, which is generated via Lanczos iterations that do not require explicit matrix computation. Furthermore, the authors introduce a variant of the method ARC−(φ1)ARC-(\varphi_1) in which the optimization step is calculated using the matrix exponential. The regularization parameters for each block are adapted independently based on the observed changes in the objective function.

Critical strengths
  1. The combination of blockwise cubic regularization with Krylov subspace methods is an interesting approach to second-order optimization. The paper also provides a clear discussion of its contributions relative to prior work.

  2. The reported experiments demonstrate substantial improvements over the considered baselines in terms of the quality of the trained models, suggesting that the proposed approach can be effective in the evaluated settings.

Critical weaknesses
  1. While the experiments include a model with 91.4M parameters, the paper does not provide a systematic analysis of how computational cost and memory consumption scale with the number of model parameters or the dataset size. Such an analysis would be particularly valuable for assessing the practical applicability of the proposed second-order method to larger-scale learning problems.

  2. Table 16 reports both training and test PSNR. While the proposed method achieves substantial improvements in training PSNR, these gains do not appear to translate into comparable improvements on the test set. In particular, SOAP achieves a higher test PSNR than ARC-block despite the latter's stronger training performance. Moreover, analogous train/test comparisons are not provided for ARC-φ1\varphi_1. This makes it difficult to assess whether the improvements in optimization performance consistently translate into better generalization.

  3. According to the text the proposed optimizer can be used as a general-purpose optimization method, which aligns the ICLR purpose of research in representation learning. However the experimental evaluation is largely restricted to a relatively narrow range of tasks, primarily implicit neural representations (INRs).

  4. Although the paper includes multi-seed experiments involving 20 images and 3 random seeds, the main results for ARC-φ1\varphi_1 under training to convergence appear to rely on a single seed per image. Consequently, the reported mean +- standard deviation primarily reflects variability across images rather than variability across random initializations.

Questions for authors
  1. Could the authors provide a more comprehensive comparison of training and test PSNR across the evaluated tasks? In particular, does the advantage of ARC-φ1\varphi_1 persist on held-out data, or is it primarily reflected in improved optimization of the training objective?

  2. How do the computational cost and memory requirements of the proposed method scale with model size and dataset size? Additionally, how well does the method accommodate mini-batch training, given the need to evaluate the objective function when deciding whether to accept each blockwise update?

3 comments

Thank you for a thorough review. Second-order optimization in neural networks is difficult to evaluate objectively against first-order methods. A second-order step uses curvature, through some approximation of the Hessian, and is more accurate than a first-order step, while at a higher cost per step. Because one sweep costs many gradients, the sweep count looks much smaller than Adam’s step count. On the FINER ESM images, two hours gives Adam about 1.8 million steps. In the same time ARC-φ1\varphi1 completes about 8,500 sweeps at Lanczos depth 25 and about 1,200 sweeps at depth 163. A sweep is one pass over the parameter blocks. On FINER there are B = 10 blocks, and at full Lanczos depth with one trial per block a sweep costs B(L+3) gradient equivalents: 10(25+3) = 280 at depth 25 and 10(163+3) = 1,660 at depth 163. A gradient equivalent is one backward pass, same as an Adam step. An extensive experiment in which the degree is determined by the residual test has not been run on FINER yet (lack of server time). On loss landscapes with cross-coupled ill-conditioning and many saddles, which first-order methods traverse poorly, a second-order method can be competitive in wall-clock time.

Neither FINER nor ViSIR have train/validate/test split by task nature, there is no held out data, and the optimizers are evaluated only on the lowest loss they can achieve. The same is often true for physics-informed neural networks (PINNs) modeling physics processes (we conducted experiments on PINNs for resistive switching in semiconductors). Best train loss does not often translate into best test loss. A new mini-batch between sweeps replaces the objective, so a step accepted on the current batch is a decrease of that batch only. The reported FINER and ViSIR results are deterministic full-batch runs. We did experiment with progressive batch approach on tasks with held out data (e.g. image classification) and obtained encouraging results. We have quite a bit of data on that task with search steps that later became subsolvers of ARC-block framework. The greater the ill-conditioning for a given dataset (e.g. a dataset of deteriorated house number images), the more likely was the second-order advantage. The experimental data in Appendices span around 10 months. ARC-phi1 as an ARC subsolver did not exist yet when SDF experiment (Table 16) was run. With a focus on FINER and ViSIR we missed to run it.

We would like to stress the contributions over the closest state of the art.

  1. The ARC-φ1\varphi1 subsolver's search step is novel. It replaces the cubic model minimizer with an exact single step decrease law for every sign pattern of the spectrum, and a bounded escape along negative modes. One trial on a single parameter block, regardless its size above 512 params, takes one geval to build the block's computation graph, L HVPs to form the Krylov subspace, one HVP for the predicted model decrease, and one geval to evaluate the function value at the trial point to accept or reject a step, for a total of L+3 gevals. Graph and vector sizes grow with block size.

  2. The Chebyshev second kind recurrence subsolver provides a degree independent memory footprint (three block-sized buffers versus L + 1 Lanczos vectors), which at 91.4M parameters is 7.0-8.7 GB against ~11 GB at comparable accuracy. While this subsolver does not give best PSNR, it provides significant memory savings, which might be beneficial for tasks where most second-order methods are infeasible due to memory requirements

  3. Against AdaCubic (published in prestigious AI journal TMLR in 2026), the closest scalable cubic regularized optimizer, ARC-block differs by design, an exact cubic step in a Krylov subspace of the true block Hessian against a Hutchinson diagonal estimate, and in outcome: 46.4 dB against 125.5 dB in the same 2 h on the same image, and 131.5 +/- 3.3 dB vs 46.7 +- 0.3 dB over 20 images with both optimizers run until no improvement observed. Inherently, AdaCubic performs worse on this loss landscape because of the diagonal Hutchinson approximation of Hessian.

  4. The dense solve cubic Newton methods (e.g. Nesterov and Polyak, Math. Progr. 2006; Doikov et al. ICML 2023, and many other publications on second-order optimizers for neural networks at top venues that we referenced) are evaluated on problems of at most a few thousand variables, using gevals (gradient equivalent, i.e. one backward pass) instead of wall clock time and without any comparison to Adam and its variants (e.g. SOAP)

  5. A landscape fingerprint measured along Adam search trajectory distinguishes axis-aligned, coupled, and saddle-dominated spectra. Thus, the ARC-block with auto-dispatch on, decides where the cubic step is the appropriate subsolver. A step dispatch based on such analysis is novel. No other published ARC framework combines all the features of ARC-block together at the time of writing (Appendices L and Q). The computation and memory costs are in Table 1, Appendices M and N.1.

Thank you for your response and for clarifying the evaluation setup and per-block costs. My main questions remain open: whether the gains from ARC-φ1\varphi_1 carry over to held-out data, how the method scales with model and dataset size, and how it performs in practice with mini-batch training. I will therefore maintain my original score

The ARC-φ1\varphi_1method has high scalability with the size of each parameter tensor and with the number of tensors.

Parameters are split by the block size. A small block, feasible for an exact cubic step (e.g. 512512 on ViSIR and 10241024 on FINER), takes a lazy exact cubic step: one full block Hessian, an eigendecomposition, refreshed every nbn_b steps. That cost is O(nb)\mathcal{O}(n_b) Hessian--vector products to build the matrix and O(nb3)\mathcal{O}(n_b^3) to factor it. On the 91.491.4M ViSIR model that is 2323 tensors and 7,9387{,}938 parameters.

Every large block takes a matrix-free step. One trial of CubicKrylov or ARC-φ1\varphi_1 costs L+3L+3 gradient equivalents: one gradient, LL products to build the Krylov subspace, one product for the model decrease, and one function evaluation. That count does not depend on nbn_b. The secular solve is O(L3)\mathcal{O}(L^3) and is negligible while L≪nbL\ll n_b. The same trial on the 88.588.5M decoder, 96.896.8\\% of ViSIR in one tensor, uses L=10L=10 products. A dense Hessian of that block would be about 3×1073\times 10^{7} GB, and a factorization would be on the order of nb3n_b^3 operations. The original ARC, one trust region on the whole model, stalls at 4.04.0 dB on this network. Our blockwise ARC-block is both feasible and performs very well on it.

The stored Lanczos basis is L+1L+1 vectors of length nbn_b. For that decoder at L=10L=10 in float32 the basis is about 3.93.9 GB. At L=15L=15 in float64 the same basis is about 1111 GB. Reorthogonalization adds O(Lnb)\mathcal{O}(L n_b) work per product. The number of blocks BB affect the sweep: one visit per tensor per sweep, and the proved rate above the coupling floor is O(B,ε−3/2)\mathcal{O}(\sqrt{B}\\,\varepsilon^{-3/2}) sweeps. ViSIR has 4545 tensors. The 2222 large ones dominate the oracle count.

If the basis no longer fits, the Chebyshev second-kind recurrence replaces it. It keeps 3 vectors of length nbn_b at any degree.

A gradient or a Hessian--vector product is one backward pass on the current batch, so its wall-clock time scales with the number of samples and with the network. The number of such calls per trial stays L+3L+3. The reported runs are full-batch on one image, FINER at about 199199k parameters and ViSIR at 91.491.4M, or the Thai-statue SDF task with 100100k training points. The paper does not include a sweep over dataset size, and it does not use a small minibatch. A larger image or a larger point cloud therefore lengthens every oracle call in proportion to the batch, and leaves the per-trial call count unchanged.

ARC-φ1\varphi_1 has the same oracle cost and the same memory as CubicKrylov at equal depth. The two steps differ only in what they do with the Ritz values after the basis exists.

On a large block the search builds the same Lanczos basis as CubicKrylov: LL Hessian--vector products with full reorthogonalization, starting from gb/∥gb∥g_b/\Vert g_b\Vert, producing QL∈Rnb×LQ_L\in\mathbb{R}^{n_b\times L} and the L×LL\times L tridiagonal TLT_L. That basis is independent of the horizon hbh_b. The step is then

sb=−∥gb∥,QLV,diag(η~hb(νi)),V⊤e1,s_b=-\Vert g_b\Vert\\,Q_L V\\,\mathrm{diag}\big(\tilde\eta_{h_b}(\nu_i)\big)\\,V^\top e_1,

where νi\nu_i are the Ritz values and η~\tilde\eta is the clamped φ1\varphi_1 multiplier. It is formed by one L×LL\times L eigendecomposition, O(L3)\mathcal{O}(L^3), and one diagonal reweighting, O(L)\mathcal{O}(L), then a lift by QLQ_L, O(Lnb)\mathcal{O}(L n_b). There is no secular bisection. CubicKrylov spends that O(L3)\mathcal{O}(L^3) on the secular solve instead. For L≪nbL\ll n_b both are negligible next to the products.

One trial is L+3L+3 gradient equivalents: one gradient, LL products for the basis, one product for the model decrease, and one function evaluation. The cost table gives CubicKrylov the same count. At equal LL, a difference in the result is a difference in the step rule, not in the oracle count. The count does not depend on nbn_b. Wall-clock time and basis memory still scale linearly with the tensor: L+1L+1 vectors of length nbn_b, about 3.93.9 GB in float32 for the 88.588.5M ViSIR decoder at L=10L=10.

A new horizon does not require a new basis. Once the Ritz decomposition is stored, each candidate hh is one diagonal reweighting and one LL-dimensional model evaluation. The ranked variant therefore tries hb/2,,hb,,2hb\\{h_b/2,\\, h_b,\\, 2h_b\\} at that cost. A rejection changes σb\sigma_b, hence hb=ch/σbh_b=c_h/\sigma_b, and the next multiplier is applied to the same Ritz values. The truncation of the basis is caught by the ratio test, not by an extra factorization.

The small blocks are unchanged. They stay on the lazy exact cubic step, at O(nb)\mathcal{O}(n_b) products per Hessian build and O(nb3)\mathcal{O}(n_b^3) for the eigendecomposition, and only for tensors below the cutoff. A dataset affects the cost only through the batch inside each product: a larger image or a larger point cloud lengthens every one of the L+3L+3 calls and does not change how many calls a trial makes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.