- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper develops ARC-block, a tensor-wise adaptive cubic regularization method with per-block regularization and loss-based acceptance. Large-block cubic subproblems are solved in Lanczos spaces using true block-Hessian products. A second variant, ARC-phi1, uses a trust-controlled exponential relaxation of the local quadratic model. The analysis gives conditional cubic sweep complexity above a cross-block coupling threshold and separate, weaker guarantees for the exponential step. Experiments on implicit neural representations include a 91.4M-parameter model and high-accuracy FINER fitting, with ablations, wall-clock comparisons, and curvature diagnostics.
Critical strengths
-
Substantial reported gains on difficult INR fitting problems. On FINER image 0, ARC-phi1 reaches 95.6 dB at approximately Adam's million-step wall-clock budget, compared with Adam's 78.2 dB best value (Section 6.3 and Table 6). Table 6 identifies the ARC-phi1 configuration as Lanczos depth 163. The 20-image study broadens the evidence beyond one image, although it uses one seed and different overall budgets. The large-model experiments and explicit wall-clock measurements make this a consequential engineering contribution.
-
Useful ablations and candid implementation diagnostics. The paper examines per-block acceptance, fresh large-block gradients, regularization floors, Krylov depth, and exponential versus cubic steps. Appendix S reports how often the cubic steps satisfy the theorem's residual condition and explicitly distinguishes truncated steps from certified ones. These diagnostics help explain the practical algorithm and identify the limits of its theoretical coverage.
Critical weaknesses
-
The second-order clause has a skipped-block gap, and the cubic rate has limited coverage of the experiments. Algorithm 1 sets the step to zero or skips a block when its gradient is zero. For the smooth, lower-bounded one-block objective at , this leaves the iterate unchanged even though and the coupling is zero. Theorem B.10 imposes its additional curvature condition on accepted steps; its full-space curvature conclusion also needs a condition or check on skipped blocks. This objection concerns the second-order conclusion, not the first-order gradient bound. Separately, Table 20 reports that only 10% of accepted cubic trials in the gradient-form runs meet the required step-form residual test. The coupling diagnostics in Appendix I.5 also put the reported counting thresholds far above the achieved gradient norms. These are acknowledged limits on the cubic theorem's relevance. The paper correctly gives ARC-phi1 a separate, weaker analysis; the cubic rate should not be used to explain that variant's high-accuracy behavior.
-
The long-run comparisons do not establish an intrinsic accuracy ceiling for the baselines. In Appendix I.5, Adam uses the selected learning rate , SOAP inherits Adam's learning rate, and AdaCubic uses its authors' settings. Only ARC-phi1 receives the extended plateau runs, with three images taking approximately 20.0–27.5 hours. The reported late deterioration of the baseline runs does not establish what learning-rate reduction or independently tuned SOAP would attain. The image-0 equal-time advantage remains meaningful, but the much larger plateau gap and its attribution to saddle escape need stronger controls, particularly with one seed per image in the long-run study.
Questions for authors
-
Soundness clarification. How should Algorithm 1 and the second-order clause of Theorem B.10 handle a zero-gradient block with negative curvature, such as at zero? Is a curvature check performed before skipping, or should the theorem add a suitable condition on skipped blocks?
-
Evidence for the baseline ceiling. What FINER accuracy is attained by Adam with a decaying learning rate or plateau-triggered reduction, and by independently tuned SOAP, under matched wall-clock budgets and stopping criteria? This would help distinguish the limitations of the reported configurations from limitations of the methods themselves.
-
Scope clarification. For the cubic configurations, which complete runs satisfy all rate assumptions, including the residual test, fresh gradients, and rejection handling? How does the positive acceptance tolerance affect the claim? Proposition A.10's discussion retains a subtractive term proportional to that tolerance, and Table 20 records accepted trials that do not meet the unadjusted decrease bound. Please distinguish these guarantees from the separate ARC-phi1 analysis.
Thank you for a thorough review. Second-order optimization in neural networks is difficult to evaluate objectively against first-order methods. A second-order step uses curvature, through some approximation of the Hessian, and is more accurate than a first-order step, while at a higher cost per step. Because one sweep costs many gradients, the sweep count looks much smaller than Adam’s step count. On the FINER ESM images, two hours gives Adam about 1.8 million steps. In the same time ARC-φ1 completes about 8,500 sweeps at Lanczos depth 25 and about 1,200 sweeps at depth 163. A sweep is one pass over the parameter blocks. On FINER there are B = 10 blocks, and at full Lanczos depth with one trial per block a sweep costs B(L+3) gradient equivalents: 10(25+3) = 280 at depth 25 and 10(163+3) = 1,660 at depth 163. A gradient equivalent is one backward pass, same as an Adam step. An extensive experiment in which the degree is determined by the residual test has not been run on FINER yet (lack of server time). On loss landscapes with cross-coupled ill-conditioning and many saddles, which first-order methods traverse poorly, a second-order method can be competitive in wall-clock time.
Neither FINER nor ViSIR have train/validate/test split by task nature, there is no held out data, and the optimizers are evaluated only on the lowest loss they can achieve. The same is often true for physics-informed neural networks (PINNs) modeling physics processes (we conducted experiments on PINNs for resistive switching in semiconductors). Best train loss does not often translate into best test loss. A new mini-batch between sweeps replaces the objective, so a step accepted on the current batch is a decrease of that batch only. The reported FINER and ViSIR results are deterministic full-batch runs. We did experiment with progressive batch approach on tasks with held out data (e.g. image classification) and obtained encouraging results. We have quite a bit of data on that task with search steps that later became subsolvers of ARC-block framework. The greater the ill-conditioning for a given dataset (e.g. a dataset of deteriorated house number images), the more likely was the second-order advantage. The experimental data in Appendices span around 10 months. ARC-phi1 as an ARC subsolver did not exist yet when SDF experiment (Table 16) was run. With a focus on FINER and ViSIR we missed to run it.
We would like to stress the contributions over the closest state of the art.
The ARC-φ1 subsolver's search step is novel. It replaces the cubic model minimizer with an exact single step decrease law for every sign pattern of the spectrum, and a bounded escape along negative modes. One trial on a single parameter block, regardless its size above 512 params, takes one geval to build the block's computation graph, L HVPs to form the Krylov subspace, one HVP for the predicted model decrease, and one geval to evaluate the function value at the trial point to accept or reject a step, for a total of L+3 gevals. Graph and vector sizes grow with block size.
The Chebyshev second kind recurrence subsolver provides a degree independent memory footprint (three block-sized buffers versus L + 1 Lanczos vectors), which at 91.4M parameters is 7.0-8.7 GB against ~11 GB at comparable accuracy. While this subsolver does not give best PSNR, it provides significant memory savings, which might be beneficial for tasks where most second-order methods are infeasible due to memory requirements
Against AdaCubic (published in prestigious AI journal TMLR in 2026), the closest scalable cubic regularized optimizer, ARC-block differs by design, an exact cubic step in a Krylov subspace of the true block Hessian against a Hutchinson diagonal estimate, and in outcome: 46.4 dB against 125.5 dB in the same 2 h on the same image, and 131.5 +/- 3.3 dB vs 46.7 +- 0.3 dB over 20 images with both optimizers run until no improvement observed. Inherently, AdaCubic performs worse on this loss landscape because of the diagonal Hutchinson approximation of Hessian.
The dense solve cubic Newton methods (e.g. Nesterov and Polyak, Math. Progr. 2006; Doikov et al. ICML 2023, and many other publications on second-order optimizers for neural networks at top venues that we referenced) are evaluated on problems of at most a few thousand variables, using gevals (gradient equivalent, i.e. one backward pass) instead of wall clock time and without any comparison to Adam and its variants (e.g. SOAP)
A landscape fingerprint measured along Adam search trajectory distinguishes axis-aligned, coupled, and saddle-dominated spectra. Thus, the ARC-block with auto-dispatch on, decides where the cubic step is the appropriate subsolver. A step dispatch based on such analysis is novel. No other published ARC framework combines all the features of ARC-block together at the time of writing (Appendices L and Q). The computation and memory costs are in Table 1, Appendices M and N.1.