- Rating
- 2
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper asks a basic question about second-order optimisers: if one builds a local quadratic model of the loss on one mini-batch and take a step, does a more faithful curvature matrix give a better step when measured on a different batch? The proxy under study is the generalised Gauss-Newton (GGN) matrix, compared against the positive part of the Hessian projected into the same Krylov subspace.
The method has three parts: 1. A geometry comparison of the two matrices in a shared subspace, which splits their difference into eigenvector misalignment and a difference in normalised spectrum, and points out that these scale-free scores don't tell the prediction error for a specific step. 2. An exact four-term accounting that splits the gap between the GGN prediction and the actual loss change into: the GGN vs positive-Hessian swap, the rest of the Hessian, a finite-step residual, and batch transfer (further split into a gradient-difference part and the rest). 3. Decision tests: seven fixed candidate steps (no-op, plus a negative-gradient and a damped Gauss-Newton direction at three radii each) are ranked by GGN, positive-Hessian and gradient-only selectors and compared to an oracle on held-out batches, plus an "equal-linear" intervention where candidates are rescaled to the same first-order decrease so only curvature can separate them.
Experiments use ResNet20 without normalisation on CIFAR-10/100 and a 262k-parameter ViT on CIFAR-10, six training runs per setting, checkpoints at epochs 0, 80 and 160. The authors report that batch transfer makes up 94-99% of the absolute error budget and is mostly the gradient differing between batches; that the GGN and positive-Hessian selectors pick the same action in every task; that under the equal-linear intervention they disagree in 5/6 runs, with the primary regret comparison inconclusive and a secondary held-out comparison favouring the GGN; and that finite-difference Hessian products inflate the misalignment score about 20x compared to exact autodiff. The paper presents this as a diagnostic protocol, not a new optimiser.
Critical strengths
1. The error accounting is a good tool and the numerics are solid.
The four-term split is exact and each term can be measured directly with Hessian/GGN-vector products plus two loss evaluations, so one can actually see where a bad prediction comes from. The matrix-vector products are checked against dense matrices on small models (App. B, errors around 1e-16), and the finite-difference vs autodiff comparison is a good practical warning - the misalignment score comes out about 20x too big with finite differences. The observation that scale-free geometry scores can't predict the error of a specific step (the A = 2B example in Sec. 3) is simple and correct. This matters for my assessment because the tool is independent of the conclusions drawn here - the tool itself produces real answers.
2. The supplementary material is very thorough and the authors are upfront about limitations.
Per-task records for all nine experiments, protocols with numerical gates, registration files with timestamps, and a README that says what is reproducible. The null primary result is reported as null. I give credit for this because the per-epoch and per-radius numbers in my weaknesses come from these files. It also means the authors can put together the stratified tables for the rebuttal easily. It doesn't change the protocol problem though.
Critical weaknesses
1. The test setup is too small to see curvature effects.
All candidate steps are fixed lengths (0.001, 0.01, 0.05 in parameter norm). At the two smaller radii the curvature part of the quadratic model is at most 12% of the gradient part, so it cannot affect anything there. At 0.05 it does matter along the gradient direction (15-60% of the gradient part, depending on the setting), but it stays close to zero along the Gauss-Newton direction (under 0.5% on CIFAR-100 and the ViT; supplement: exp12b1/stratified_summary.csv). That means curvature has exactly one move available in this protocol: switch from the gradient step to the Gauss-Newton direction at the largest radius. It can never choose a smaller gradient step, because the radius grid jumps by 5-10x and the Gauss-Newton direction at full radius wins on the linear term alone. So it isn't surprising that the gradient difference between batches dominates the error, or that swapping the GGN for the positive Hessian never changes the pick. In fact every selector picks the biggest radius in all 720 tasks per setting (exp12a/action_frequencies.csv), so the curvature model never changes a step length at all, and the one time it changes the direction (CIFAR-10, epoch 160) it is wrong in all 240 tasks. For comparison, the Gauss-Newton solve itself (Eq. 22, before normalisation) proposes steps of length roughly 3-27 in parameter norm, the same unit as the radii, at eight of the nine setting/epoch combinations (0.1 at CIFAR-10 epoch 160; endpoint_step_norm in the unit records, averaged over runs), so that direction is only ever tested at under 2% of its own scale on average. The paper does say "under the fixed equal-radius protocol" in Sec. 1 and 6.2, but the abstract and conclusion drop that. I think this is the main issue: the headline results are mostly a consequence of the step sizes, and fixing it requires a different experiment.
2. The results are averages that hide where the claims stop holding.
I checked the supplementary records, which are broken down by epoch, and they tell a different story from the main text. The "curvature can change the decision" result (equal-linear challenge) only happens at epoch 0, i.e. at initialisation: the GGN and positive-Hessian selectors disagree in 840/1200 tasks at epoch 0 and in 0/2400 tasks at epochs 80 and 160 (exp13a/task_metrics.csv). At the trained checkpoints both quadratic selectors are oracle-compatible in only 80 of 1920 held-out tasks (4%), while the gradient-only selector is compatible in 1450 of 1920 (76%) (exp13a/task_metrics.csv). That undercuts the "secondary targets favor the GGN" claim: the GGN still beats the positive Hessian, but the selector with no curvature at all beats both. Same thing with Table 1: the compatibility numbers average a regime where everything is 100% compatible (epoch 80, all settings) with epochs 0 and 160, where it ranges from 0% to 92% (0% for the GGN selector on CIFAR-10 at epoch 160) (exp12a/stratified_summary.csv), so the paper's "reliable" label (defined in Sec. 5, applied in Table 1 and App. C) is a property of the average, not of any actual checkpoint. Also, the full signed-Hessian quadratic picks the same action as the GGN in every single task (3600/3600 equal-linear, 100% on the original tasks), so the "more faithful" reference the paper compares against is the odd one out, and "curvature fidelity" here really means fidelity to one particular projection/truncation choice. Bottom line: the abstract's claims (matching first-order decrease "makes curvature differences affect selection", secondary targets "favor the GGN") only hold at epoch 0, and the paper's own data show it.
Questions for authors
-
Step sizes. Is there any case in your data where the curvature term changed a selection for the better? From the supplementary records it looks like every selector picks the largest radius (0.05) in every task, and the one time Q_G picks a different direction than the gradient-only selector (CIFAR-10, epoch 160) it is wrong in all 240 tasks. Can you report results at step lengths where the quadratic term is at least comparable to the linear term, e.g. the damped GGN step at a few fractions of its own length (roughly 3-27 in parameter norm), or a radius grid that goes up to around 1?
-
Per-epoch breakdown. Please report Tables 1 and 3 split by epoch (0/80/160). The supplementary strata show the equal-linear disagreement is 840/1200 tasks at epoch 0 and 0/2400 at epochs 80 and 160, and that selector compatibility is 100% at epoch 80 in all three settings but 0-21% at epoch 0 on CIFAR-10/100 and at epoch 160 on CIFAR-10 and the ViT. How should the abstract's claims be read given this, and why is epoch 0 (initialisation) the registered primary stratum in the first place?
-
The reference operator. The full signed-Hessian quadratic agrees with the GGN in every task (3600/3600 equal-linear, 100% on the original tasks), so the projected-then-positive Hessian is the odd one out. Why is it the right "more faithful" reference? Did you compare the construction-batch prediction error of Q_Hplus and Q_G directly, i.e. |dL_c - Q| for each? Is Q_Hplus actually a better predictor anywhere?
Response to Reviewer vq5H
1. Step sizes and curvature-driven gains. The original radii measure local behavior; the added experiments test direction and step-length changes over a wider range. We reuse the original six CIFAR-10 ResNet runs, fixing two unit directions, m=192 and damping 0.01, with eight radii {0.001,0.003,0.01,0.03,0.05,0.1,0.3,1} and no-op. The table gives actual loss changes on a disjoint 256-example utility batch. At initialization, GGN changes direction at radius 1, achieves descent in 6/6 runs and reaches the measured menu minimum; GGN−Linear is −0.058206 (descriptive paired-run 95% CI [−0.109439, −0.027837]). At epoch 160, GGN chooses radii 0.05/0.1/0.3 in 4/1/1 runs, while Linear selects 1 throughout. Thus curvature changes direction at initialization and also reduces step length after training. The positive trained-epoch loss changes identify the remaining overshoot relative to no-op. To test the solve's own scale, a separate menu uses r=κ‖s_G,raw‖ with κ={1/32,1/8,1/2,1}; mean raw norms are 3.43/6.84/0.102 at epochs 0/80/160. At κ=1, negative-gradient quadratic/linear magnitude ratios are 2.61/16.55/2.01. On this menu at initialization, mean loss changes are +1.705128 for GGN and −0.009773 for H+, showing how the curvature reference affects larger steps. These are additional reply analyses; each menu has its own oracle.
2. Epoch breakdown and registration. Table 9 (p. 21) reports epochwise transfer shares and original-menu compatibility for all three settings; CIFAR-10 ResNet shares at epochs 0/80/160 are 93.88/97.75/89.06%. On that original menu, epoch-80 compatibility is 100% in all settings. The additional table below gives CIFAR-10 ResNet equal-linear results at m=192, η=0.25: each entry is oracle-compatible count; mean regret ×10⁶. C is construction and U comprises four disjoint utility targets per run. The full signed Hessian matches GGN in every cell. The trained-epoch contrast has a concrete explanation: at epoch 160, GGN−Linear construction loss is −0.000572, whereas utility loss is +0.003195; the transfer difference +0.003767 accounts for the reversal. Matching construction slopes leaves batch-gradient and higher-order differences in the utility-loss gap. Linear breaks its nonzero score tie using the preregistered smaller-radius rule. Initialization was chosen after a score-only feasibility check found GGN/H+ disagreement in 5/6 runs at each η, compared with 0/6 at trained epochs. The operators and original-action losses already existed; the new equal-linear finite losses were measured after registration. Across configurations, the 840/1,200 initialization disagreements are 168/240 distinct decisions evaluated on five targets; trained disagreement is 0/2,400. Trained utility compatibility is 1,450/1,920 for Linear and 80/1,920 for each quadratic. These are repeated evaluations, with six training runs as inference units. The primary H+−GGN construction-regret difference is 0.790×10⁻⁶, 95% CI [−0.373, 2.513]×10⁻⁶; the favorable secondary utility comparison occurs at initialization (Table 3, p. 8; Appendix C.12, p. 20).
3. Reference operator and prediction accuracy. The reference H+,res=V[V⊤HcV]+V⊤ takes the positive part after projection, placing the measured convex Hessian curvature and GGN in identical coordinates. Replacing GGN with this reference removes the resolved pair-error term; its interaction with the remaining signed curvature and finite-step errors determines absolute prediction accuracy. Direct construction-batch comparisons show lower MAE at epoch 80 by 29.4%/56.3%/39.5% on CIFAR-10 ResNet/CIFAR-100 ResNet/CIFAR-10 ViT. The matched original-menu choices stay identical and oracle-compatible in 24/24 evaluations per setting (Table 10, p. 21): prediction improves while the choice stays oracle-compatible. The full signed Hessian also selects the GGN action in all 3,600 equal-linear evaluations.