acceptodds
Under review as a conference paper at ICLR 2027

From Curvature Fidelity to Step Reliability: What GGN Proxies Capture and Miss

Abstract

Curvature-aware optimizers train neural networks by using a local loss model to guide the direction and size of parameter updates. They often replace the Hessian with the generalized Gauss–Newton (GGN) matrix, a positive-semidefinite surrogate that supports practical second-order computation. But when does a more faithful curvature model lead to a more reliable update, especially when the update is evaluated on a different batch? We propose a diagnostic framework linking curvature comparisons to predicted and realized loss changes. Within a common measured subspace, operator-pair geometry distinguishes differences in curvature directions from differences in normalized spectral shape. An exact error accounting then separates the effect of replacing the resolved GGN from the remaining Hessian contribution, finite-step nonlinearity, and evaluation-batch transfer. Automatic-differentiation products and dense reference checks support these measurements. Across independent ResNet and vision-transformer image-classification cohorts, transfer between batches dominates the measured absolute-error budget, largely through gradient differences. The GGN and resolved positive-Hessian models select identical updates from the original candidates; matching their predicted first-order decrease makes curvature differences affect selection. The primary loss comparison remains inconclusive, while secondary evaluation targets favor the GGN selector. These results explain why curvature fidelity, influence on step selection, and realized loss reduction must be assessed together, providing a basis for diagnosing which part of a local model matters for an update.

Primary area:
learning theory
Keywords:
Curvature Approximation, Generalized Gauss–Newton, Hessian Geometry, Stochastic Optimization, Loss-Model Error Decomposition, Numerical Reliability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_vq5HOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks a basic question about second-order optimisers: if one builds a local quadratic model of the loss on one mini-batch and take a step, does a more faithful curvature matrix give a better step when measured on a different batch? The proxy under study is the generalised Gauss-Newton (GGN) matrix, compared against the positive part of the Hessian projected into the same Krylov subspace.

The method has three parts: 1. A geometry comparison of the two matrices in a shared subspace, which splits their difference into eigenvector misalignment and a difference in normalised spectrum, and points out that these scale-free scores don't tell the prediction error for a specific step. 2. An exact four-term accounting that splits the gap between the GGN prediction and the actual loss change into: the GGN vs positive-Hessian swap, the rest of the Hessian, a finite-step residual, and batch transfer (further split into a gradient-difference part and the rest). 3. Decision tests: seven fixed candidate steps (no-op, plus a negative-gradient and a damped Gauss-Newton direction at three radii each) are ranked by GGN, positive-Hessian and gradient-only selectors and compared to an oracle on held-out batches, plus an "equal-linear" intervention where candidates are rescaled to the same first-order decrease so only curvature can separate them.

Experiments use ResNet20 without normalisation on CIFAR-10/100 and a 262k-parameter ViT on CIFAR-10, six training runs per setting, checkpoints at epochs 0, 80 and 160. The authors report that batch transfer makes up 94-99% of the absolute error budget and is mostly the gradient differing between batches; that the GGN and positive-Hessian selectors pick the same action in every task; that under the equal-linear intervention they disagree in 5/6 runs, with the primary regret comparison inconclusive and a secondary held-out comparison favouring the GGN; and that finite-difference Hessian products inflate the misalignment score about 20x compared to exact autodiff. The paper presents this as a diagnostic protocol, not a new optimiser.

Critical strengths

1. The error accounting is a good tool and the numerics are solid.

The four-term split is exact and each term can be measured directly with Hessian/GGN-vector products plus two loss evaluations, so one can actually see where a bad prediction comes from. The matrix-vector products are checked against dense matrices on small models (App. B, errors around 1e-16), and the finite-difference vs autodiff comparison is a good practical warning - the misalignment score comes out about 20x too big with finite differences. The observation that scale-free geometry scores can't predict the error of a specific step (the A = 2B example in Sec. 3) is simple and correct. This matters for my assessment because the tool is independent of the conclusions drawn here - the tool itself produces real answers.

2. The supplementary material is very thorough and the authors are upfront about limitations.

Per-task records for all nine experiments, protocols with numerical gates, registration files with timestamps, and a README that says what is reproducible. The null primary result is reported as null. I give credit for this because the per-epoch and per-radius numbers in my weaknesses come from these files. It also means the authors can put together the stratified tables for the rebuttal easily. It doesn't change the protocol problem though.

Critical weaknesses

1. The test setup is too small to see curvature effects.

All candidate steps are fixed lengths (0.001, 0.01, 0.05 in parameter norm). At the two smaller radii the curvature part of the quadratic model is at most 12% of the gradient part, so it cannot affect anything there. At 0.05 it does matter along the gradient direction (15-60% of the gradient part, depending on the setting), but it stays close to zero along the Gauss-Newton direction (under 0.5% on CIFAR-100 and the ViT; supplement: exp12b1/stratified_summary.csv). That means curvature has exactly one move available in this protocol: switch from the gradient step to the Gauss-Newton direction at the largest radius. It can never choose a smaller gradient step, because the radius grid jumps by 5-10x and the Gauss-Newton direction at full radius wins on the linear term alone. So it isn't surprising that the gradient difference between batches dominates the error, or that swapping the GGN for the positive Hessian never changes the pick. In fact every selector picks the biggest radius in all 720 tasks per setting (exp12a/action_frequencies.csv), so the curvature model never changes a step length at all, and the one time it changes the direction (CIFAR-10, epoch 160) it is wrong in all 240 tasks. For comparison, the Gauss-Newton solve itself (Eq. 22, before normalisation) proposes steps of length roughly 3-27 in parameter norm, the same unit as the radii, at eight of the nine setting/epoch combinations (0.1 at CIFAR-10 epoch 160; endpoint_step_norm in the unit records, averaged over runs), so that direction is only ever tested at under 2% of its own scale on average. The paper does say "under the fixed equal-radius protocol" in Sec. 1 and 6.2, but the abstract and conclusion drop that. I think this is the main issue: the headline results are mostly a consequence of the step sizes, and fixing it requires a different experiment.

2. The results are averages that hide where the claims stop holding.

I checked the supplementary records, which are broken down by epoch, and they tell a different story from the main text. The "curvature can change the decision" result (equal-linear challenge) only happens at epoch 0, i.e. at initialisation: the GGN and positive-Hessian selectors disagree in 840/1200 tasks at epoch 0 and in 0/2400 tasks at epochs 80 and 160 (exp13a/task_metrics.csv). At the trained checkpoints both quadratic selectors are oracle-compatible in only 80 of 1920 held-out tasks (4%), while the gradient-only selector is compatible in 1450 of 1920 (76%) (exp13a/task_metrics.csv). That undercuts the "secondary targets favor the GGN" claim: the GGN still beats the positive Hessian, but the selector with no curvature at all beats both. Same thing with Table 1: the compatibility numbers average a regime where everything is 100% compatible (epoch 80, all settings) with epochs 0 and 160, where it ranges from 0% to 92% (0% for the GGN selector on CIFAR-10 at epoch 160) (exp12a/stratified_summary.csv), so the paper's "reliable" label (defined in Sec. 5, applied in Table 1 and App. C) is a property of the average, not of any actual checkpoint. Also, the full signed-Hessian quadratic picks the same action as the GGN in every single task (3600/3600 equal-linear, 100% on the original tasks), so the "more faithful" reference the paper compares against is the odd one out, and "curvature fidelity" here really means fidelity to one particular projection/truncation choice. Bottom line: the abstract's claims (matching first-order decrease "makes curvature differences affect selection", secondary targets "favor the GGN") only hold at epoch 0, and the paper's own data show it.

Questions for authors
  1. Step sizes. Is there any case in your data where the curvature term changed a selection for the better? From the supplementary records it looks like every selector picks the largest radius (0.05) in every task, and the one time Q_G picks a different direction than the gradient-only selector (CIFAR-10, epoch 160) it is wrong in all 240 tasks. Can you report results at step lengths where the quadratic term is at least comparable to the linear term, e.g. the damped GGN step at a few fractions of its own length (roughly 3-27 in parameter norm), or a radius grid that goes up to around 1?

  2. Per-epoch breakdown. Please report Tables 1 and 3 split by epoch (0/80/160). The supplementary strata show the equal-linear disagreement is 840/1200 tasks at epoch 0 and 0/2400 at epochs 80 and 160, and that selector compatibility is 100% at epoch 80 in all three settings but 0-21% at epoch 0 on CIFAR-10/100 and at epoch 160 on CIFAR-10 and the ViT. How should the abstract's claims be read given this, and why is epoch 0 (initialisation) the registered primary stratum in the first place?

  3. The reference operator. The full signed-Hessian quadratic agrees with the GGN in every task (3600/3600 equal-linear, 100% on the original tasks), so the projected-then-positive Hessian is the odd one out. Why is it the right "more faithful" reference? Did you compare the construction-batch prediction error of Q_Hplus and Q_G directly, i.e. |dL_c - Q| for each? Is Q_Hplus actually a better predictor anywhere?

1 comment
AuthorsOpenReview ↗

Response to Reviewer vq5H

1. Step sizes and curvature-driven gains. The original radii measure local behavior; the added experiments test direction and step-length changes over a wider range. We reuse the original six CIFAR-10 ResNet runs, fixing two unit directions, m=192 and damping 0.01, with eight radii {0.001,0.003,0.01,0.03,0.05,0.1,0.3,1} and no-op. The table gives actual loss changes on a disjoint 256-example utility batch. At initialization, GGN changes direction at radius 1, achieves descent in 6/6 runs and reaches the measured menu minimum; GGN−Linear is −0.058206 (descriptive paired-run 95% CI [−0.109439, −0.027837]). At epoch 160, GGN chooses radii 0.05/0.1/0.3 in 4/1/1 runs, while Linear selects 1 throughout. Thus curvature changes direction at initialization and also reduces step length after training. The positive trained-epoch loss changes identify the remaining overshoot relative to no-op. To test the solve's own scale, a separate menu uses r=κ‖s_G,raw‖ with κ={1/32,1/8,1/2,1}; mean raw norms are 3.43/6.84/0.102 at epochs 0/80/160. At κ=1, negative-gradient quadratic/linear magnitude ratios are 2.61/16.55/2.01. On this menu at initialization, mean loss changes are +1.705128 for GGN and −0.009773 for H+, showing how the curvature reference affects larger steps. These are additional reply analyses; each menu has its own oracle.

EpochLinear ΔLGGN ΔLH+ ΔL
0+0.035890−0.022316−0.012556
80+1.441897+0.013609+0.013609
160+20.309421+0.016791+0.021134

2. Epoch breakdown and registration. Table 9 (p. 21) reports epochwise transfer shares and original-menu compatibility for all three settings; CIFAR-10 ResNet shares at epochs 0/80/160 are 93.88/97.75/89.06%. On that original menu, epoch-80 compatibility is 100% in all settings. The additional table below gives CIFAR-10 ResNet equal-linear results at m=192, η=0.25: each entry is oracle-compatible count; mean regret ×10⁶. C is construction and U comprises four disjoint utility targets per run. The full signed Hessian matches GGN in every cell. The trained-epoch contrast has a concrete explanation: at epoch 160, GGN−Linear construction loss is −0.000572, whereas utility loss is +0.003195; the transfer difference +0.003767 accounts for the reversal. Matching construction slopes leaves batch-gradient and higher-order differences in the utility-loss gap. Linear breaks its nonzero score tie using the preregistered smaller-radius rule. Initialization was chosen after a score-only feasibility check found GGN/H+ disagreement in 5/6 runs at each η, compared with 0/6 at trained epochs. The operators and original-action losses already existed; the new equal-linear finite losses were measured after registration. Across configurations, the 840/1,200 initialization disagreements are 168/240 distinct decisions evaluated on five targets; trained disagreement is 0/2,400. Trained utility compatibility is 1,450/1,920 for Linear and 80/1,920 for each quadratic. These are repeated evaluations, with six training runs as inference units. The primary H+−GGN construction-regret difference is 0.790×10⁻⁶, 95% CI [−0.373, 2.513]×10⁻⁶; the favorable secondary utility comparison occurs at initialization (Table 3, p. 8; Appendix C.12, p. 20).

Epoch/targetLinear: count; regretGGN/full H: count; regretH+: count; regret
0/C3/6; 2.6386/6; 0.1944/6; 0.983
0/U5/24; 131.24119/24; 11.6207/24; 111.277
80/C2/6; 27.7614/6; 8.0574/6; 8.057
80/U24/24; 00/24; 2102.8430/24; 2102.843
160/C0/6; 572.1306/6; 06/6; 0
160/U13/24; 1488.5522/24; 4683.3752/24; 4683.375

3. Reference operator and prediction accuracy. The reference H+,res=V[V⊤HcV]+V⊤H_{+,\mathrm{res}}=V[V^\top H_cV]_+V^\top takes the positive part after projection, placing the measured convex Hessian curvature and GGN in identical coordinates. Replacing GGN with this reference removes the resolved pair-error term; its interaction with the remaining signed curvature and finite-step errors determines absolute prediction accuracy. Direct construction-batch comparisons show lower MAE at epoch 80 by 29.4%/56.3%/39.5% on CIFAR-10 ResNet/CIFAR-100 ResNet/CIFAR-10 ViT. The matched original-menu choices stay identical and oracle-compatible in 24/24 evaluations per setting (Table 10, p. 21): prediction improves while the choice stays oracle-compatible. The full signed Hessian also selects the GGN action in all 3,600 equal-linear evaluations.

Reviewer_Ukk9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper constructs a diagnostic framework represented by curvature. (i) Operator-to-geometry: Compare the positive part of GGN and projected Hessian in a common Krylov subspace, and decompose the coupling gap into an alignment term D^geo and a spectral shape term D^spec. (ii) Exact four-term identity (Equation 16): Decompose the gap between predicted and actual loss changes into resolved pairs, unresolved signed Hessian terms, finite-step cubic residuals, and batch transfer from construction batch to evaluation batch. (iii) Experiments use ResNet20 (CIFAR-10/100, without normalization) and ViT (CIFAR-10) with 261,000 parameters, each trained 6 times, with fixed candidate step radii of {0.001, 0.01, 0.05}. It is found that batch transfer accounts for 93–98% of the absolute error budget, mainly from first-order gradient differences (Tables 1–2). Among the 720 tasks in each queue, GGN and the positive Hessian selector select exactly the same actions. Only in the constructed "equal-linear" challenge do the two diverge (5 out of 6 runs), where the primary regret results are uncertain and the secondary utility tends to favor GGN (Table 3).

Critical strengths
  1. Valuable Problem Perspective: The question is well-posed. Moving beyond simple matrix similarity, the authors investigate whether differences in curvature approximations actually influence final action selection, making the "pair-correction leverage" framework genuinely valuable.
  2. Solid Measurement Design: The methodology is rigorous, featuring matrix-free automatic-differentiation products, validation against dense matrices/small networks, and four-term decomposition closure errors down to 10^-17. Training runs and repeated measurements are also meticulously distinguished.
  3. Candid Reporting of Negative Results: The authors demonstrate commendable transparency by openly discussing insignificant main regret effects and the failure of several prespecified early tests (Sec. 6.1, App. C).
  4. Insightful Practical Cautions: The paper highlights crucial technical caveats: projecting-then-positive differs from positive-then-project (Eq. 3); normalized geometry obscures absolute scale (Eq. 10); and finite-difference D^geo is roughly 20x larger than automatic-differentiation values, exposing high sensitivity to numerical methods.
Critical weaknesses
  1. Confounding Factors in Radius-Dependent Error Accounting: The dominance of batch transfer (EbatchE_{batch}) appears to be an artifact of the small step radii (r∈{0.001,0.01,0.05}r \in \{0.001, 0.01, 0.05\}). Because the transfer term is first-order (O(r)O(r)) while curvature terms are second-order (O(r2)O(r^2)), the first-order term inevitably dominates. The paper lacks error-vs-radius curves to demonstrate how error proportions shift as the radius increases. 
  2. Limited Practical Differentiation in Action Selection: Under the original protocol, GGN and positive Hessian selectors yield identical actions across all tasks (720/720). Surprisingly, a pure linear selector outperforms QGQ_G on C10 ResNet (54.2% vs. 40.3%, Table 3), indicating that curvature can degrade decision quality—a counter-intuitive finding left undiscussed. Curvature effects only surface under a heavily constrained stress test ("equal-linear descent") with extremely tiny regret differences (10−610^{-6} to 10−410^{-4})and in-built GGN bias. 
  3. High Mathematical and Symbolic Barrier: Sections 3 and 4 introduce an excessively dense notation system (ρA,Zα,β,Dα,βgeo\rho_A, Z_{\alpha,\beta}, D_{\alpha,\beta}^{geo}, etc.), creating a severe cognitive load. The paper would benefit significantly from intuitive graphical illustrations or conceptual summaries to bridge abstract operator geometry with practical optimization utility.
Questions for authors

1.Can you draw the proportion of four error terms as a function of radius (such as r from 1e-3 to 1)? At what radius will the E-batch no longer dominate? 2.Can you put the proportions of E2 pairs, E2 unresolved, R3 and E2 batches together in Table 1? 3.Please report the specific values of D ^ geo and D ^ spec, and whether they are related to E-pair, selector consistency, or regret. 4.Why is the linear selector better than QG in the CIFAR-10 ResNet queue (54.2% vs. 40.3%)? Does this mean that the curvature model is actually damaging decision-making there? 5.Can at least one actual agent (such as K-FAC or diagonal GGN) be added to the same decomposition set?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Ukk9

1. Error sources across radii. The radius sweep shows where batch transfer gives way to finite-step error. All 18 CIFAR-100 checkpoints pass numerical checks for radii up to 1. At epoch 160, mean transfer shares first fall below half at the tested radius 0.5: 35.5% along the negative gradient and 49.0% along GGN. The finite-step residual first becomes the largest mean component at radii 0.5 and 1, respectively (§7.3, p. 8; Fig. 3, p. 9). These crossings locate the transition on the measured grid; the registered statistical dominance test additionally accounts for uncertainty across runs.

2. Complete four-source budget. Table 11 (p. 21) reports all four components together. For CIFAR-10 ResNet, their absolute shares are 1.350% resolved pair, 0.805% unresolved signed curvature, 4.283% finite-step residual, and 93.562% batch transfer. The signed allocation also measures how each component reinforces or cancels the net error. Transfer contributes 70.81% of the squared-error allocation, with a descriptive 95% interval [37.11%, 95.33%].

3. Geometry values and their associations. Additional analysis gives six-run means of (Dgeo,Dspec)(D^{\rm geo},D^{\rm spec}) at m=192m=192, α=β=0.5\alpha=\beta=0.5. In epoch order 0/80/160, these are CIFAR-10 ResNet: (0.4676,1.1739)/(0.0622,0.1513)/(0.0420,0.4236); CIFAR-100 ResNet: (0.7227,0.1122)/(0.0372,0.0076)/(0.0147,0.0307); CIFAR-10 ViT: (0.4013,0.8465)/(0.1250,0.0475)/(0.0843,0.0470). Pair error combines this normalized geometry with scale and exposure along the step (§3, pp. 3–4). Its associations therefore vary by direction: for CIFAR-10 ViT at initialization, Spearman correlations between DgeoD^{\rm geo} and ∣Epair∣/r2|E_{\rm pair}|/r^2 are −0.943 [−1.000,−0.515] along the negative gradient and +0.657 [0.000,1.000] along GGN. At epoch 160, correlations with mean GGN line-search regret are +0.880 [0.492,1.000] for CIFAR-100 ResNet and +0.086 [−0.920,1.000] for CIFAR-10 ResNet. These are six-run descriptive 95% intervals; all estimable DspecD^{\rm spec}–regret intervals cross zero. Original-task GGN/Hessian agreement is constantly one, giving an undefined correlation. The complementary predictive test in Appendix C.9 (p. 19) shows increased loss-gap MAE when geometry is added to directional curvature.

4. Why Linear wins and how we check construction bias. In the original 720 CIFAR-10 ResNet tasks, define every difference as GGN choice minus Linear choice. The linear difference +0.00980 and quadratic difference −0.02073 give a predicted advantage −0.01093. Finite-step error +0.01115 and transfer +0.00226, together with the remaining signed terms, reverse it to actual loss +0.00232. This explains the lower GGN compatibility: 480 tasks retain the same action, 100 lose compatibility, and 140 are incompatible under both selectors. Gains within the last group partly offset the losses, yielding compatibility 54.2% versus 40.3% and paired mean regret difference +0.0023 [−0.0021,+0.0056]. To check the GGN-generated candidate's role in equal-linear selection, an additional CIFAR-10 control offers negative-gradient, GGN-generated and positive-Hessian-generated directions plus no-op. Both curvature generators use damping 0.01, Krylov budget 32 and radius cap 0.05. Across epochs 0/80/160, external line-search regrets average 0.000270 (Linear), 0.002028 (GGN) and 0.007473 (positive Hessian). Each quadratic selects its own generated direction; Linear's tie rule selects the negative-gradient action. This control uses m=32m=32; the original m=192m=192 primary regret interval spans zero.

5. Diagonal GGN and K-FAC. Exact parameter-diagonal GGN and convolutional K-FAC enter the same decomposition through an explicit proxy-replacement term. On the bounded CIFAR-100 grid, both select negative-gradient radius 1, raising held-out loss in all 24 evaluations at each trained epoch (Table 5, p. 18; Appendix C.10, p. 19). After shared gradient/direction computation, median added scoring costs are 0.129 s for direct directional GGN and 0.722 s for exact diagonal GGN (C.7, p. 18).

6. Presentation. Figure 1 (p. 2) now follows the reader's decision process: compare curvature, locate the signed prediction errors, then assess the selected step. Section 5 (pp. 5–6) uses a worked example to distinguish the correction that changes the preferred candidate from the error contrast that makes the new candidate improve held-out loss.

Reviewer_o6JZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses an important question: whether more faithful curvature approximations lead to better update decisions. The diagnostic framework is clear, the numerical validation is careful, and the reporting is transparent. However, the methodological novelty is modest, the geometric metrics lack demonstrated incremental predictive value, and the primary comparison does not establish a decision-quality advantage. I therefore lean toward rejection despite the paper’s useful diagnostic insights.

Critical strengths
  1. The paper clearly distinguishes operator similarity, prediction accuracy, and update quality.
  2. The four-term decomposition separates curvature approximation, unresolved curvature, finite-step effects, and batch transfer.
  3. Dense-reference checks, subspace sensitivity analyses, and run-level statistical aggregation strengthen the measurements.
Critical weaknesses
  1. The error decomposition is algebraic, and the geometric analysis builds on established quantities. Its added value over direct directional-curvature measurements remains unclear.
  2. Gradient-transfer error is first order in step size, whereas curvature corrections are second order. Stronger controls are needed to establish how broadly transfer dominance holds.
  3. The original selectors always agree, and the primary regret comparison after intervention remains inconclusive. Better update decisions have therefore not been demonstrated.
  4. Two directions, three radii, and CIFAR-based experiments provide limited evidence of robustness across candidate sets and optimization settings.
Questions for authors

See Weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer o6JZ

1. Added value beyond directional curvature. Direct directional curvature gives the quadratic score of a fixed step. Our framework explains how that score changes when an operator is replaced and how its error affects competing candidates (§§3–5, pp. 3–6). Common-subspace geometry separates alignment and normalized spectral shape; restoring scale and directional exposure connects these quantities to step-specific error. We tested its predictive increment by leaving out one complete run and adding geometry to a baseline with state, radius, gradient inner products and directional quadratics. MAE changes were +0.000361 [−0.000065,+0.001155] on CIFAR-10 ResNet, +0.000117 [+0.000052,+0.000179] on CIFAR-100 ResNet, and +0.000004 [−0.000084,+0.000112] on CIFAR-10 ViT (descriptive 95% intervals; six runs each). These results place geometry in a structural explanatory role, with directional curvature as the practical scoring baseline; Appendix C.9 (p. 19) gives the separate wider-grid analysis. Independent-batch debiasing was studied by Tatzel et al., ICLR 2025. Our additional contribution is to relate fixed-candidate gradient, curvature and joint interventions to signed error differences and independently evaluated selection quality.

2. Radius and transfer controls. Small-radius first-order dominance follows the expected local scaling: Ebatch(rd)=r(ge−gc)⊤d+O(r2)E_{\rm batch}(rd)=r(g_e-g_c)^\top d+O(r^2) when directional gradients differ (Eq. 19, p. 5). We therefore measure the finite-radius range over which this term controls the budget. In additional epoch-160 tests, increasing radius from 0.001 to 1 lowers mean absolute transfer shares from 99.12% to 12.62% for CIFAR-10/AdamW and from 99.64% to 14.37% for SVHN/SGD. Shares use the sum of four mean absolute components as denominator, computed within each run before averaging six runs. At radius 1, finite-step residuals become largest. Transfer grows in absolute magnitude, while other errors grow faster. The curves thus locate the transition from batch-transfer error to finite-step model error.

3. Improvement in selected updates. The original equal-linear primary construction-regret difference is 0.790×10⁻⁶, with 95% CI [−0.373,2.513]×10⁻⁶ (Table 3, p. 8). To test an actionable consequence of the diagnosis, the separate preregistered CIFAR-10/SGD experiment fixes candidates and curvature, changes only the scoring gradient using disjoint calibration batches, and seals scores and choices before target evaluation (§7.4, p. 9; Table 8, p. 20). At epoch 160, mean regret falls from 0.017773 to 0.007761, with all six new runs improving; the paired difference is −0.010012 [−0.012277,−0.007746]. Prediction MAE rises, while the same calibration gradient with linear scoring gives regret 0.016873. This isolates an improvement in the choice itself and the value of retaining curvature after gradient calibration. The descriptive epoch-80 check clarifies the stage dependence: the original choice already attains the oracle minimum in all 24 evaluations; calibration changes one choice, giving a mean regret difference of +0.000617 [0,0.001851].

4. Dataset, optimizer and candidate robustness. Additional experiments train six new ResNet20 runs for each of SVHN/SGD and CIFAR-10/AdamW and evaluate epochs 80 and 160, with fixed candidates scored before target losses. Alongside the original seven actions, a 25-action menu supplies negative-gradient, damped GGN and damped positive-Hessian directions at eight radii from 0.001 to 1, plus no-op. A separate retrospective multi-direction analysis uses ten directions and four 61-action menus on a development cohort. At epoch 160 on the original menu, calibrated-minus-original regret is −0.038175 [−0.056531,−0.016601] for AdamW and −0.003639 [−0.008747,+0.000521] for SVHN (prespecified 97.5% paired intervals). AdamW's mean selected loss change moves from +0.045872 to +0.007698, showing smaller overshoot. SVHN's interval crosses zero, and its epoch-80 original-menu mean regret increases by 0.002729. On the 25-action menu, calibration lowers regret in both settings at both epochs.

Reviewer_m2scOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates when the fidelity of generalized Gauss–Newton (GGN) curvature approximations translates into reliable parameter updates. It introduces an operator geometry analysis and an exact decomposition of loss prediction error into resolved curvature mismatch, unresolved curvature, finite-step nonlinearity, and cross-batch transfer. Experiments on ResNet and vision transformer models show that cross-batch gradient differences dominate the measured error budget. Curvature differences affect update selection when first-order decreases are matched, although the primary comparison does not establish a significant advantage for either selector.

Critical strengths

The paper addresses an important question: when does curvature approximation fidelity translate into reliable parameter updates? It proposes a diagnostic framework for decomposing loss prediction errors and finds that cross-batch gradient differences dominate the measured error budget in the studied settings. The work combines theoretical analysis with controlled experiments and is clearly structured.

Critical weaknesses
  1. The proposed framework primarily provides post hoc diagnostics, and its additional practical value for guiding optimization remains unclear.
  2. The theoretical analysis largely consists of algebraic identities decomposing prediction error. A component’s large share of the absolute-error budget does not necessarily establish that the corresponding mechanism is decisive for the final prediction error or incorrect step selection, particularly when error components can cancel.
Questions for authors

Does the dominance of cross-batch gradient differences reveal a general optimization mechanism, or does it primarily reflect the expected scaling of first-order versus second-order terms under small steps and finite batch sizes?

1 comment
AuthorsOpenReview ↗

Response to Reviewer m2sc

1. Practical value of the diagnosis. The diagnosis led to a gradient-only calibration test: keep the candidates and curvature fixed, and replace the scoring gradient using a disjoint batch. In six new, preregistered CIFAR-10 runs at epoch 160, mean held-out regret falls from 0.017773 to 0.007761, with all six runs improving (paired difference -0.010012, 95% CI [-0.012277,-0.007746]; §7.4, p. 9; Table 8, p. 20). Each run uses four disjoint calibration/evaluation pairs of 256 examples per batch; scores and choices are sealed before evaluation. The seven candidates include no-op. With the same calibrated gradient, linear scoring gives regret 0.016873; retaining the original GGN gives a difference of -0.009111 [-0.016200,-0.002022]. Curvature-only and joint-calibration controls reach 0.019516 and 0.012035. In the descriptive epoch-80 follow-up, the original choice already attains the menu oracle in 24/24 evaluations; calibration changes one choice, adding 0.000617 mean regret. Among these quadratic controls, gradient-only calibration gives the lowest mean regret at epoch 160. Overall MAE rises from 0.017231 to 0.022246, and all-pairs error-contrast MAE rises from 0.025808 to 0.035086, while contrasts anchored at the original winner improve from 0.033889 to 0.026991. Selection depends on the competing actions' contrasts.

2. Cancellation and incorrect step selection. We account for cancellation with Aj=E[ejϵ]/E[ϵ2]A_j=\mathbb E[e_j\epsilon]/\mathbb E[\epsilon^2], where ϵ=∑jej\epsilon=\sum_j e_j and ∑jAj=1\sum_jA_j=1. Negative contributions retain cancellation. Transfer's signed allocations are 70.81%, 94.76% and 98.39% on CIFAR-10 ResNet, CIFAR-100 ResNet and CIFAR-10 ViT, with descriptive 95% intervals [37.11%,95.33%], [92.87%,96.37%] and [97.84%,98.84%] (Table 11, p. 21). The CIFAR-10 interval spans 50%, quantifying the uncertainty beyond its 93.56% absolute share. For original winner aa and challenger bb, define m=q(b)−q(a)m=q(b)-q(a) and D=e(a)−e(b)D=e(a)-e(b). The challenger improves held-out loss exactly when D>mD>m (Eq. 22, p. 5), so common errors cancel in the relevant contrast. In the additional fixed-m=192m=192, seven-action utility analysis, correcting complete measured transfer reduces regret by 0.005373, 0.000890 and 0.000913 in those three settings. Correcting only its first-order part changes regret by -0.005452, +0.001608 and -0.000496. For CIFAR-100, the complete-transfer and first-order-only changes have 95% intervals [-0.001449,-0.000332] and [+0.000479,+0.002683]. These are offline mechanism tests using evaluation information; the independent calibration above tests an actionable correction. The CIFAR-100 reversal shows that the remaining transfer terms carry differences that matter for ranking, even when their absolute magnitudes are smaller.

3. First-order scaling and its measured range. Small-radius first-order dominance follows the local expansion. For a fixed unit direction dd and twice-differentiable batch losses, Ebatch(rd)=r(ge−gc)⊤d+12r2d⊤(He−Hc)d+o(r2)E_{\rm batch}(rd)=r(g_e-g_c)^\top d+\tfrac12r^2d^\top(H_e-H_c)d+o(r^2) (Eq. 19, p. 5). A nonzero directional gradient difference leads as r→0r\to0. Our finite-radius measurements establish where this approximation describes the observed error. The original six CIFAR-10 runs and 18 checkpoints are tested at eight radii from 0.001 to 1 and fractions of each raw damped-GGN step norm. At epoch 0, the first-order share within transfer along GGN falls from 98.66% at radius 0.05 to 33.80% at one raw step length; at epoch 80, the negative-gradient share falls from 86.62% to 27.87%. Mean raw lengths are 3.43 and 6.84. In the separate four-source CIFAR-100 budget at epoch 160, total transfer's mean share first falls below half at tested radius 0.5 (Fig. 3, p. 9). With a direction fixed independently of both sampled batches, IID sampling predicts variance proportional to 1/Bc+1/Be1/B_c+1/B_e. Enlarging both batches from 64 to 1024 gives RMS ratios 0.251, 0.253 and 0.265 at epochs 0/80/160, close to 0.25. Thus batch size controls absolute sampling noise, while radius and training stage determine its balance with curvature and finite-step effects.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response to Reviewers

Preregistered selection gains. Thank you for the detailed reviews. In a separate preregistered CIFAR-10 experiment, we kept curvature and candidates fixed, replaced only the scoring gradient with one from a disjoint batch, and froze the scores and choices before evaluation. At epoch 160, all six new runs improved. Mean held-out regret fell from 0.01777 to 0.00776 (paired difference −0.01001, 95% CI [−0.01228, −0.00775]), even as prediction MAE rose. In the descriptive epoch-80 follow-up, the original choice already reaches the menu minimum in 24/24 evaluations; calibration changes one choice and adds 0.000617 mean regret (§7.4, p. 9 and Table 8, p. 20).

Two conditions for better choices. Section 5 separates a change in scoring preference from an improvement in held-out loss. In the worked CIFAR-100 case, gradient calibration changes the choice on both grids. Relative to the original choice, the new step raises held-out loss by 0.0163 on the original grid and lowers it by 0.3793 on the grid capped at radius 1 (§5, pp. 5–6). On that wider grid, all 18 checkpoints pass the numerical checks (§7.3, p. 8). Figure 2 (p. 9) plots the preferred direction across radii, and Fig. 3 (p. 9) shows the corresponding error shares.

Repeated updates and cost. After five updates, two-direction validation beats direct directional GGN in all six runs at both trained epochs (Table 6, p. 19). At a common cumulative L20 update-time cap of 0.35 s, the conservative policy completes more updates and reaches lower mean held-out loss than negative-gradient validation at both epochs (Table 4, p. 10).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.