acceptodds
Under review as a conference paper at ICLR 2027

Revisiting the Dimension Dependence of Zeroth-Order Optimization under Hessian Trace-Norm Conditions

Abstract

Zeroth-order (ZO) optimization is broadly applied to high-dimensional problems where gradients are unavailable or costly, yet classical theory predicts rates that scale with the ambient dimension. Recent work replaces the dimension with a spectral quantity of the Hessian but assumes a fixed Hessian or a curvature envelope on a neighborhood or on the whole space. We show that, for the standard two-point Gaussian estimator, a pointwise Hessian trace-norm condition suffices: the sum of the absolute Hessian eigenvalues is bounded by a constant at every point. Under this condition, we prove convergence to first-order stationary points at rate for general non-convex objectives and, under the Polyak–ojasiewicz (PL) condition with constant , linear convergence with contraction factor . A “Twisted Valley” function whose principal curvature direction sweeps through every coordinate direction satisfies the condition with constants independent of the dimension, whereas the envelope conditions of prior analyses hold on it only with a trace that grows linearly in the dimension. Numerically, on this function and on the forward-only adaptation of a CNN and a Vision Transformer, the loss reached by the Gaussian estimator within a fixed query budget is insensitive to the dimension, while uniform coordinate probing degrades with it.

Primary area:
optimization
Keywords:
zeroth order optimization, dimension dependency

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_xguLOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper analyzes standard two-point Gaussian zeroth-order optimization under a uniform pointwise bound on the Hessian trace norm. It establishes nonconvex stationarity and PL convergence guarantees without requiring a common Hessian envelope, with explicit smoothing and noise terms. A Twisted Valley construction demonstrates a separation between the proposed condition and global envelope assumptions. Synthetic and neural-adaptation experiments illustrate how Gaussian probing behaves as the parameter dimension increases.

Critical strengths
  1. The paper provides a nontrivial extension of effective-dimension analysis beyond common Hessian envelopes. It clarifies that pointwise concentration of curvature can support favorable convergence guarantees even when the dominant curvature directions vary with location.

  2. The Twisted Valley construction makes the theoretical contribution concrete: its trace-norm and PL constants remain independent of dimension, while any global Hessian envelope must have a trace that grows with dimension.

Critical weaknesses
  1. The practical importance of the assumption relaxation could be explained more clearly. The paper would benefit from identifying realistic optimization problems where the uniform trace-norm condition is plausible and a common low-trace envelope is substantially more restrictive.

  2. The neural experiments provide limited validation of this practical connection. The head-only control achieves similar performance, and the selected step sizes exceed the theoretically guaranteed range. Although these limitations are acknowledged, the experiments do not clearly establish the relevance of the newly covered regime to practical training.

  3. The theoretical novelty is relatively focused: the estimator and effective-dimension measure are established, and the PL result follows from a standard argument. The significance of removing the common-envelope requirement could be clearer.

Questions for authors
  1. Clarification of the theoretical comparison: In Section 5.2, the local-envelope trace bound depends on a neighborhood radius that itself depends on the step size and iterate. Under precisely which conditions does the claimed linear growth with dimension hold? How does the comparison change when the step size depends on dimension?

  2. Additional evidence relevant to the empirical assessment: Can the authors provide trajectory-level diagnostics showing that successful ZO runs encounter substantial changes in dominant curvature directions while maintaining controlled Hessian trace norms? The existing nonrotating and head-only controls are useful, but such diagnostics would more directly connect the observed optimization behavior to the distinctive setting covered by the analysis.

2 comments
AuthorsOpenReview ↗

Response to reviewer xguL (1/2)

We address your concerns below.

W1 and Q2: Practical relevance and trajectory-level diagnostics

Thank you for raising this important issue. As you pointed out, the submitted version compared the leading eigenspaces only at two nearby points of one run and did not follow their change along the trajectory. To address this, we conducted an additional experiment that computes the exact Hessian every 500500 queries along one ZO-Gauss run on the adapted CNN and ViT.

For each iterate zz, the table below reports the overlap ∥Uz⊤Ux∥F2/k\Vert U_z^\top U_x\Vert_F^2/k between the leading kk-dimensional eigenspaces of the Hessian at zz and at the final iterate xx after 40004000 queries, where UzU_z and UxU_x are their orthonormal bases.

Queries0010001000200020003000300040004000
CNN, k=5k=50.360.360.720.720.760.760.900.9011
CNN, k=100k=1000.190.190.550.550.590.590.700.7011
ViT, k=5k=50.410.410.830.830.890.890.930.9311
ViT, k=100k=1000.290.290.630.630.700.700.730.7311

These results show that successful ZO-Gauss runs encounter substantial changes in the dominant curvature directions while the trace norm of the Hessian stays bounded. This is precisely the regime that our pointwise condition covers. A common envelope, in contrast, would have to dominate all the directions met along the run. We added these results to Appendix E (Figure 8) and refer to them in Section 6.2.

W2: Head-only control and step sizes in the neural experiments

Head-only control. We agree that the validation in the submitted version was limited. The width sweep shows that the added dimensions do not slow ZO-Gauss but slow ZO-Coord. Since ZO-Gauss reaches within the budget the same loss as without the adapter (r=0r=0), however, the sweep does not show whether ZO-Gauss uses these dimensions.

To test this, we added a sweep that starts the head at the optimum of the head-only model, whose training loss we denote by fLPf_{\mathrm{LP}} (Appendix F.4, Figure 11). Since fLPf_{\mathrm{LP}} is the minimum over the head alone, the loss goes below it only through the adapter. The table below reports the decrease fLP−ff_{\mathrm{LP}}-f of the training loss after 4000040000 queries (median over five seeds), with gradient descent (GD) at the same step size as ZO-Gauss.

d=647d=64714211421245324534517451786458645
CNN, ZO-Gauss0.0230.0230.0540.0540.0590.0590.0620.0620.0570.057
CNN, GD0.0210.0210.0560.0560.0620.0620.0620.0620.0570.057
CNN, ZO-Coord0.0020.0020.0010.0010.0030.0030.0020.0020.0010.001
ViT, ZO-Gauss0.0640.0640.1620.1620.1930.1930.1390.1390.0870.087
ViT, GD0.0600.0600.1620.1620.1990.1990.1510.1510.0910.091
ViT, ZO-Coord0.0600.0600.0410.0410.0060.0060.0010.0010.0010.001

ZO-Gauss lowers the loss below fLPf_{\mathrm{LP}} for every dd, by about as much as GD with the same step size, whereas ZO-Coord hardly lowers it for large dd.

Step sizes. We agree that the selected step sizes exceed 1/(100H)1/(100H). As we remark after the proof of Proposition A.8, the constant 100100 is not optimized and may leave room for improvement. The theory predicts that the step size depends on dd only through HH, and the selected step sizes agree with this. On the Twisted Valley, the tuned step size of ZO-Gauss in Appendix D is 0.30.3 for every dd from 5050 to 5000050000. In the width sweep of Table 1, dd grows thirteenfold from 647647 to 86458645, and the classical step size 1/(dL)1/(dL) would decrease by more than an order of magnitude. The selected step size of ZO-Gauss decreases by at most a factor of three, from 3×10−33\times10^{-3} to 10−310^{-3} on the CNN, and does not change on the ViT.

W3: Theoretical novelty and significance

Thank you for helping us clarify our theoretical contribution. We revised Section 2 and Remark 4.6 to explain the technical contribution of removing the curvature envelope. Beyond quadratics, the Hessian enters the descent inequality at points that depend on the random direction, and bounding it there by its operator norm costs a factor of dd. We explain the two tools by which we avoid this factor, an averaging over the rotations that fix the gradient and a bound along rays based on the Gaussian Poincaré inequality and an integrated Bochner identity.

AuthorsOpenReview ↗

Response to reviewer xguL (2/2)

Q1: Clarification of the theoretical comparison in Section 5.2

We thank the reviewer for this question. Following it, we revised Section 5.2 in two respects.

First, we corrected the description of the envelope conditions. The submitted version described both DPZero and MeZO by a one-sided bound and stated that our condition is implied by both, which holds for DPZero but not for MeZO. The revised Section 5.2 distinguishes the two. DPZero requires a global two-sided envelope, −A⪯∇2f(y)⪯A-A\preceq\nabla^2f(y)\preceq A for every yy. MeZO requires a local one-sided envelope, ∇2f(y)⪯At\nabla^2f(y)\preceq A_t on a ball around xtx_t of radius at least ηd∥∇f(xt)∥\eta d\lVert\nabla f(x_t)\rVert. Proposition 5.2(i) now requires only a symmetric AA, instead of A⪰0A\succeq0, and applies to both.

Second, we added to Section 5.2 the step-size regime in which the linear growth holds. It holds at a step size η\eta independent of dd, as in the analysis of MeZO. The radius ηd∥∇f(xt)∥\eta d\lVert\nabla f(x_t)\rVert then grows linearly in dd, and by Proposition 5.2(i), the trace of the envelope grows linearly as well. If the step size depends on dd, keeping the radius bounded requires η=O(1/d)\eta=\mathcal{O}(1/d). This reduces the decrease per iteration by a factor of dd. In both regimes, the envelope analysis pays a factor of dd on the Twisted Valley, whereas our guarantee holds at h=1/(100H)h=1/(100H) with HH independent of dd.

Reviewer_QGo6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits the dimension dependence of zeroth-order optimization under a pointwise Hessian trace-norm bound. The authors establish convergence guarantees for the standard two-point Gaussian estimator, including an O(H/T)O(H/T) stationarity bound for general nonconvex objectives and linear convergence under the PL condition, up to finite-difference and noise errors. The main theoretical contribution is removing the curvature-envelope assumptions used in prior analyses. Numerical experiments on a synthetic Twisted Valley function and neural network adaptation tasks are presented to illustrate the results.

Critical strengths
  1. The paper provides a technically nontrivial extension of existing zeroth-order convergence analyses, establishing guarantees under a pointwise Hessian trace-norm condition without requiring a common curvature envelope.

  2. The Twisted Valley construction provides an interesting theoretical example that illustrates the distinction between pointwise curvature bounds and existing envelope-based assumptions.

Critical weaknesses
  1. The Hessian trace-norm condition corresponds to the effective dimension ED1 introduced by Yue et al. (2023), and the algorithm is the standard two-point Gaussian estimator. While removing the curvature-envelope assumption represents a meaningful theoretical extension, the paper introduces neither a new optimization method nor a new measure of effective dimension. The contribution is therefore primarily an extension of existing convergence theory, and its broader significance is not sufficiently established.

Yue et al. “Zeroth-order Optimization with Weak Dimension Dependency.” COLT 2023.

  1. The Twisted Valley is a carefully constructed example in which only a few coordinates are active at each point. While it illustrates the theoretical distinction between the assumptions, Appendix D shows nearly identical convergence behavior for ZO-Gauss on the rotating and non-rotating variants. Other sufficiently dense random perturbation methods also exhibit similar dimension-insensitive behavior. These observations leave the practical significance of removing the curvature-envelope assumption largely unclear.

  2. In the CNN and ViT experiments, increasing the number of trainable parameters provides essentially no improvement over the head-only model, suggesting that the observed dimension insensitivity is largely attributable to the additional parameters not being effectively utilized. Furthermore, the experiments do not establish the global Hessian trace-norm assumption, and the empirically selected step sizes fall outside the theoretical regime. Consequently, the connection between the theoretical guarantees and the observed practical performance remains unclear.

Questions for authors
  1. Can the authors further clarify the conceptual significance of the theoretical advance over Yue et al. (2023), beyond removing the curvature-envelope assumption?

  2. Can the authors provide stronger empirical evidence of weak dimension dependence in settings where the additional dimensions meaningfully contribute to optimization, and clarify how the observed behavior relates to the theoretical guarantees?

2 comments
AuthorsOpenReview ↗

Response to reviewer QGo6 (1/2)

We appreciate the reviewer's careful reading and constructive questions. We address your concerns below.

W1 and Q1: Significance of the theoretical advance over Yue et al. (2023)

Thank you for this question, which helped us sharpen the comparison with prior work. We revised Section 2 and Remark 4.6 to state what our analysis adds to Yue et al. (2023).

Our result shows that ED1\mathrm{ED}_1 governs the standard two-point method beyond quadratics. Yue et al. analyze this method only for quadratics. For non-quadratic objectives, they assume a Lipschitz continuous Hessian and use methods based on cubic regularization with a subroutine of dd coordinate differences, whose complexities contain terms linear in dd. Theorems 4.1 and 4.3 show that ED1\mathrm{ED}_1 alone governs the standard two-point method for non-convex objectives, with no term linear in dd and without a Lipschitz continuous Hessian.

The technical difficulty beyond quadratics lies in the descent inequality, which involves the Hessian at points that depend on the random direction. An operator-norm bound at these points would introduce a factor of dd. Our proof avoids it by averaging over the rotations that fix the gradient and by a bound along rays based on the Gaussian Poincaré inequality and an integrated Bochner identity.

W2: Twisted Valley and the non-rotating control

Thank you for this careful reading of Appendix D. The identical behavior of ZO-Gauss on the rotating and non-rotating variants is in line with Assumption 3.2, which constrains the curvature at each point but not how its directions change. The envelope analyses, in contrast, pay a factor of dd on the rotating variant but not on the non-rotating one. The identical curves thus show that this factor of dd is a cost of the envelope analyses and not of ZO-Gauss. We revised Section 6.1 to state this relation to our theory.

As you pointed out, the Twisted Valley is a constructed example, and the submitted version examined the curvature of the neural networks only at two nearby points of one run. To examine the change of curvature directions in practical training, we computed the exact Hessian every 500500 queries along one ZO-Gauss run on the adapted CNN and ViT. The overlap between the leading 55-dimensional eigenspaces at the initial and the final iterate is 0.360.36 (CNN) and 0.410.41 (ViT), while the trace norm of the Hessian stays bounded. The dominant curvature directions thus change substantially along successful ZO-Gauss runs, which is the regime that our pointwise condition covers and that the Twisted Valley isolates. These results now appear in Appendix E (Figure 8), and Section 6.2 refers to them.

AuthorsOpenReview ↗

Response to reviewer QGo6 (2/2)

W3 and Q2: Weak dimension dependence when the added dimensions contribute

Thank you for pointing out that the width sweep alone does not separate weak dimension dependence from unused dimensions. As you noted, ZO-Gauss reaches within the budget the same loss as without the adapter (r=0r=0), and we agree that the validation in the submitted version was limited in this respect.

To provide the evidence asked for in Q2, we added a sweep in which the added dimensions contribute. We start the head at the optimum of the head-only model, whose training loss we denote by fLPf_{\mathrm{LP}}. Any decrease below fLPf_{\mathrm{LP}} must then come from the adapter.

The table below compares the decrease fLP−ff_{\mathrm{LP}}-f of the training loss after 4000040000 queries for ZO-Gauss, ZO-Coord, and gradient descent (GD) at the step size of ZO-Gauss (medians over five seeds).

d=647d=64714211421245324534517451786458645
CNN, ZO-Gauss0.0230.0230.0540.0540.0590.0590.0620.0620.0570.057
CNN, GD0.0210.0210.0560.0560.0620.0620.0620.0620.0570.057
CNN, ZO-Coord0.0020.0020.0010.0010.0030.0030.0020.0020.0010.001
ViT, ZO-Gauss0.0640.0640.1620.1620.1930.1930.1390.1390.0870.087
ViT, GD0.0600.0600.1620.1620.1990.1990.1510.1510.0910.091
ViT, ZO-Coord0.0600.0600.0410.0410.0060.0060.0010.0010.0010.001

GD lowers the loss below fLPf_{\mathrm{LP}} at every dd, which shows that the added dimensions contribute. ZO-Gauss lowers it by about as much as GD at every dd, whereas ZO-Coord hardly lowers it for large dd. We added these results to Appendix F.4 (Figure 11) and refer to them in Section 6.2.

Trace-norm condition. On the Twisted Valley, Proposition 5.1 proves the condition with constants independent of dd. For the neural networks, the trace norm stays bounded along the ZO-Gauss run of Appendix E (Figure 8), and at the initial point of the sweep above, it hardly changes as dd grows (Figure 11).

Step sizes. We agree that the selected step sizes lie above 1/(100H)1/(100H). The constant 100100 is not optimized, as we note after the proof of Proposition A.8, and there may be room to improve it. The prediction that the step size depends on dd only through HH, however, does not depend on this constant, and the selected step sizes agree with it. In Appendix D, tuning the step size of ZO-Gauss on the Twisted Valley selects 0.30.3 for every dd from 5050 to 5000050000. In the width sweep of Table 1, the selected step size of ZO-Gauss decreases by at most a factor of three as dd grows thirteenfold, from 3×10−33\times10^{-3} to 10−310^{-3} on the CNN, and does not change on the ViT. In the sweep from the head-only optimum, ZO-Gauss uses the same step size for every dd.

Reviewer_hJH3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes the standard two-point Gaussian zeroth-order estimator under ∥∇2f(x)∥∗≤H\|\nabla^2 f(x)\|_*\le H. It proves an O(HΔ/T)O(H\Delta/T) stationarity bound, up to finite-difference and noise terms, and linear convergence to a residual neighborhood under the PL condition. The analysis allows curvature directions to vary without requiring a common Hessian envelope. The proof combines rotational averaging with Gaussian functional inequalities to control the direction-dependent update.

Critical strengths
  1. Building on the spectral quantity introduced by Yue et al. (2023), the paper obtains general nonconvex guarantees without a common Hessian envelope or Hessian-Lipschitz assumption.

  2. Propositions 5.1–5.2 give dimension-independent trace-norm and PL constants while quantifying the cost of an envelope covering changing curvature directions.

Critical weaknesses

The experiments provide limited evidence for the practical implications of the new analysis. The nonrotating control closely matches the Twisted Valley results, and head-only adaptation performs similarly to wider adapters. Selected step sizes exceed the theorem’s range, while sampled Hessians do not establish the uniform bound.

Questions for authors
  1. In Section 5.2, could the authors distinguish two-sided envelopes from MeZO’s one-sided condition and specify the step-size regime behind the linear-in-dd envelope cost?
3 comments
AuthorsOpenReview ↗

Response to reviewer hJH3 (1/2)

We thank the reviewer for recognizing our theoretical contribution. We address your concerns below.

Weakness: Evidence for the practical implications

Thank you for raising these concrete points on the experiments. We address them in the order of the review.

Non-rotating control. That ZO-Gauss gives identical curves with and without the rotation agrees with Assumption 3.2, which bounds the curvature at each point and places no constraint on how its directions change. The envelope analyses pay a factor of dd only on the rotating variant. The agreement of the two curves therefore indicates that this factor arises from the envelope analyses rather than from ZO-Gauss. We revised Section 6.1 to make this connection explicit.

Head-only adaptation. We agree that this part of the validation was limited in the submitted version. Without the adapter (r=0r=0), ZO-Gauss already reaches the same loss within the budget, and the width sweep therefore leaves open whether ZO-Gauss makes use of the added dimensions. We added a sweep in which the head starts at the optimum of the head-only model, where the training loss equals fLPf_{\mathrm{LP}}. As fLPf_{\mathrm{LP}} is the lowest loss that the head alone can reach, any decrease below it is due to the adapter. The table below lists fLP−ff_{\mathrm{LP}}-f after 4000040000 queries (medians over five seeds) for ZO-Gauss, ZO-Coord, and gradient descent (GD) run at the step size of ZO-Gauss.

d=647d=64714211421245324534517451786458645
CNN, ZO-Gauss0.0230.0230.0540.0540.0590.0590.0620.0620.0570.057
CNN, GD0.0210.0210.0560.0560.0620.0620.0620.0620.0570.057
CNN, ZO-Coord0.0020.0020.0010.0010.0030.0030.0020.0020.0010.001
ViT, ZO-Gauss0.0640.0640.1620.1620.1930.1930.1390.1390.0870.087
ViT, GD0.0600.0600.1620.1620.1990.1990.1510.1510.0910.091
ViT, ZO-Coord0.0600.0600.0410.0410.0060.0060.0010.0010.0010.001

For every dd, ZO-Gauss goes below fLPf_{\mathrm{LP}} by nearly the same amount as GD with the same step size, while ZO-Coord makes little progress once dd is large. These results are now in Appendix F.4 (Figure 11), and Section 6.2 refers to them.

Step sizes. The step sizes selected in the experiments are indeed larger than 1/(100H)1/(100H). The constant 100100 has not been optimized (see the remark after the proof of Proposition A.8), and it may be possible to improve it. What the theory predicts independently of this constant is that the step size depends on dd only through HH, and the selected step sizes are consistent with this. On the Twisted Valley, the step size selected by tuning ZO-Gauss is 0.30.3 for all dd between 5050 and 5000050000 (Appendix D). In Table 1, a thirteenfold increase of dd changes the selected step size of ZO-Gauss by at most a factor of three on the CNN (3×10−33\times10^{-3} to 10−310^{-3}) and not at all on the ViT. The sweep from the head-only optimum above also uses one step size for all dd.

Uniform bound. On the Twisted Valley, the bound holds with constants independent of dd by Proposition 5.1. For the neural networks, we added an experiment that evaluates the exact Hessian every 500500 queries along a ZO-Gauss run on the adapted CNN and ViT. Along the run, the leading eigenspaces move substantially while the trace norm remains bounded. Between the initial and the final iterate, the leading 55-dimensional eigenspaces overlap by 0.360.36 (CNN) and 0.410.41 (ViT). The curvature directions thus change under a bounded trace norm in practical training as well, which is the regime that our pointwise condition covers. Appendix E (Figure 8) now reports these results, and Section 6.2 refers to them.

AuthorsOpenReview ↗

Response to reviewer hJH3 (2/2)

Question: Envelope conditions and the step-size regime in Section 5.2

Thank you for this question. We revised Section 5.2 to answer both of its parts.

First, Section 5.2 now separates the two envelope conditions. The submitted version described DPZero and MeZO by the same one-sided bound and stated that both imply our condition, which is true for DPZero but not for MeZO. The revised text describes the condition of DPZero as a global two-sided envelope, −A⪯∇2f(y)⪯A-A\preceq\nabla^2f(y)\preceq A for all yy, and that of MeZO as a local one-sided envelope, ∇2f(y)⪯At\nabla^2f(y)\preceq A_t on a ball around xtx_t whose radius is at least $\eta d\lVert\nabla f(x_t)\rVert $. Proposition 5.2(i) has been relaxed from A⪰0A\succeq0 to any symmetric AA and therefore covers both conditions.

Second, Section 5.2 now specifies the step-size regime behind the linear cost. The cost is linear in dd when the step size η\eta does not depend on dd, as in the analysis of MeZO. In this case, the radius ηd∥∇f(xt)∥\eta d\lVert\nabla f(x_t)\rVert is proportional to dd, and by Proposition 5.2(i), the trace of the envelope grows linearly as well. A step size that depends on dd can keep the radius bounded only with η=O(1/d)\eta=\mathcal{O}(1/d), which shrinks the decrease per iteration by a factor of dd. The envelope analysis therefore pays a factor of dd on the Twisted Valley in either regime, while our guarantee holds at h=1/(100H)h=1/(100H) with HH independent of dd.

Reviewer_hJH3OpenReview ↗

Thank you for the clarifications and additional experiments

Thank you for the clarifications and additional experiments. The results starting from the head-only optimum and the Hessian diagnostics strengthen the empirical support. Overall, the rebuttal strengthens the paper, and I maintain my weak acceptance recommendation.

Comments on the paper

1 comment

We thank all reviewers for their careful reading and constructive feedback. We have uploaded a revised version, in which changes are marked in red. All numbers of sections, results, and figures in our responses refer to the revised version. The main changes are as follows.

  • Comparison with Yue et al. (2023) (Section 2, Remark 4.6). We clarify that their effective dimension ED1\mathrm{ED}_1 alone governs the standard two-point method beyond quadratics, and explain the technical difficulty and the tools by which we avoid a factor of dd.
  • Envelope conditions (Section 5.2). We distinguish the global two-sided envelope of DPZero from the local one-sided envelope of MeZO, and state the step-size regime in which the cost of the envelope grows linearly in dd.
  • Non-rotating control (Section 6.1). We state that the identical behavior of ZO-Gauss with and without the rotation is in line with our pointwise condition.
  • Curvature along the trajectory (Appendix E, Figure 8). Along a ZO-Gauss run on the CNN and ViT, the leading eigenspaces of the Hessian change substantially while the trace norm stays bounded.
  • Sweep from the head-only optimum (Appendix F.4, Figure 11). ZO-Gauss lowers the loss below the head-only optimum for every dd, by about as much as gradient descent with the same step size, whereas ZO-Coord hardly does so for large dd.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.