acceptodds
Under review as a conference paper at ICLR 2027

Accelerating LLM Pretraining with Zeroth-Order Curvature on Idle Serving Compute

Abstract

LLM pretraining is increasingly compute-intensive, while substantial inference serving compute remains idle during off-peak periods. However, such serving hardware is difficult to use for conventional first-order training due to its memory, networking and other system requirements. Zeroth-order (ZO) optimization offers a natural interface to this forward-only compute resource, but its gradient estimates break down in pretraining. Unlike fine-tuning, which can confine updates to a small parameter subspace, pretraining must update every parameter, and the alignment of a ZO gradient estimate with the true gradient shrinks as the number of updated parameters grows. We make a different observation: noisy ZO estimates can still provide useful curvature information, even when they are too inaccurate to serve as gradients. Based on this insight, we propose ZO-Curv, which leverages idle forward-only devices to estimate curvature and uses it to adapt and refine the update used during first-order training, thereby accelerating the process. We provide convergence guarantees that characterize the effect of finite-query curvature estimation. Experiments on LLMs from 125M to 1.5B parameters show that ZO-Curv consistently accelerates pretraining. At 770M parameters, ZO-Curv reaches AdamW's 20K-step validation loss in 3.9 hours instead of 7.5 hours, while achieving a lower final loss of 2.70 versus 2.84 under the same step budget. These results show that idle model serving compute can accelerate LLM pretraining when zeroth-order queries are used to estimate curvature rather than gradients directly.

Primary area:
optimization
Keywords:
large language model pretraining, Zeroth-Order optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cZAEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ZO-Curv, a hybrid optimization framework that leverages idle inference-serving GPUs to accelerate LLM pretraining. Unlike conventional zeroth-order (ZO) methods that estimate descent directions, ZO-Curv uses forward-only queries to estimate diagonal curvature, while retaining exact first-order gradients on the training node. The estimated curvature is incorporated into a Sophia-style preconditioned update.

The authors provide theoretical results characterizing the finite-query curvature estimator and establish conditional convergence guarantees. Experiments on models ranging from 125M to 1.5B parameters demonstrate improved pretraining efficiency. For a 770M model, ZO-Curv reaches AdamW's target validation loss in 3.98 hours versus 7.53 hours, while achieving a lower final validation loss (2.70 vs. 2.84). Additional experiments investigate query budgets, batch-size tradeoffs, and downstream performance.

Critical strengths

S1. Interesting use of zeroth-order information for curvature estimation.

The paper identifies a useful distinction between estimating descent directions and estimating curvature. Using noisy ZO estimates for preconditioning rather than directly updating model parameters is an interesting approach to exploiting forward-only hardware. The proposed framework offers a practical way to incorporate otherwise unused inference resources into pretraining.

S2. Comprehensive empirical evaluation with promising improvements.

The experiments cover multiple model scales, query budgets, and downstream tasks. The comparison with SophiaG demonstrates that the forward-only curvature estimator can achieve similar optimization efficiency per step. The additional experiment with disjoint GPU groups also provides initial evidence that the approach can work with concurrent computation.

Critical weaknesses

W1. The practical feasibility and end-to-end efficiency are insufficiently validated.

Most reported wall-clock results exclude curvature computation on the serving devices (Section 4.1). With the default query budget of q=512, each curvature refresh requires 1,024 forward evaluations. The paper does not quantify the serving capacity required to sustain this workload, the overhead of frequently transferring model snapshots, or the impact of competing inference requests.

Although Figure 6 provides a concurrent deployment experiment, it uses GPU groups within the same 8×H100 node, rather than a realistic heterogeneous serving cluster. Consequently, the reported approximately 2× acceleration depends on assumptions about the availability and throughput of idle compute that are not sufficiently validated.

W2. The contribution of meaningful coordinate-wise curvature information is not sufficiently isolated.

According to Proposition 1, the expected estimator contains a uniform damping term proportional to the total GNB curvature divided by q. When the parameter dimension is much larger than q, this term can dominate the average coordinate-wise curvature signal. It is therefore unclear whether the empirical gains primarily arise from informative diagonal preconditioning or from a simpler form of global adaptive normalization.

The paper lacks an ablation against a scalar preconditioner or another control that removes the coordinate-specific curvature signal while preserving the optimizer's other components. This makes it difficult to establish whether the proposed curvature estimator is necessary for the observed improvements.

Questions for authors
  1. Serving capacity and realistic speedups: How many serving GPUs are required to sustain q=512 curvature queries with the reported one-step or three-step staleness? Can the authors provide an end-to-end evaluation that includes communication overhead, total GPU-hours, and interference with inference workloads? A time-to-target comparison with SophiaG using its original ten-step curvature refresh schedule would also be informative.

  2. Contribution of diagonal curvature: How does ZO-Curv compare with a scalar preconditioner based on the same curvature statistics? Can the authors report the coordinate-wise correlation between the estimated ZO curvature and the reference GNB diagonal? These results would help determine whether coordinate-specific curvature information drives the improvement.

  3. Applicability of the theoretical guarantees: The convergence analysis requires the curvature distortion bound δ_T to be smaller than 1. However, substituting the experimental EMA coefficient β₂=0.99 into Equation (60) appears to yield a bound substantially greater than 1 at the evaluated model dimensions, even with optimistic constants. Can the authors clarify whether the theoretical guarantees are non-vacuous in the experimental regime?

3 comments

Rebuttal:

Thank you for your thoughtful review and for the concrete suggestions on systems accounting and ablations. We hope that our rebuttal further strengthens the soundness of our work. Below are the answers to your concerns and questions:

W1 & Q1. What serving capacity, communication cost and end-to-end GPU-hours does the deployment need?

All numbers use q=512q=512 and a curvature minibatch of 8 sequences. Capacity is converted to L40S cards using the peak bf16 throughput ratio H100 : L40S ≈2.73\approx2.73.

Serving capacity. One refresh costs 5.3 first-order steps, both in measured time and in FLOPs, at every model size. Staleness (s=1s=1 or s=3s=3) only hides latency. The pool must still finish one refresh per step, which requires about 116 L40S(8×5.3×2.738\times5.3\times2.73) for an 8×8\times H100 node. A lower refresh frequency reduces this. With the ten-step schedule of the original Sophia (2 steps for communication, 8 for queries), 16 L40S suffice, and the training node never waits for curvature.

Communication. Per refresh, a 1.5B model sends a 3.0 GB bf16 snapshot and receives a 6.0 GB fp32 vector. At 100 Gbps, this round trip, including the all-reduce inside the pool, takes less than 2 s, below one training step (2.446 s). The average bandwidth with k=10k=10 is about 2.9 Gbps.

End-to-end cost (1.5B, 20K steps):

MethodHardwareWall-clockH100-hIdle serving GPU-hLoss
AdamW8×8\timesH10013.59 h108.7–2.806
SophiaG, k=10k=108×8\timesH10014.90 h119.2–2.713
ZO-Curv, k=10k=108×8\timesH100 + 16 L40S≤13.70 h≤109.6219 L40S-h2.72
ZO-Curv, k=1k=18×8\timesH100 + ≈116 L40S13.70 h109.6≈1,590 L40S-h2.692

We agree that the ~2× acceleration refers to reaching AdamW's final loss. Against ten-step SophiaG, the per-step gain is 6–12%, and we will state this explicitly.

W2 & Q2. Does coordinate-wise curvature matter, compared with a scalar preconditioner and the GNB diagonal?

We agree that the damping dominates the average: it exceeds the mean GNB entry by d/q≈2.4×105d/q\approx2.4\times10^5 at 125M. We therefore ran both controls you asked for.

Scalar-preconditioner control. At every step, we replace htZOh_t^{\rm ZO} by its coordinate mean over all parameters. Everything else is unchanged (qq, γ\gamma, η\eta, ρ\rho, β2\beta_2, momentum, schedule). This scalar preconditioner has exactly the same global scale as ZO-Curv and lacks only the coordinate-specific signal.

Variant (125M, q=512q=512)Steps to target loss(3.04)
ZO-Curv (full)14.3K
Scalar (coordinate mean of htZOh_t^{\rm ZO})20K

The scalar variant is worse by 5.7K, so global adaptive normalization alone does not explain the improvement. The coordinate information is concentrated on the high-curvature coordinates, whose GNB entries exceed the damping.

Coordinate-wise agreement. On the same checkpoint, minibatch and pseudo-labels, the single-step ZO estimate has a cosine of about 0.005 with the SophiaG GNB diagonal. It is also above the agreement of ZO gradients in LoRA fine-tuning (9.4×10−49.4\times10^{-4}), where ZO is known to be effective (Fig. 2(b)).

Q3. Are the theoretical guarantees non-vacuous when δT≫1\delta_T\gg1 at β2=0.99\beta_2=0.99?

You are right that, as stated, δT≫1\delta_T\gg1 in our regime. However, δT<1\delta_T<1 comes only from writing the distortion symmetrically as (1±δT)Dt(1\pm\delta_T)D_t, and the argument does not need it. Descent and O(1/T)O(1/T) stationarity use only the lower side, D^t⪰mDt\widehat D_t\succeq mD_t with m>0m>0. This holds deterministically: hZOh^{\rm ZO} is a square, so the floor gives d^t,i≥ϵ0\hat d_{t,i}\ge\epsilon_0 and m=ϵ0/dmax⁡m=\epsilon_0/d_{\max} for any β2\beta_2, qq, ε\varepsilon and dd. The upper side enters only the constant of the local linear rate. We will restate the analysis with mTDt⪯D^t⪯MTDtm_TD_t\preceq\widehat D_t\preceq M_TD_t, where mT=max⁡{1−δT,ϵ0/dmax⁡}m_T=\max\{1-\delta_T,\epsilon_0/d_{\max}\} and MT=1+δTM_T=1+\delta_T. The results become descent for η≤mT/α\eta\le m_T/\alpha, min⁡tΨt=O(1/T)\min_t\Psi_t=O(1/T), and a contraction of 1−μDαmTMT1-\frac{\mu_D}{\alpha}\frac{m_T}{M_T}. After this change, no condition contradicts our setting (β2=0.99\beta_2=0.99, q=512q=512, ε=10−4\varepsilon=10^{-4}, dd up to 1.5×1091.5\times10^9). The guarantee is qualitative, as in Sophia's analysis. It explains why ZO curvature noise cannot break descent, but it does not predict the exact speed of convergence. We will make this scope explicit.

Thank you again for helping us improve the clarity and rigor of our work.

The serving-capacity accounting and the scalar-preconditioner ablation address my main concerns regarding practicality and the contribution of coordinate-wise curvature. I will raise my score accordingly.

Dear Reviewer,

Thank you for your positive feedback and for acknowledging our additional experiments and analysis. We sincerely appreciate your thoughtful review and constructive suggestions, which have helped us improve our work.

Reviewer_7EeCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ZO-Curv, a hybrid optimization framework that leverages idle inference GPUs to accelerate LLM pretraining. The method retains first-order gradient computation on training GPUs while utilizing idle inference GPUs to estimate curvature through forward passes. These curvature estimates are then incorporated into Sophia-style preconditioned updates to optimize the model.

Critical strengths
  1. The research motivation is well-founded and has practical value.
  2. Using zeroth-order methods to estimate curvature is an insightful approach.
  3. The method is validated across model scales and settings.
Critical weaknesses
  1. According to Proposition 1 and the analysis in Section 4.3, the uniform damping term may require ZO-Curv to use more zeroth-order queries to maintain training performance as model size increases, leading to higher computational overhead and potentially limiting its scalability.
  2. The main experiments show that ZO-Curv achieves significant training speedups over SophiaG with curvature updates at every step. However, the results in Appendix D show that when SophiaG uses its default setting of updating curvature every 10 steps, its per-step training time is 6–12% longer than that of ZO-Curv. ZO-Curv offers limited per-step speedup over default SophiaG, while its speedup in reaching the same validation loss remains unclear.
  3. The paper lacks a direct comparison of curvature estimates between SophiaG and ZO-Curv under identical conditions. Additional experiments are recommended to clarify their differences in numerical scale and per-parameter estimates.
Questions for authors

Can ZO-Curv update curvature every 10 steps, as SophiaG does? When both methods use the same curvature update frequency, how does ZO-Curv compare with SophiaG in terms of training performance and practical speedup?

2 comments

Rebuttal:

Thank you for your constructive comments. We hope that the clarifications and additional experiments below address your concerns. Below are the answers to your concerns and questions:

W1. The uniform damping may require more queries as models grow, limiting scalability.

We agree that the damping λt=∥htG∥1/q\lambda_t=\|h_t^{\rm G}\|_1/q grows with the total curvature, and that larger models saturate at larger qq (Fig. 7). However, neither the theory nor the data imply that qq must grow with dd. In the theory, qq only changes the reference preconditioner from the GNB diagonal to a damped GNB diagonal, and the distortion depends on dd and qq only through log⁡(qdT/ζ)\log(qdT/\zeta). A coordinate receives its own scale when its share of the curvature mass exceeds about 1/q1/q. This share depends on how concentrated the curvature is, not on the number of parameters. Empirically, the fraction of SophiaG's improvement over AdamW that ZO-Curv recovers does not shrink with scale:

qq41664256512
355M0.170.470.911.121.13
770M0.280.480.630.870.91
1.5B0.250.560.810.921.00

For every q≥8q\ge8, 1.5B recovers at least as much as 770M, although it has about twice the parameters.

W2 & Q. Can ZO-Curv update curvature every 10 steps? How do ZO-Curv and SophiaG compare when both refresh curvature every 10 steps?

Yes, ZO-Curv can refresh every 10 steps, and we have run this setting at all four sizes. For ZO-Curv, the estimate computed at θt\theta_t is applied only from step t+10t+10 to t+19t+19. Its curvature is therefore one interval staler than SophiaG's. Times are for the training node, 20K steps, 8×8\times H100.

ModelSophiaG k=10k{=}10: loss / time (h)ZO-Curv k=10k{=}10: loss / time (h)Per-step speedup
125M3.03 / 1.483.03 / 1.368.1%
355M2.82 / 3.772.83 / 3.468.2%
770M2.72 / 8.492.74 / 7.5311.3%
1.5B2.71 / 14.902.72 / 13.638.5%

At the same frequency, ZO-Curv reaches the loss of SophiaG despite its staler curvature. We agree that the per-step gain is modest. It is, however, the whole curvature cost that SophiaG still pays on the training node: 9–13% above AdamW, against 1–5% for ZO-Curv, plus extra memory (68.9 vs. 64.4 GB at 1.5B). More importantly, the refresh frequency is a free choice for ZO-Curv but not for SophiaG. Fresher curvature costs SophiaG training time, whereas for ZO-Curv it only adds serving load. At the same training-node cost, ZO-Curv can therefore refresh every step and end 0.02–0.04 below ten-step SophiaG at every size (Fig. 12). We will clarify that the ~2× speedup in the main text is measured against AdamW. We will also make ten-step SophiaG the main baseline and add the table above.

W3. A direct comparison of the two curvature estimates is missing.

We added this comparison. On a 125M checkpoint, we fix one curvature minibatch and one set of pseudo-labels. On them we compute the SophiaG GNB estimate htGh_t^{\rm G} by backpropagation and the ZO-Curv estimate htZOh_t^{\rm ZO} from q=512q=512 forward-only queries.

The single-step cosine similarity is about 0.005. For context, it is above the cosine between the ZO gradient and the exact gradient in LoRA fine-tuning (9.4×10−49.4\times10^{-4}), where ZO is known to work, and in pretraining (4.1×10−54.1\times10^{-5}) (Fig. 2(b)). Replacing htZOh_t^{\rm ZO} by its coordinate mean while keeping its scale increases the number of steps required to reach the target loss(14.3K -> 20K). The per-parameter information therefore matters in training.

We will add this comparison as a new appendix section.

Thank you again for your valuable suggestions.

Dear Reviewer,

We kindly ask you to consider our responses to your comments and would be happy to address any further questions or concerns. We sincerely appreciate your time and consideration.

Reviewer_vkh8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes ZO-Curv, which uses idle forward-only serving devices to estimate the diagonal Gauss-Newton-Bartlett (GNB) curvature of Sophia-G during LLM pretraining, while the training node computes the exact first-order gradient. Serving nodes evaluate q central-difference queries of a sampled-label loss. The averaged directional estimate is squared, EMA-smoothed, floored, and used as the denominator of a clipped momentum update. Proposition 1 shows that this estimate is unbiased for the GNB diagonal plus a uniform damping ∥htG∥1/q\|h^G_t\|_1/q, and Theorems 1 to 3 give descent, local linear convergence, and an EMA concentration bound for a deterministic version. On GPT-2 models from 125M to 1.5B trained on OpenWebText, ZO-Curv matches SophiaG per step and, counting only training-node time, reaches the 20K-step AdamW loss at 770M in 3.98 h against 7.53 h.

Critical strengths
  1. Section 2.2 and Table 1 give clear negative evidence that ZO as a descent direction fails at pretraining scale. Pure ZO stalls at 8.01 at 770M, and InstructedZO ends at 3.37 against 2.84 for AdamW. Keeping the exact gradient on the training node is a sound response, and Fig. 10 shows that the forward-only substitute for the GNB backward pass tracks SophiaG step for step at all four sizes.

  2. Proposition 1 is correct and states the exact cost of a finite query budget. I checked Lemma 1 (Eqs. 22 to 30). Averaging the q directional estimates before squaring gives damping ∥htG∥1/q\|h^G_t\|_1/q, a factor of q below squaring each estimate first (p. 22). Anyone designing a ZO curvature estimator for large d needs this calculation.

Critical weaknesses
  1. The preconditioner is close to a scalar. By Eq. (4) and Eq. (13), E[htZO]=(1+1/q)htG+λt1\mathbb{E}[h_t^{ZO}] = (1+1/q) h_t^G + \lambda_t \mathbf{1} with λt=∥htG∥1/q\lambda_t = \Vert h_t^G \Vert_1 / q. The entries of htGh_t^G sum to qλtq \lambda_t, so at most q coordinates can have ht,iG≥λth_{t,i}^G \ge \lambda_t. At 770M with q = 512, every other coordinate (about 7.7×1087.7 \times 10^8 of them) has an expected denominator between γλt\gamma \lambda_t and about 2γλt2 \gamma \lambda_t. This scalar falls as 1/q with γ and η fixed, so the q-sweep in Fig. 7 also changes the effective step size and the clipped share. At q = 1 the estimate is BΔ2(u⊙u)B \Delta^2 (u \odot u), with essentially no coordinate information, yet it beats AdamW in every row of Fig. 8. Please add a run with htZOh_t^{ZO} replaced by its coordinate mean, and report the clipped fraction.

  2. Sec. 4.1 excludes the 2q = 1024 forward passes per step from timing, and the curvature batch size is never stated. At 0.66 AdamW steps per two-point query (Fig. 3(b)), a curvature batch the size of the training batch would make that about 340 AdamW steps per step. Each refresh also sends θt\theta_t to every serving node and returns a d-dimensional zˉt\bar z_t, the per-step traffic of data-parallel training that Sec. 1 says serving platforms lack. SophiaG refreshes every step. With the standard k = 10, Figs. 11 and 12 put it 6 to 12% slower per step than ZO-Curv and 0.01 to 0.04 higher in loss, so offloading buys far less than 2×. Please report serving FLOPs and bandwidth per step and make ten-step SophiaG the main baseline.

Questions for authors
  1. Fig. 8 reports 3.03 for AdamW at 355M (q = 0, B = 512), while Fig. 12 and Fig. 13(b) report 2.90 for AdamW at 355M after 20K steps, and the ZO-Curv values at q = 512 agree across these figures (2.77 to 2.78). If 2.90 is correct, no cell in the B = 256 row reaches it (best 3.01) and the RQ3 conclusion reverses. Which step budget and AdamW settings does Fig. 8 use, and why does AdamW at 125M read about 3.07 at 20K steps in Fig. 10(a) but 3.27 in Fig. 12?

  2. What are q, the curvature batch, the global batch, and the per-step time of each method in Fig. 6? The appendix that describes this deployment appears as "Appendix ??". ZO-Curv ends at 2.81 there against 2.70 in Table 1 after the same 20K steps, which is close to the 2.84 of AdamW.

  3. Theorem 3 needs δT<1\delta_T < 1. For the 770M runs ΛT=log⁡(16qdT/ζ)\Lambda_T = \log(16qdT/\zeta) is about 40, so with β2 = 0.99 and Cb≥2C_b \ge 2 the term 2κ⋅43Cb(1−β2)ΛT22\kappa\cdot\frac{4}{3}C_b(1-\beta_2)\Lambda_T^2 in Eq. (60) already exceeds 80, and ϑT\vartheta_T in Eq. (59) grows as ε2d3/2\varepsilon^2 d^{3/2}. For which β2 and ε is the bound non-vacuous at d near 10910^9?

3 comments

Thank you for your very careful review and for cross-checking our numbers and constants in such detail. Below are the answers to your concerns and questions:

W1. The ZO preconditioner is close to a scalar; please add a coordinate-mean ablation with the clipped fraction.

We agree with your calculation. By Proposition 1, the damping λt\lambda_t exceeds the average GNB entry by a factor of d/qd/q (about 2.4×1052.4\times10^5 at 125M with q=512q=512). The bulk of the coordinates therefore shares a nearly common scale, and the qq-sweep changes the damping and the effective step size together. We ran the ablation you requested at 125M. At every step, htZOh_t^{\rm ZO} is replaced by its coordinate mean hˉt1\bar h_t\mathbf 1, and everything else stays the same. This keeps the scalar level of the denominator, and hence the effective step size, identical. Only the variation across coordinates is removed.

Variant (125M, q=512q=512)Steps to target loss(3.04)Clipped coordinates (%)
ZO-Curv (full)14.3K77.32
htZO→h_t^{\rm ZO}\to coordinate mean20K14.8

Removing coordinate-wise variation increases the number of steps required to reach the target loss, so coordinate information contributes beyond scalar normalization. Your bound also shows where this information is: it sits on the heavy-tailed, high-curvature coordinates with ht,iG≳λth^{\rm G}_{t,i}\gtrsim\lambda_t, which a scalar has to treat like the bulk. We will state this in RQ2, add the ablation to the appendix.

W2. Serving FLOPs and bandwidth are not reported, and ten-step SophiaG should be the main baseline.

We apologize for omitting the curvature batch: each query uses 8 sequences, not the 512-sequence training batch. One refresh therefore costs 512×0.66/64≈512\times0.66/64\approx 5.3 AdamW steps, not 340. The FLOPs ratio 2q⋅2N⋅8K / 6N⋅512K=5.332q\cdot2N\cdot8{\rm K}\,/\,6N\cdot512{\rm K}=5.33 agrees with this measurement and does not depend on model size. Each refresh moves one bf16 snapshot and one fp32 vector; data and pseudo-labels are sampled on the serving side. This is one broadcast and one reduction per refresh:

1.5B, q=512q=512Refresh every step (k=1k=1)Refresh every 10 steps (k=10k=10)
Serving FLOPs per training step5.3 AdamW steps (2.5×10162.5\times10^{16})0.53 AdamW steps
Traffic per refresh (down / up)3.0 GB / 6.0 GB3.0 GB / 6.0 GB
Transfer time at 100 Gbps0.72ms0.72ms
Serving GPUs for an 8×8\times H100 node≈116 L40S16 L40S

We adopt ten-step SophiaG as the main baseline and also run ZO-Curv with k=10k=10. Our curvature is applied with the deployment delay, so it is 10–19 steps old, against 0–9 steps for SophiaG. ZO-Curv reaches 3.03 / 2.83 / 2.74 / 2.72, compared with 3.03 / 2.82 / 2.72 / 2.71 for SophiaG (125M → 1.5B). We agree that offloading gives no 2× speedup over ten-step SophiaG. The 2× refers to reaching AdamW's final loss, and we will correct the framing. Against ten-step SophiaG, offloading gives the following:

  • No curvature cost on the training node. ZO-Curv is 6–12% faster per step and uses less memory (64.4 vs. 68.9 GB at 1.5B).
  • Fresher curvature at the same training-node cost. Curvature is refreshed more often by adding serving GPUs, not training time. With k=1k=1, ZO-Curv ends 0.02–0.04 below ten-step SophiaG at every size.

Q1. Why are the AdamW numbers inconsistent across Fig. 8, 10, 12 and 13, and does the RQ3 conclusion still hold?

You are right, and we have traced the cause. Fig. 10(a)(b), Fig. 12(b) and Fig. 13(b) come from an earlier set of runs on an 8×8\times A800 node. The main text, Fig. 8 and Fig. 12(a) use our 8×8\times H100 runs. We apologize for the confusion. On H100, AdamW reaches 3.27 at 125M and 3.03 at 355M after 20K steps, which matches Fig. 12(a) and Fig. 8. Fig. 8 uses a 20K-step budget for every cell and the AdamW settings of Appendix B: β=(0.9,0.95)\beta=(0.9,0.95), weight decay 0.1, cosine schedule with 2K warmup, and clipping at 1.0. The RQ3 conclusion does not reverse. It compares cells of one heatmap that share the same hardware, budget and protocol, and there (B=256,q=32)(B{=}256,q{=}32) matches (B=512,q=0)(B{=}512,q{=}0) at 3.03. We will correct the data in Fig. 10(a)(b), Fig. 12(b), and Fig. 13(b) in the appendix.

Q2. What are the settings of Fig. 6, and why does ZO-Curv end at 2.81 there?

"Appendix ??" is a formatting error: the deployment appendix was dropped from the PDF, and we will restore it. Fig. 6 uses GPT-2 355M on one 8×8\times H100 node split into two 4-GPU groups, with global batch 512, q=32q=32, curvature batch 8 and staleness s=3s=3. The per-step times are 1.24 s for AdamW, 2.608 s for SophiaG and 1.32 s for ZO-Curv. The 2.70 in Table 1 is at 770M with q=512q=512, so the two numbers are not comparable. The synchronous (B=512,q=32)(B{=}512,q{=}32) cell of Fig. 8 at 355M reaches 2.807, and the deployed run reaches 2.812, so the deployment itself costs almost nothing. AdamW reaches 2.84 only because it runs for more steps. It needs about 41K steps (≈14 h) to reach 2.81, whereas ZO-Curv needs 20K steps (≈7.3 h). Fig. 6 therefore shows ZO-Curv reaching a lower loss in about half of AdamW's time.

Q3. For which β2\beta_2 and ε\varepsilon is the bound with δT<1\delta_T<1 non-vacuous at d≈109d\approx10^9?

Your calculation is correct. As stated, δT<1\delta_T<1 would require β2≳1−5×10−5\beta_2\gtrsim1-5\times10^{-5} and ε=O(d−3/4)\varepsilon=O(d^{-3/4}), which is not our regime. However, the condition is an artifact of writing the distortion symmetrically as (1±δT)Dt(1\pm\delta_T)D_t, and the proof does not need it:

  • Lower side. Descent and O(1/T)O(1/T) stationarity only need D^t⪰mDt\widehat D_t\succeq m D_t for some m>0m>0. Because hZOh^{\rm ZO} is a square, d^t,i≥ϵ0\hat d_{t,i}\ge\epsilon_0, so m=ϵ0/dmax⁡m=\epsilon_0/d_{\max} holds deterministically for every β2\beta_2 and ε\varepsilon.
  • Upper side. 1+δT1+\delta_T enters only the constant of the local linear rate, so any finite value suffices.

We will restate Lemma 5 and Theorems 1–3 with mTDt⪯D^t⪯MTDtm_TD_t\preceq\widehat D_t\preceq M_TD_t, where mT=max⁡{1−δT,ϵ0/dmax⁡}m_T=\max\{1-\delta_T,\epsilon_0/d_{\max}\} and MT=1+δTM_T=1+\delta_T. The results become descent for η≤mT/α\eta\le m_T/\alpha, min⁡tΨt=O(1/T)\min_t\Psi_t=O(1/T), and a local contraction of 1−μDαmTMT1-\frac{\mu_D}{\alpha}\frac{m_T}{M_T}. After this change, no condition restricts β2\beta_2 or ε\varepsilon. The ε2d3/2\varepsilon^2d^{3/2} term only enters MTM_T: it can rescale the update but cannot flip its sign. The guarantee is qualitative, as is Sophia's analysis at this scale. We do not claim that it certifies our learning rates. It shows that curvature noise cannot reverse descent.

Thank you again for these detailed comments, which will make the revised paper considerably more accurate.

Dear Reviewer,

We kindly ask you to consider our responses to your comments and would be happy to address any further questions or concerns. We sincerely appreciate your time and consideration.

Comments on the paper

2 comments

Dear Reviewers,

We would like to thank all reviewers for their constructive feedback. We are encouraged that the reviewers recognize our paper:

  • has “well-founded” research motivation with “practical value” (Reviewer 7EeC), offering a way to utilize “otherwise unused inference resources into pretraining” (Reviewer cZAE).
  • proposes “an insightful approach” (Reviewer 7EeC) to curvature estimation, with an “interesting use of zeroth-order information” (Reviewer cZAE).
  • provides “comprehensive empirical evaluation with promising improvements” (Reviewer cZAE), validated across “model scales and settings” (Reviewer 7EeC).
  • presents sound theoretical analysis, with “Proposition 1” confirmed to be “correct” (Reviewer vkh8).

We have been working diligently on improving the paper in several aspects, addressing your concerns and problems.Below, we summarise the main improvements, all of which will be included in the revised version.

1. Stronger second-order baseline: ten-step SophiaG

  • We now use SophiaG with curvature refreshed every 10 steps (the default schedule of the original Sophia) as the main second-order baseline. We also ran ZO-Curv with the same k=10k=10 schedule at all four model sizes (125M–1.5B, 20K steps, 8×8\times H100).
  • ZO-Curv's curvature is one interval older (10–19 steps vs. 0–9 steps). Even so, it matches SophiaG's final loss, runs 8–11% faster per step, and uses less memory on the training node (64.4 vs. 68.9 GB at 1.5B).
ModelSophiaG k=10k{=}10: loss / time (h)ZO-Curv k=10k{=}10: loss / time (h)Per-step speedup
125M3.03 / 1.483.03 / 1.368.1%
355M2.82 / 3.772.83 / 3.468.2%
770M2.72 / 8.492.74 / 7.5311.3%
1.5B2.71 / 14.902.72 / 13.638.5%
  • Because curvature estimation is offloaded, the refresh frequency can be chosen freely: refreshing more often adds serving load but no training time. With k=1k=1, ZO-Curv reaches a 0.02–0.04 lower loss than ten-step SophiaG at every model size for the same training-node cost.
  • We now state explicitly that the ~2× acceleration is measured against reaching AdamW's final loss. We also report the per-step gain over SophiaG separately.

2. Complete systems accounting for the offloaded deployment

  • Serving compute. Each curvature query uses a minibatch of 8 sequences, so one refresh costs about 5.3 first-order steps. The measured time and the FLOP count agree on this figure, and it holds at every model size.
  • Serving capacity. With k=10k=10, 16 L40S GPUs per 8×8\times H100 training node are enough, and the training node never waits for curvature. With k=1k=1, about 116 L40S GPUs are needed.
  • Communication. Each refresh of the 1.5B model sends a 3.0 GB bf16 snapshot to the serving pool and receives a 6.0 GB fp32 vector back. At 100 Gbps, this round trip takes less than 2 s including the all-reduce inside the pool, which is shorter than one training step (2.446 s). The average bandwidth at k=10k=10 is about 2.9 Gbps.
  • End-to-end cost (1.5B, 20K steps):
MethodHardwareWall-clockH100-hServing GPU-hLoss
AdamW8×8\timesH10013.59 h108.7–2.806
SophiaG, k=10k=108×8\timesH10014.90 h119.2–2.713
ZO-Curv, k=10k=108×8\timesH100 + 16 L40S≤13.70 h≤109.6219 L40S-h2.72
ZO-Curv, k=1k=18×8\timesH100 + ≈116 L40S13.70 h109.6≈1,590 L40S-h2.692
  • Deployment details. We added a deployment appendix that gives the full Fig. 6 setup: 355M model, two 4-GPU groups, global batch 512, q=32q=32, curvature batch 8, staleness s=3s=3. Per-step times are 1.24 s for AdamW, 2.608 s for SophiaG, and 1.32 s for ZO-Curv. The asynchronous deployed run reaches nearly the same loss as the synchronous run (2.812 vs. 2.807). ZO-Curv reaches a loss of 2.81 in about 7.3 h, while AdamW needs about 14 h.

3. New evidence for coordinate-wise curvature information

  • Scalar-preconditioner control. At every step, we replaced htZOh_t^{\rm ZO} with its coordinate mean. This keeps the global scale and every other setting unchanged and removes only the coordinate-specific signal. The full ZO-Curv reaches the target loss in about 30% fewer steps (14.3K vs. 20K). Coordinate-wise curvature therefore contributes more than global adaptive normalization.
Variant (125M, q=512q=512)Steps to target loss (3.04)Clipped coordinates (%)
ZO-Curv (full)14.3K77.32
Coordinate mean of htZOh_t^{\rm ZO}20K14.8
  • Direct comparison with the GNB diagonal. We used the same checkpoint, minibatch, and pseudo-labels for both estimates. The single-step ZO estimate has a cosine similarity of about 0.005 with SophiaG's GNB diagonal. This is higher than the cosine similarity between ZO gradients and exact gradients in LoRA fine-tuning (9.4×10−49.4\times10^{-4}), a setting where ZO methods are known to work well.
  • Where the signal sits. The useful coordinate information is concentrated in the heavy-tailed, high-curvature coordinates whose GNB entries exceed the damping level.

4. Scalability with respect to the query budget

  • Theory. In our analysis, dd and qq appear only inside a logarithmic term. A coordinate gets its own scale once its share of the total curvature mass exceeds roughly 1/q1/q. That share depends on how concentrated the curvature is, not on the number of parameters.
  • Experiments. The fraction of SophiaG's improvement over AdamW that ZO-Curv recovers does not shrink as models grow. For q≥16q\ge16, the 1.5B model recovers at least as much as the 770M model, even though it has about twice as many parameters.
qq41664256512
355M0.170.470.911.121.13
770M0.280.480.630.870.91
1.5B0.250.560.810.921.00

5. Strengthened theoretical analysis

  • We generalised the analysis from a symmetric distortion bound, (1±δT)Dt(1\pm\delta_T)D_t, to an asymmetric one, mTDt⪯D^t⪯MTDtm_TD_t\preceq\widehat D_t\preceq M_TD_t. Here mT=max⁡{1−δT,ϵ0/dmax⁡}m_T=\max\{1-\delta_T,\epsilon_0/d_{\max}\} and MT=1+δTM_T=1+\delta_T.
  • The lower bound holds deterministically. Because hZOh^{\rm ZO} is a square, the floor guarantees d^t,i≥ϵ0\hat d_{t,i}\ge\epsilon_0.
  • With these bounds, we obtain descent for η≤mT/α\eta\le m_T/\alpha, min⁡tΨt=O(1/T)\min_t\Psi_t=O(1/T) stationarity, and a local contraction rate of 1−μDαmTMT1-\frac{\mu_D}{\alpha}\frac{m_T}{M_T}. The guarantees now hold for any β2\beta_2, ε\varepsilon, qq, and dd, including our practical setting (β2=0.99\beta_2=0.99, q=512q=512, ε=10−4\varepsilon=10^{-4}, dd up to 1.5×1091.5\times10^9).
  • Like Sophia's analysis, the guarantee is qualitative. It shows that ZO curvature noise cannot reverse descent, and we state this scope explicitly.

6. Presentation and reproducibility

  • We now report all AdamW results under a single 8×8\times H100 protocol, using a 20K-step budget and the Appendix B hyperparameters: β=(0.9,0.95)\beta=(0.9,0.95), weight decay 0.1, cosine schedule with 2K warmup, and clipping at 1.0. Figs. 10(a)(b), 12(b), and 13(b) are updated accordingly. The RQ3 conclusion is unchanged: (B=256,q=32)(B{=}256,q{=}32) matches (B=512,q=0)(B{=}512,q{=}0) at 3.03.
  • New appendix sections cover the scalar-preconditioner ablation, the ZO-vs-GNB comparison, the full k=10k=10 results, and the systems cost analysis.

We thank the reviewers again for their insightful feedback, which has substantially improved the rigor, completeness, and clarity of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.