- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ZO-Curv, a hybrid optimization framework that leverages idle inference-serving GPUs to accelerate LLM pretraining. Unlike conventional zeroth-order (ZO) methods that estimate descent directions, ZO-Curv uses forward-only queries to estimate diagonal curvature, while retaining exact first-order gradients on the training node. The estimated curvature is incorporated into a Sophia-style preconditioned update.
The authors provide theoretical results characterizing the finite-query curvature estimator and establish conditional convergence guarantees. Experiments on models ranging from 125M to 1.5B parameters demonstrate improved pretraining efficiency. For a 770M model, ZO-Curv reaches AdamW's target validation loss in 3.98 hours versus 7.53 hours, while achieving a lower final validation loss (2.70 vs. 2.84). Additional experiments investigate query budgets, batch-size tradeoffs, and downstream performance.
Critical strengths
S1. Interesting use of zeroth-order information for curvature estimation.
The paper identifies a useful distinction between estimating descent directions and estimating curvature. Using noisy ZO estimates for preconditioning rather than directly updating model parameters is an interesting approach to exploiting forward-only hardware. The proposed framework offers a practical way to incorporate otherwise unused inference resources into pretraining.
S2. Comprehensive empirical evaluation with promising improvements.
The experiments cover multiple model scales, query budgets, and downstream tasks. The comparison with SophiaG demonstrates that the forward-only curvature estimator can achieve similar optimization efficiency per step. The additional experiment with disjoint GPU groups also provides initial evidence that the approach can work with concurrent computation.
Critical weaknesses
W1. The practical feasibility and end-to-end efficiency are insufficiently validated.
Most reported wall-clock results exclude curvature computation on the serving devices (Section 4.1). With the default query budget of q=512, each curvature refresh requires 1,024 forward evaluations. The paper does not quantify the serving capacity required to sustain this workload, the overhead of frequently transferring model snapshots, or the impact of competing inference requests.
Although Figure 6 provides a concurrent deployment experiment, it uses GPU groups within the same 8×H100 node, rather than a realistic heterogeneous serving cluster. Consequently, the reported approximately 2× acceleration depends on assumptions about the availability and throughput of idle compute that are not sufficiently validated.
W2. The contribution of meaningful coordinate-wise curvature information is not sufficiently isolated.
According to Proposition 1, the expected estimator contains a uniform damping term proportional to the total GNB curvature divided by q. When the parameter dimension is much larger than q, this term can dominate the average coordinate-wise curvature signal. It is therefore unclear whether the empirical gains primarily arise from informative diagonal preconditioning or from a simpler form of global adaptive normalization.
The paper lacks an ablation against a scalar preconditioner or another control that removes the coordinate-specific curvature signal while preserving the optimizer's other components. This makes it difficult to establish whether the proposed curvature estimator is necessary for the observed improvements.
Questions for authors
-
Serving capacity and realistic speedups: How many serving GPUs are required to sustain q=512 curvature queries with the reported one-step or three-step staleness? Can the authors provide an end-to-end evaluation that includes communication overhead, total GPU-hours, and interference with inference workloads? A time-to-target comparison with SophiaG using its original ten-step curvature refresh schedule would also be informative.
-
Contribution of diagonal curvature: How does ZO-Curv compare with a scalar preconditioner based on the same curvature statistics? Can the authors report the coordinate-wise correlation between the estimated ZO curvature and the reference GNB diagonal? These results would help determine whether coordinate-specific curvature information drives the improvement.
-
Applicability of the theoretical guarantees: The convergence analysis requires the curvature distortion bound δ_T to be smaller than 1. However, substituting the experimental EMA coefficient β₂=0.99 into Equation (60) appears to yield a bound substantially greater than 1 at the evaluated model dimensions, even with optimistic constants. Can the authors clarify whether the theoretical guarantees are non-vacuous in the experimental regime?
3 comments
The serving-capacity accounting and the scalar-preconditioner ablation address my main concerns regarding practicality and the contribution of coordinate-wise curvature. I will raise my score accordingly.
Dear Reviewer,
Thank you for your positive feedback and for acknowledging our additional experiments and analysis. We sincerely appreciate your thoughtful review and constructive suggestions, which have helped us improve our work.
Rebuttal:
Thank you for your thoughtful review and for the concrete suggestions on systems accounting and ablations. We hope that our rebuttal further strengthens the soundness of our work. Below are the answers to your concerns and questions:
All numbers use q=512 and a curvature minibatch of 8 sequences. Capacity is converted to L40S cards using the peak bf16 throughput ratio H100 : L40S ≈2.73.
Serving capacity. One refresh costs 5.3 first-order steps, both in measured time and in FLOPs, at every model size. Staleness (s=1 or s=3) only hides latency. The pool must still finish one refresh per step, which requires about 116 L40S(8×5.3×2.73) for an 8× H100 node. A lower refresh frequency reduces this. With the ten-step schedule of the original Sophia (2 steps for communication, 8 for queries), 16 L40S suffice, and the training node never waits for curvature.
Communication. Per refresh, a 1.5B model sends a 3.0 GB bf16 snapshot and receives a 6.0 GB fp32 vector. At 100 Gbps, this round trip, including the all-reduce inside the pool, takes less than 2 s, below one training step (2.446 s). The average bandwidth with k=10 is about 2.9 Gbps.
End-to-end cost (1.5B, 20K steps):
We agree that the ~2× acceleration refers to reaching AdamW's final loss. Against ten-step SophiaG, the per-step gain is 6–12%, and we will state this explicitly.
We agree that the damping dominates the average: it exceeds the mean GNB entry by d/q≈2.4×105 at 125M. We therefore ran both controls you asked for.
Scalar-preconditioner control. At every step, we replace htZO by its coordinate mean over all parameters. Everything else is unchanged (q, γ, η, ρ, β2, momentum, schedule). This scalar preconditioner has exactly the same global scale as ZO-Curv and lacks only the coordinate-specific signal.
The scalar variant is worse by 5.7K, so global adaptive normalization alone does not explain the improvement. The coordinate information is concentrated on the high-curvature coordinates, whose GNB entries exceed the damping.
Coordinate-wise agreement. On the same checkpoint, minibatch and pseudo-labels, the single-step ZO estimate has a cosine of about 0.005 with the SophiaG GNB diagonal. It is also above the agreement of ZO gradients in LoRA fine-tuning (9.4×10−4), where ZO is known to be effective (Fig. 2(b)).
You are right that, as stated, δT≫1 in our regime. However, δT<1 comes only from writing the distortion symmetrically as (1±δT)Dt, and the argument does not need it. Descent and O(1/T) stationarity use only the lower side, Dt⪰mDt with m>0. This holds deterministically: hZO is a square, so the floor gives d^t,i≥ϵ0 and m=ϵ0/dmax for any β2, q, ε and d. The upper side enters only the constant of the local linear rate. We will restate the analysis with mTDt⪯Dt⪯MTDt, where mT=max{1−δT,ϵ0/dmax} and MT=1+δT. The results become descent for η≤mT/α, mintΨt=O(1/T), and a contraction of 1−αμDMTmT. After this change, no condition contradicts our setting (β2=0.99, q=512, ε=10−4, d up to 1.5×109). The guarantee is qualitative, as in Sophia's analysis. It explains why ZO curvature noise cannot break descent, but it does not predict the exact speed of convergence. We will make this scope explicit.
Thank you again for helping us improve the clarity and rigor of our work.