- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a Block-Heterogeneous Smoothness framework, which includes definitions of smoothness and a weighted stationarity measure. Under this framework, it studies block-preconditioned GD compared to GD. The theoretical analysis gives upper bounds for both Block-PGD and GD, together with a quadratic lower-bound example establishing necessary linear dependence on block heterogeneity for GD under the weighted stationarity measure. The paper additionally extends the analysis to softmax regression, providing a local result, and to square-root diagonal preconditioning. The experimental part compares SGD with Adam on synthetic softmax regression, compares Block-PGD with GD on a separable quadratic function, and includes an empirical curvature analysis for a small language model.
Critical strengths
The theoretical analysis is well motivated, and its proofs are short and easy to check. The developed framework provides a useful perspective on optimization under block heterogeneity. The extension to softmax regression gives the theoretical results a concrete application beyond quadratic problems.
Critical weaknesses
-
The softmax experiments could connect more directly to the theory. The theory analyses Block-PGD and GD, while Section 7.2 compares Adam with SGD and the comparison of GD with Block-PGD is only shown on a quadratic task. Extending the softmax regression comparison would help reinforce theoretical insights. A comparison of SGD with block-preconditioned SGD would also be appropriate.
-
The general separation in Corollary 5 has a limited scope. Theorem 4 and Proposition 11 establish a separation on quadratic problems. For general nonconvex objectives, the paper compares upper bounds without a matching lower bound. Although the paper acknowledges this limitation, it restricts the scope of the theoretical separation. Extending the experiments in Section 7.1 and Figure 2 to include a comparison of GD and Block-PGD on nonquadratic, nonconvex objectives with known block-smoothness constants would strengthen the empirical evidence.
Presentation issues
- The repeated caveats, for example in lines 70–71, 138, 253–254, and 323, interrupt the exposition. Consolidating them into the discussion or limitations section would improve the presentation.
- In Theorem 4, the general indexing with is unnecessary for the two-block construction. Using and would simplify the statement.
- The title of Section 5 "Application to Transformers" overstates its scope. The paper should either discuss how the analysis applies beyond the softmax output head or use a narrower section title.
Questions for authors
-
Could you include a comparison of SGD with block-preconditioned SGD on the softmax regression task?
-
Could you extend the experiments in Section 7.1 to nonquadratic, nonconvex objectives with known block-smoothness constants?
We would like to add a few clarifications in response to the questions above.
Relation to the closest theory. Gluon's deterministic Euclidean specialization with its gradient-dependent term set to zero recovers the inverse-smoothness update and weighted-gradient descent bound. We now credit that result explicitly and do not claim the shared descent argument or coordinate change as new. Liu et al. study AdaGrad with an adaptive, data-dependent geometry under anisotropic smoothness, while Xie et al. study Adam and blockwise updates in ℓ∞ geometry. Our comparison instead fixes declared block scales, uses one common block quadratic majorizer, and evaluates ∑k∥∇kf∥2/Lk. The paper's additional results are the matched scalar-GD comparison, the quadratic examples showing necessary linear heterogeneity dependence under loss and ordinary-gradient criteria, the stochastic comparison, and the explicit scale-misspecification analysis. The joint upper model allows cross-block interactions, but we do not present that fact alone as the novelty.
Figure 3(c) and the softmax intercept. The original panel did not plot smoothness. For each class, it sums the squared norm of the class-conditioned minibatch gradient and then normalizes all class totals by their maximum. It is therefore a normalized class-conditioned gradient-energy statistic, not a smoothness estimate. The revision separates this descriptive statistic from the Hessian-based curvature diagnostic. The training model is affine and includes class-specific intercepts. Under the shared-covariance Gaussian construction, these intercepts absorb the unequal class priors, so the population posterior remains representable. The separate legacy diagnostic omitted the intercept and is not used to establish well-specification or curvature scaling.
Hitting times. The revised audit counts actual updates from t=0 and keeps non-hits censored. At heterogeneity 1,000 and absolute weighted-squared-gradient target 10−3, reconstructed GD does not hit within 3,000 recorded iterates, while Block-PGD hits after one update. We therefore assign no finite ratio to this cell; the previous substitution of the iteration budget is removed.
Time-resolved second-moment audit. The earlier GPT-2 reruns measured only the final checkpoint. We added one declared trajectory at steps 25, 50, 100, 150, 200, 250, and 300. At each checkpoint we evaluated 11,008,896 architecture coordinates and 11,136 selected output-row coordinates. Among the architecture coordinates, 6,336 have zero bias-corrected second moment at every checkpoint. Across the seven checkpoints, the 1st percentile of the moment-to-curvature ratio ranges from 2.54×10−13 to 3.96×10−12, while the 99th percentile ranges from 3.51×10−7 to 1.79×10−6. For the selected output rows, the positive max/min spread ranges from 7.66×104 to 4.76×106. The zero moments rule out a positive uniform lower constant on the measured architecture coordinates, and the remaining ratios span many orders of magnitude. Thus the measurements do not verify useful constants a,b>0 satisfying aLk≤vt,i≤bLk across coordinates and steps. The conditional square-root corollary therefore should not be used to explain full-model Adam in this experiment. We will include the single-run protocol and raw record in the supplement and keep the Transformer claim at this diagnostic level.
Scope of the conclusion. When valid block scales are known, block preconditioning can improve on scalar-step GD. The GPT-2 measurements do not show that Adam learns the uniform scales required by the conditional corollary.