- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets two weaknesses of Leon (Jiang et al., 2026), the projection-free FTRL variant of one-sided Shampoo: it needs an extra scalar δ tuned by hand, and its analysis assumes uniformly bounded gradients. The authors first prove a gradient-stability property of the preconditioning operator (Lemma 4), which lets δ be taken arbitrarily small, removes the hyperparameter and yields dimension-independent guarantees beyond bounded gradients. They then extend UniXGrad-style accumulation of squared gradient differences from scalar step sizes to general preconditioners (Lemma 7), giving a Nesterov-accelerated projection-free method with O(ε-2/(1+3ν)) for ν-Holder smooth convex problems and O(ε-2) in the stochastic convex case. The paper is entirely theoretical.
Critical strengths
1Lemma 4 is the real technical core. The bound ‖∇Ψk+1(m) - ∇Ψk(m)‖2 ≤ η2 (‖√(Sk+1)‖tr - ‖√(Sk)‖tr) ties the change of the regularizer directly to the change of the gradient, which is exactly what cancels the negative Bregman terms that Lemma 3 introduces. The logic closes cleanly; it does not read as an estimate forced to fit. 2Generalizing UniXGrad's gradient-difference accumulation to general preconditioners (Lemma 7) is useful beyond this paper: the earlier analyses of Kavis et al. and Rodomanov et al. cover only scalar step sizes, and bounding ‖∇fk(xk+1) - ∇fk(xk)‖ through the Bregman divergence is the step that makes the extension work. 3The unification is substantive rather than cosmetic. Assumption 1 places AdaGrad, one-sided Shampoo and scalar step sizes in one framework, and Theorem 5 matches the known optimal rates at both ν = 1 and ν = 0, with the stochastic term σ R / √(K+1) cleanly separated from the δ term. 4The proofs are written to be checked: every step in Appendix B names the lemma, the Lowner-Heinz inequality or the differentiability property of the trace square root that it uses.
Critical weaknesses
1.There are no experiments at all. "Practical" appears three times in the abstract, "first practical variant" once, and the paper contains not a single numerical result - not even a toy matrix problem. For a submission whose stated purpose is to remove a practical nuisance parameter, this is the central problem. Even ten lines of synthetic matrix optimization against Leon and USGD, with per-iteration timing (the second gradient sample in Algorithm 2 is not free), would change the review substantially. 2.The novelty is repair rather than construction. The algorithmic skeleton - FTRL with Ψ*k, the fk construction for Nesterov, UniXGrad-style accumulation - is inherited. The increment is one stability lemma plus more careful bookkeeping. Theorem 3 has the same functional form as Theorem 3.2 of Jiang et al., only with weaker hypotheses, and Theorem 5's rate is the known optimum for this family. The "what was previously impossible" column is not filled. 3.The δ story needs to be sharpened. The bounds in Theorems 1 and 3 both contain δ R dim(X), which comes from the initialization S-1 = δ2 I, since ‖√(S0)‖tr = δ dim(X); sending δ to zero removes it. So where exactly did the original analysis get stuck? Which inequality required δ to be tied to the gradient bound? Without an answer, a reader may conclude that the dimension dependence was an accounting artifact rather than a real obstacle - and that is precisely the question that determines what this paper is worth. 4.The bridge from the assumptions to deep learning is narrow. Assumption 2(ii) and 4(ii) measure second moments in a fixed B, and Assumption 3 requires Holder smoothness with respect to ‖.‖B, while the algorithm runs an adaptive Sk. Section 3.3 offers E[G GT] ≤ C2 with C of low effective rank as the motivating model, but no evidence is given that transformer gradients live there. The rescaling argument for the √(n) factors in G and ρ is also loose: the rescaling changes ε and γ as well, so Definition 1 should be re-derived rather than declared invariant. 5.The practical question is left unanswered: what δ should one use? At δ = 0, Sk is singular and the pseudoinverse is needed (Lemma 9). But in bf16, δ = 1e-10 - the value people actually use in Adam-like code - will likely wreck the conditioning of Sk1/2. Is the Adam analogy with ε = 1e-8 valid here? 6.The noise injected by line 10 of Algorithm 2 is never discussed. That line accumulates out(ĝk+1 - gk) from two independent samples. The second moment of the difference carries a 2 σ2 noise term, so Sk ends up driven largely by noise, which weakens exactly the adaptivity the method preserves. The theory absorbs this into σ, but the practical consequence - the reason UniXGrad is often considered fragile - is left out.
Questions for authors
1.Can you add numerical experiments, even on a synthetic matrix problem, against Leon, USGD and a Muon-style method? 2.At which step of Jiang et al.'s proof does δ have to be coupled to the gradient bound? 3.What δ do you recommend in practice, is it stable in bf16, and how far is it from the pseudoinverse (δ = 0) version? 4.What is the explicit update of Algorithm 2 in the matrix case, and how does it differ line by line from Algorithm 1? 5.How does the noise inflation from accumulating gradient differences affect adaptivity, and is there a correction for it?