- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes STRATA, an optimizer dedicated to attention, that exploits attention structure to precondition Query and Key weight updates. Its derivation uses a Kronecker approximation of the expected generalized Gauss–Newton matrix, and its practical preconditioners are constructed from the Query, Key, and Value weights and the Value gradients. The Value weights and all remaining parameters are updated by a base optimizer, such as Muon.
Critical strengths
I would like to thank the authors. Exploiting the Query+Key coupling in attention to build a preconditioner is a novel and interesting idea. The method adds no peak memory and at most about 5% time per step (Table 1), and the paper includes several ablations.
Critical weaknesses
W1 - Results & Reproducibility. Figure 2 shows only a minimal improvement of N-STRATA vs Muon. Since there are many hyperparameters involved, there is a concern that this small improvement may come from hyperparameter choices rather than from the method. For example, Table 2 shows that Muon was tuned over only three learning rate pairs, and the learning rate of the STRATA Query/Key update is not reported at all. A finer search for Muon might close the gap? A convincing figure would show a few learning curves for Muon, each with a different learning rate, and a few curves for N-STRATA. This way we can see the sensitivity to the learning rate, and see that the improvement is really due to the N-STRATA method and not to a simply better hyperparameter choice. Additionally, no code is supplied, so I did not have the ability to reproduce the curve easily and check that the code matches to the paper. Code would make it easier to reproduce and validate the claims that indeed the method is superior to Muon. Optimizers for LLMs are also used for fine-tuning, but the paper evaluates only pretraining. It would help to know whether STRATA also works well for fine-tuning, or whether this is a limitation of the method.
W2 - Scale. I know that it may be computationally difficult at times, but experiments with larger models would make the case for the method more convincing. In Table 1, the wall‑clock speedups for the 130M, 350M, and 1.3B models are 31%, 23%, and 11%, so the benefit of switching to STRATA seems to decrease as the model grows. This further motivates the question of how it behaves in larger models. Since fine-tuning is much cheaper (fewer steps) than pretraining, fine-tuning an existing larger pretrained model with STRATA could be a feasible way to test it at larger scale.
W3 - Fragmentation. For the 1.3B model, the optimizer was changed from STRATA to N-STRATA. So users are expected to use a different optimizer at different model sizes? Can N-STRATA be used on the smaller models too, and how does it perform there?
W4 - Math assumptions. A small-scale model analysis or matrix–vector-product diagnostic would help assess the accuracy of the Kronecker approximation of the generalized Newton-Gauss matrix. This Kronecker approximation (Lemma 3.1) is the basis of the whole derivation of the preconditioner, yet it is never checked. Theorems 3.3 and 3.4 only analyze later approximation steps, and the only empirical check (Fig. 5(c)) covers only the last step.
W5 - Math. In Eq. (45), the authors replace the product of input entries by its upper bound inside the expectation. That step is only valid if the term multiplying it is never negative. That term can be negative, and the theorem's own assumptions force it to be negative on average.
Questions for authors
-
Line 1146: the AdamW learning rates are not listed in increasing order
-
Table 1 caption: please write "training loss" instead of "loss", since the step savings are measured on the training loss curves (Figs. 1 and 2).
-
What learning rate was used for the STRATA Query/Key update, and how was it tuned?
Dear authors,
I would appreciate a response to my review. If my main concerns are addressed, I will consider raising my score; if they remain unaddressed at the end of the discussion period, I will have to maintain my current score.