acceptodds
Under review as a conference paper at ICLR 2027

Breaking the Black Box: An Architecture-Specialized Preconditioning Optimizer for Attention Mechanism

Abstract

The increasing scale of deep learning models underscores the importance of efficient optimization. Existing matrix-based optimizers exploit parameter matrix structure through gradient statistics, activation statistics, or matrix transformations, but generally do not explicitly incorporate the analytical dependencies induced by specific network architectures. In this work, we propose STRATA, a structure-aware preconditioning optimizer for Transformer training. Moving from a black-box to a “gray-box” perspective, we explicitly exploit the coupling between the Query and Key projections in Attention to identify an architecture-specific structural curvature factor. We develop a two-stage approximation that transforms this activation-dependent quantity into a practical preconditioner constructed only from readily available model parameters and parameter gradients. We further characterize the approximation errors introduced by the two stages under a stylized model of the Attention distribution. Empirically, we evaluate STRATA on LLaMA-style pre-training from 130M to 1.3B parameters. Across the tested model scales, STRATA consistently improves optimization efficiency over AdamW and Muon and achieves end-to-end wall-clock speedups of 11%–31% relative to Muon, with negligible additional peak-memory overhead.

Primary area:
optimization
Keywords:
Attention-aware Optimization, Precondition, Pre-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4QtqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes STRATA, an optimizer dedicated to attention, that exploits attention structure to precondition Query and Key weight updates. Its derivation uses a Kronecker approximation of the expected generalized Gauss–Newton matrix, and its practical preconditioners are constructed from the Query, Key, and Value weights and the Value gradients. The Value weights and all remaining parameters are updated by a base optimizer, such as Muon.

Critical strengths

I would like to thank the authors. Exploiting the Query+Key coupling in attention to build a preconditioner is a novel and interesting idea. The method adds no peak memory and at most about 5% time per step (Table 1), and the paper includes several ablations.

Critical weaknesses

W1 - Results & Reproducibility. Figure 2 shows only a minimal improvement of N-STRATA vs Muon. Since there are many hyperparameters involved, there is a concern that this small improvement may come from hyperparameter choices rather than from the method. For example, Table 2 shows that Muon was tuned over only three learning rate pairs, and the learning rate of the STRATA Query/Key update is not reported at all. A finer search for Muon might close the gap? A convincing figure would show a few learning curves for Muon, each with a different learning rate, and a few curves for N-STRATA. This way we can see the sensitivity to the learning rate, and see that the improvement is really due to the N-STRATA method and not to a simply better hyperparameter choice. Additionally, no code is supplied, so I did not have the ability to reproduce the curve easily and check that the code matches to the paper. Code would make it easier to reproduce and validate the claims that indeed the method is superior to Muon. Optimizers for LLMs are also used for fine-tuning, but the paper evaluates only pretraining. It would help to know whether STRATA also works well for fine-tuning, or whether this is a limitation of the method.

W2 - Scale. I know that it may be computationally difficult at times, but experiments with larger models would make the case for the method more convincing. In Table 1, the wall‑clock speedups for the 130M, 350M, and 1.3B models are 31%, 23%, and 11%, so the benefit of switching to STRATA seems to decrease as the model grows. This further motivates the question of how it behaves in larger models. Since fine-tuning is much cheaper (fewer steps) than pretraining, fine-tuning an existing larger pretrained model with STRATA could be a feasible way to test it at larger scale.

W3 - Fragmentation. For the 1.3B model, the optimizer was changed from STRATA to N-STRATA. So users are expected to use a different optimizer at different model sizes? Can N-STRATA be used on the smaller models too, and how does it perform there?

W4 - Math assumptions. A small-scale model analysis or matrix–vector-product diagnostic would help assess the accuracy of the Kronecker approximation of the generalized Newton-Gauss matrix. This Kronecker approximation (Lemma 3.1) is the basis of the whole derivation of the preconditioner, yet it is never checked. Theorems 3.3 and 3.4 only analyze later approximation steps, and the only empirical check (Fig. 5(c)) covers only the last step.

W5 - Math. In Eq. (45), the authors replace the product of input entries by its upper bound inside the expectation. That step is only valid if the term multiplying it is never negative. That term can be negative, and the theorem's own assumptions force it to be negative on average.

Questions for authors
  • Line 1146: the AdamW learning rates are not listed in increasing order

  • Table 1 caption: please write "training loss" instead of "loss", since the step savings are measured on the training loss curves (Figs. 1 and 2).

  • What learning rate was used for the STRATA Query/Key update, and how was it tuned?

Reviewer_6Ca1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces STRATA -- a preconditioning optimiser for Transformers. Building on top of a base optimiser, it modifies the attention-specific update design to explicitly incorporate the coupling between Q and K projections. Motivated by a curvature structure introduced by attention, STRATA uses a two-stage approximation leveraged as a right-side preconditioner, resulting in an end-to-end speedup of 11-31%.

Critical strengths
  1. The paper is clearly written and is easy to follow and understand. The method itself is cleanly and solidly built upon Q-K coupling, the paper includes all necessary derivations. The second-order approximation is elegantly constructed and results in activation-free surrogate based on value parameter and its gradient. The introduced memory overhead from STRATA is negligible [Table 1], while wall-clock speedup is substantial, especially in the setups with 130-350M parameters.
Critical weaknesses
  1. While paper provides ablations on the design of STRATA [section G.2], I could not find comparisons when an existing preconditioner is applied to query and key weights specifically. Keeping the optimiser fixed for all other parameters would help to decouple the improvement coming directly from STRATA vs an improvement coming from a better conditioned update for attention parameters.

  2. The experimental scope is somewhat limited to one model family. While I sympathise with a limited computational budget, from the current set of experiments it is unclear whether it would generalise to other models beyond LLaMA.

Together, these two weaknesses leave uncertainty whether the gains depend on STRATA approximation or solely on having better attention weight conditioning, and how well STRATA generalises to other architectures. Existing experiments with AdamW and Muon show a good improvement, but more ablations would improve the paper evidence. If addressed by the authors, I would be happy to reconsider my score and engage in the discussion.

Questions for authors
  1. [clarification request] In Section 4.2, the authors introduce N-STRATA for 1.3B model to stabilise its training. According to Figure 5(b), N-STRATA achieves better validation loss. At the same time, judging from Table 1, it is unclear whether regular STRATA would give any advantage over base optimiser. Secondly, does normalisation give any advantage for models <1B or is this normalisation beneficial only for bigger models?

Remarks:

  • Citation in line 107 on page 2 spans across two pages.
Reviewer_kquvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper is motivated by the observation that existing matrix optimizers are architecture-agnostic. Methods such as Shampoo, K-FAC and Muon build preconditioners from generic gradient or activation statistics and ignore the analytical dependencies between parameters induced by the architecture. The authors therefore propose a "gray-box" view that explicitly exploits the coupling between the Query and Key projections in Transformer attention.

The core idea is to derive a curvature term from the attention structure and approximate it by an activation-free preconditioner. The authors first isolate, from the generalized Gauss-Newton matrix of WQW_Q, a right-side curvature factor that depends on the Key representation. They then apply two approximations: keeping the main term of the softmax gradient, and using the Value gradient to remove the dependence on activations. This yields a right-sided preconditioner that depends only on WVW_V, ∇L(WV)\nabla L(W_V) and WKW_K and is applied to the Query and Key updates, while all other parameters are still updated by Muon. Under specific assumptions, the authors bound the error of both approximation stages and provide a convergence analysis. In LLaMA pretraining from 130M to 1.3B parameters, STRATA saves 12–32% of training steps relative to Muon and achieves 11–31% end-to-end speedups with negligible memory overhead.

Critical strengths
  1. A new and interesting source of preconditioning information. Instead of building the preconditioner from generic gradient or activation statistics, the paper derives it from the analytical coupling between the Query and Key projections in attention. The idea of using the Value gradient to obtain an activation-free surrogate is creative, and the resulting preconditioner is only d×dd\times d per head and can be computed with a few Newton–Schulz iterations. I find this "gray-box" direction worth exploring, because architecture-specific structure is largely unused by current matrix optimizers.

  2. Practically relevant results and a broad set of ablations. The method is evaluated on LLaMA pretraining up to 1.3B parameters and reports end-to-end wall-clock speedups over Muon, not only step counts, with essentially no additional peak memory. The paper also reports zero-shot downstream results and ablations on bilateral versus right-sided preconditioning, the preconditioning order, momentum, column-wise normalization, and the approximation residual during training. If the gains hold up, this would be a cheap and useful addition to Muon-based training.

Critical weaknesses

Technical issues

The following issues affect my assessment of whether the theoretical analysis supports the claims made in the paper.

  1. The approximation chain from the structural curvature to the final preconditioner is not justified.

    • Theorem 3.3. The bound ∥M−G∥F≤O(R(1−α)/2)∥M∥F\Vert M-G\Vert_F\le O(R^{(1-\alpha)/2})\Vert M\Vert_F never shows that the error is small: the bound on the relative error is at best O(1)O(1) (for α=1\alpha=1) and grows with RR for α<1\alpha<1.
    • Theorem 3.4. The assumptions require XX to be independent of the other random variables, yet both the softmax output SS and Q=∂L∂ZWV⊤Q=\frac{\partial L}{\partial Z}W_V^\top are functions of XX. The proof also bounds E[Δab]\mathbb{E}[\Delta_{ab}] and then uses it as a bound on E∣Δab∣\mathbb{E}\vert\Delta_{ab}\vert (Eqs. (48)–(49)), which does not follow. The final bound O(δ2D0D2R1/2)O(\delta^2D_0D^2R^{1/2}) is not compared with ∥Ktarget∥F\Vert K_{\text{target}}\Vert_F, so it says nothing about the relative accuracy of the surrogate.
    • Lemma 3.1 (minor). The factorization E[K⊤G⊤GK]⊗E[X⊤X]\mathbb{E}[K^\top G^\top GK]\otimes\mathbb{E}[X^\top X] is essentially a K-FAC-style approximation [1] with weight sharing across tokens [2]. Besides the stated independence assumption, it also requires neglecting cross-token terms, which is not mentioned. The lemma also uses the true Fisher, with labels sampled from the model, while the implementation uses gradients with respect to the true labels, i.e., an empirical-Fisher-type quantity, which can behave differently [3].

    As a result, there is no theoretical evidence that HQH_Q approximates the curvature factor it is derived from. The empirical check in Fig. 5(c) only states that the scale-aligned residual "remains consistently controlled", without indicating what level of residual would be acceptable or how it compares with a trivial baseline (e.g., an identity preconditioner).

  2. The convergence result does not cover the algorithm. Theorem C.2 sets the step size using the minimum of λiγ\lambda_i^\gamma over the whole future trajectory, analyzes a single matrix with an exact H−γH^{-\gamma}, and does not include the actual setting, in which only WQW_Q and WKW_K are preconditioned, the other parameters use Muon, H−1/2H^{-1/2} is approximated by five coupled Newton–Schulz steps, and column-wise normalization is applied at 1.3B scale. I understand this as a simplification commonly made in non-convex optimization analyses, and I personally consider it a minor issue, but the authors should acknowledge and clarify it in the paper.

Experimental issues

The following issues affect my assessment of whether the experimental results support the claims made in the paper.

  1. The absolute gains are small and shrink with scale. The headline numbers measure the training step at which STRATA first reaches Muon's final training loss, within runs that use the same cosine schedule decaying to zero. Since the loss curves are very flat near the end of training, this metric converts a small loss gap into a large proportion of saved steps. In absolute terms, the improvement in validation loss is small, for example about 0.016 at 130M (Table 3). The gains also shrink with scale, from 32% and 30% of steps at 130M and 350M to 12% at 1.3B, and the 1.3B result additionally relies on column-wise normalization. This makes it difficult to judge how much benefit to expect at larger scales.

  2. The source of the gains is unclear. At 1.3B the method requires an additional column-wise normalization (N-STRATA), taken from prior work [4], and the gain shrinks to 12% of steps. Appendix G.2.4 only compares STRATA with N-STRATA; there is no ablation that applies the same column-wise normalization to the Q/K gradients without the structural preconditioner, i.e., Ncol(∇L(W))\mathcal{N}_{\mathrm{col}}(\nabla L(W)) instead of Ncol(∇L(W)H−γ)\mathcal{N}_{\mathrm{col}}(\nabla L(W)H^{-\gamma}). Without it, it is unclear how much of the improvement at this scale comes from the derived structural preconditioner rather than from the normalization. In addition, the claim that STRATA "guides the model toward flatter local minima with superior generalization" is not supported by any flatness or generalization measurement; a lower validation loss alone does not establish it.

References

[1] James Martens and Roger Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In International Conference on Machine Learning (ICML), 2015.

[2] Runa Eschenhagen, Alexander Immer, Richard E. Turner, Frank Schneider, and Philipp Hennig. Kronecker-factored approximate curvature for modern neural network architectures. In Advances in Neural Information Processing Systems (NeurIPS), 2023.

[3] Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical Fisher approximation for natural gradient descent. In Advances in Neural Information Processing Systems (NeurIPS), 2019.

[4] Athanasios Glentis, Jiaxiang Li, Andi Han, and Mingyi Hong. A minimalist optimizer design for LLM pretraining. arXiv preprint arXiv:2506.16659, 2025.

Questions for authors
  1. Does HQH_Q actually capture the curvature it is derived from? Fig. 5(c) compares KapproxK_{\text{approx}} with Ktarget=X⊤M⊤MXK_{\text{target}}=X^\top M^\top MX, which only checks the second approximation stage; it does not cover the first stage (G→MG\to M) or the full factor E[K⊤G⊤GK]\mathbb{E}[K^\top G^\top GK], and a scale-aligned residual does not show whether the dominant eigenspaces are aligned. Could the authors provide more direct evidence on a small model? For example, (a) compare the eigenspaces of HQH_Q with those of the exact factor E[K⊤G⊤GK]\mathbb{E}[K^\top G^\top GK], and (b) train with the exact factor and check whether STRATA matches its performance. This would show whether the gains come from the claimed mechanism.

  2. How were the hyperparameters of STRATA tuned? Appendix G.1 states that STRATA shares the Muon and AdamW learning rates of the Muon baseline for the other parameters, and Appendix G.2.6 fixes the damping coefficient to α~=0.01\tilde\alpha=0.01. Was the learning rate for WQW_Q and WKW_K tuned separately, how was α~\tilde\alpha chosen, and what was the total tuning budget compared with the baselines?

  3. Why are the WiC scores identical across model scales? In Tables 4 and 5, all three optimizers obtain exactly the same WiC accuracy at 350M and 1.3B (AdamW 50.16, Muon 45.77, STRATA 51.25). Since WiC is one of the main sources of STRATA's higher average downstream score, could the authors check whether this is a reporting error or whether the models produce degenerate predictions on this task?

If the authors can provide reasonable explanations and supporting evidence for these questions, and revise or clarify the points raised in the weaknesses, I would be happy to raise my score.

1 comment

Dear authors,

I would appreciate a response to my review. If my main concerns are addressed, I will consider raising my score; if they remain unaddressed at the end of the discussion period, I will have to maintain my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.