- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the implicit bias of "momentum-free factorwise polar flow, a continuous-time idealization of per-head Muon" for a single attention head. In particular, the guarantees in this paper concern the idealized flow rather than practical Muon training.
With existing attention max-margin formulations and spectral optimization geometry, it formulates a width-h max-margin problem and its dual Ky Fan norm relaxation.
In Theorem 4.1, they first fix the direction of the negative loss gradient. If the initial key and query matrices are sufficiently aligned with this gradient, their normalized product converges to a matrix formed from the gradient's leading singular vectors.
In Theorem 5.2, the paper then allows the gradient direction to change during training and study the margin, which is the score of a prescribed target token minus that of a competing token. The minimal gap across all queries and competitors converges to the optimal value induced by the width and norm constraints, given a trajectory-dependent condition that ensures enough accumulated progress toward optimum. This divergent integrated coercivity condition is an extra assumption on the QK matrices and gradient, while cannot be guaranteed by arbitrary network initializations.
For exact softmax attention with prescribed target tokens and fixed representations, value matrix V, and readouts, Theorem 6.2 establishes the convergence of the product W in a local region under suitable certificate and initialization conditions.
In Theorem 6.3, with additional strict-complementarity and dual-nondegeneracy conditions, they prove the convergence of the normalized gradient weights.
The numerical examples illustrate the convergence results, initialization sensitivity, and the distinction between product convergence and weight identification.
Critical strengths
They provide a technically sound characterization of factorwise polar flow in a single attention head.
The authors provide a careful analysis separately for margin, product, and gradient-weight convergence. They clarify what they can derive from which set of assumptions.
Critical weaknesses
My main concern is the significance of the theoretical extension.
- The setting and the message conveyed are not very practical.
The paper studies single-head, fixed-representation, momentum-free factorwise polar flow with exact polar updates. Although they acknowledge it is different from the per-head Muon setting in the motivation, the reviewer feels they do not sufficiently establish which new behavior this particular idealization can explain, considering existing attention max-margin and spectral-factorization analyses leveraged in the paper can already explain some important training behavior.
The trajectory divergent integrated coercivity condition is strong and has limited use for actual training. The paper does not establish how the required condition can be inferred from the data and initialization.
Similarly, the local assumptions already place training in a good regime. Theorems 6.2 and 6.3 use a certificate to identify the optimal product and require suitable initialization in its neighborhood. The examples establish that the assumptions can hold, while practitioners will doubt whether they will meet those assumptions in general.
- The numerical examples lack complete reproduction code. There is no executable reproduction code or explicit promise to open-source a complete public release. In particular, the sentence "the selected sentences, feature tensors, and readouts needed to reconstruct those examples are not included in the source archive" in the Reproducibility Statement is confusing; what is the source archive?
Questions for authors
Please clarify whether the complete reproduction code will be released upon publication, which I believe is essential for the community to study and develop new theory/methods.
If possible, please follow the ICLR instructions to provide an anonymous repo during review, hopefully including the scripts for reproducing each figure and table.
If a code release is not currently possible, please provide at least a README identifying the scripts, required inputs, and execution steps.
4 comments
Our response to Reviewer 3skY (2/2)
Our references
- Ataee Tarzanagh, D., Li, Y., Thrampoulidis, C., and Oymak, S. Transformers as support vector machines. NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023a.
- Ataee Tarzanagh, D., Li, Y., Zhang, X., and Oymak, S. Max-margin token selection in attention mechanism. NeurIPS, 2023b.
- Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.
- Du, S. S., Hu, W., and Lee, J. D. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. NeurIPS, 2018.
- Kang, C., Yun, J., Shin, B., Cho, Y., and Yun, C. Uniform spectral growth and convergence of Muon in LoRA-style matrix factorization. arXiv:2602.06385, 2026.
- Rhee, M., Simon, J., and Karkada, D. Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamics. arXiv:2606.30509, 2026.
- Sheen, H., Chen, S., Wang, T., and Zhou, H. H. Implicit regularization of gradient flow on one-layer softmax attention. arXiv:2403.08699, 2024.
- Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. EMNLP, 2013.
Thank you for clarifying the theoretical motivation. The example showing that two factorizations of the same key--query product can give different update directions helped readers understand why the factors need to be studied separately. I also appreciate the code and reproduction instructions, which should make it easier for others to check and build on the results.
As a practitioner outside this area of theory, my remaining reservation concerns what the results let us predict before training.
- For a new problem with a changing gradient, Theorem 5.2 still gives me no way to tell from the data and initialization whether its trajectory condition will hold.
- For Theorems 6.2 and 6.3, the new experiments show that randomly initialized key and query factors can approach the certified target on two instances with frozen BERT features. This strengthens the evidence beyond initialization near the target. It remains open whether, when training a causal language model from scratch, with representations evolving throughout training, it would enter the regime described by the local theory.
I can confirm my concerns are partially resolved and would maintain my original score, while leaning more toward acceptance if the AC judges the theoretical advance sufficient in its own right.
Our follow-up to reviewer 3skY
We appreciate your thoughtful follow-up and understand your remaining concerns. Our work aims to establish a theoretical foundation for understanding the implicit bias of factorwise spectral optimization in attention. Extending these results to practical Muon training with evolving representations and arbitrary initializations remains an important open problem that we leave for future work.
Thank you again for your constructive feedback and consideration.
Our response to Reviewer 3skY (1/2)
We want to thank you for the careful summary and for the clear questions. We start with significance, since you mention it as your main concern.
1. About significance. We agree that our setting is idealized, and we want to explain what this idealization isolates: the role of the factorization itself. Existing attention max-margin results either parameterize W directly or use Euclidean gradient flow on the factorized product (Ataee Tarzanagh et al., 2023a; 2023b; Sheen et al., 2024). Under Euclidean gradient flow, K⊤K−Q⊤Q is conserved (Du et al., 2018), so from a balanced initialization the product follows its own flow. In our momentum-free, exact-polar idealization of per-head Muon, this quantity is not conserved, and two factorizations of the same product can induce different product velocities (Appendix F.1). Factorwise spectral updates have been studied for squared-loss matrix factorization (Kang et al., 2026; Rhee et al., 2026), where the data fix the target product. In attention, the gradient direction moves with the attention weights, and the question becomes which max-margin solution is selected. Knowing the max-margin geometry does not answer it: Appendix F.1 gives an attention-realizable stationary point whose margin-to-optimum ratio a2/(1+a2) tends to zero. Our results give conditions under which the dynamics reach the optimum: a sufficient alignment criterion for a fixed gradient (Theorem 4.1), a dissipation bound for a moving gradient (Theorem 5.2), and local convergence of the normalized product in exact softmax attention without assuming that the product or the gradient weights converge (Theorem 6.2). Please also see points 2 and 3 of our general response.
2. About the condition Ξ(t)→∞. We agree that Ξ(t)→∞ is a condition on the trajectory, and that it cannot be checked from the data before training. We use Theorem 5.2 to isolate what the factor geometry contributes when the gradient moves arbitrarily. Before the theorem, the revision explains what positive separation η(t) means (rev. l.329–330), and after the theorem it gives two sufficient conditions (rev. l.359–360): η(t)≥η0>0 for large t, or 2∫t0∞∣Z′∣op<λmin(G(t0))−σh+1(Z(t0)) (Eq. (C.9)). They make the assumption concrete, and they are still trajectory conditions. Section 6 then works with local conditions, and Appendix F.2 gives a convergent example with Ξ(∞)<∞.
3. About the local assumptions in Section 6. We understand the concern that these assumptions place training in a good regime, and we want to separate two kinds of conditions. The certificate conditions (MM), (SC), and (ND) are finite-dimensional checks once a candidate dual solution p⋆ and its active set are available: an SVD of Δ⋆, the margins of W⋆=polarh(Δ⋆), and the Hessian of p↦∣Δ(p)∣(h) at p⋆. The initialization requirement (a small initial energy and a large enough logit scale) is a substantive restriction, and we do not give an a priori guarantee that an arbitrary initialization meets it. A global softmax-attention solution with positive prefactors, uniformly regular base weights, and Ξ(t)→∞ eventually meets these thresholds under (MM) (end of Appendix E), and Theorem 6.3 then still requires (TC), (SC), (ND), and ∣S∣≥2. The revised remark after Theorem 6.2 states that this is a local guarantee (rev. l.411–412), and Section 7 states that the results give no computable basin of attraction.
4. About code. We apologize for the misleading wording in the original Reproducibility Statement. Our initial submission contained only the manuscript PDF, without the experimental code or feature tensors. We have now uploaded the experimental code as a zip file in the supplementary material. Its README and
RUN_PAPER.mdprovide instructions for reproducing the figures and tables. In particular,python reproduce.py verifychecks the saved results,python reproduce.py smokeruns short fresh integrations, andpython reproduce.py run --experiment allreruns the synthetic experiments. For the frozen-feature examples (Appendix G.3), the archive includes the inputs of case A and the certificate matrices of all three cases, together with a script to reconstruct cases B and C from public BERT and SST-2 releases. We have updated the Reproducibility Statement and Appendix G.3 accordingly.Thank you again for the helpful suggestions and we are happy to discuss any further question.