acceptodds
Under review as a conference paper at ICLR 2027

Implicit Bias of Factorwise Spectral Optimization in Self-Attention

Abstract

In self-attention, logits depend on the product , yet per-head Muon, used in recent LLMs, updates and separately. The training dynamics therefore depend on the factorization, not only on the product and its gradient. We ask which normalized product these updates select, and study this question through the momentum-free factorwise polar flow, a continuous-time idealization of per-head Muon. For a fixed gradient, sufficiently aligned factors keep full-rank gradients and, above the largest omitted singular value, converge to the leading singular directions. When factor alignment accumulates without bound, the minimum margin of the normalized product converges to the optimal width- margin characterized by the Ky Fan norm. For exact softmax attention, when each query has a prescribed target key, we prove local convergence of the product near the target, while the factors approach an orbit of common rotations. If the gradient weights on these target–competitor comparisons are also locally identifiable, they converge as well, at a polynomial rate in optimizer time. Numerical experiments illustrate that the product can converge even when its limit does not determine the gradient weights.

Primary area:
learning theory
Keywords:
Implicit Bias, Self-Attention, Muon, Matrix Factorization, Ky Fan Norm, Max-Margin Optimization, Spectral Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3skYOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the implicit bias of "momentum-free factorwise polar flow, a continuous-time idealization of per-head Muon" for a single attention head. In particular, the guarantees in this paper concern the idealized flow rather than practical Muon training.

With existing attention max-margin formulations and spectral optimization geometry, it formulates a width-h max-margin problem and its dual Ky Fan norm relaxation.

In Theorem 4.1, they first fix the direction of the negative loss gradient. If the initial key and query matrices are sufficiently aligned with this gradient, their normalized product WW converges to a matrix formed from the gradient's hh leading singular vectors.

In Theorem 5.2, the paper then allows the gradient direction to change during training and study the margin, which is the score of a prescribed target token minus that of a competing token. The minimal gap across all queries and competitors converges to the optimal value induced by the width and norm constraints, given a trajectory-dependent condition that ensures enough accumulated progress toward optimum. This divergent integrated coercivity condition is an extra assumption on the QK matrices and gradient, while cannot be guaranteed by arbitrary network initializations.

For exact softmax attention with prescribed target tokens and fixed representations, value matrix V, and readouts, Theorem 6.2 establishes the convergence of the product W in a local region under suitable certificate and initialization conditions.

In Theorem 6.3, with additional strict-complementarity and dual-nondegeneracy conditions, they prove the convergence of the normalized gradient weights.

The numerical examples illustrate the convergence results, initialization sensitivity, and the distinction between product convergence and weight identification.

Critical strengths

They provide a technically sound characterization of factorwise polar flow in a single attention head.

The authors provide a careful analysis separately for margin, product, and gradient-weight convergence. They clarify what they can derive from which set of assumptions.

Critical weaknesses

My main concern is the significance of the theoretical extension.

  • The setting and the message conveyed are not very practical.

The paper studies single-head, fixed-representation, momentum-free factorwise polar flow with exact polar updates. Although they acknowledge it is different from the per-head Muon setting in the motivation, the reviewer feels they do not sufficiently establish which new behavior this particular idealization can explain, considering existing attention max-margin and spectral-factorization analyses leveraged in the paper can already explain some important training behavior.

The trajectory divergent integrated coercivity condition is strong and has limited use for actual training. The paper does not establish how the required condition can be inferred from the data and initialization.

Similarly, the local assumptions already place training in a good regime. Theorems 6.2 and 6.3 use a certificate to identify the optimal product and require suitable initialization in its neighborhood. The examples establish that the assumptions can hold, while practitioners will doubt whether they will meet those assumptions in general.

  • The numerical examples lack complete reproduction code. There is no executable reproduction code or explicit promise to open-source a complete public release. In particular, the sentence "the selected sentences, feature tensors, and readouts needed to reconstruct those examples are not included in the source archive" in the Reproducibility Statement is confusing; what is the source archive?
Questions for authors

Please clarify whether the complete reproduction code will be released upon publication, which I believe is essential for the community to study and develop new theory/methods.

If possible, please follow the ICLR instructions to provide an anonymous repo during review, hopefully including the scripts for reproducing each figure and table.

If a code release is not currently possible, please provide at least a README identifying the scripts, required inputs, and execution steps.

4 comments
AuthorsOpenReview ↗

Our response to Reviewer 3skY (1/2)

We want to thank you for the careful summary and for the clear questions. We start with significance, since you mention it as your main concern.

1. About significance. We agree that our setting is idealized, and we want to explain what this idealization isolates: the role of the factorization itself. Existing attention max-margin results either parameterize WW directly or use Euclidean gradient flow on the factorized product (Ataee Tarzanagh et al., 2023a; 2023b; Sheen et al., 2024). Under Euclidean gradient flow, K~⊤K~−Q~⊤Q~\widetilde K^\top\widetilde K-\widetilde Q^\top\widetilde Q is conserved (Du et al., 2018), so from a balanced initialization the product follows its own flow. In our momentum-free, exact-polar idealization of per-head Muon, this quantity is not conserved, and two factorizations of the same product can induce different product velocities (Appendix F.1). Factorwise spectral updates have been studied for squared-loss matrix factorization (Kang et al., 2026; Rhee et al., 2026), where the data fix the target product. In attention, the gradient direction moves with the attention weights, and the question becomes which max-margin solution is selected. Knowing the max-margin geometry does not answer it: Appendix F.1 gives an attention-realizable stationary point whose margin-to-optimum ratio a2/(1+a2)a^2/(1+a^2) tends to zero. Our results give conditions under which the dynamics reach the optimum: a sufficient alignment criterion for a fixed gradient (Theorem 4.1), a dissipation bound for a moving gradient (Theorem 5.2), and local convergence of the normalized product in exact softmax attention without assuming that the product or the gradient weights converge (Theorem 6.2). Please also see points 2 and 3 of our general response.

2. About the condition Ξ(t)→∞\Xi(t)\to\infty. We agree that Ξ(t)→∞\Xi(t)\to\infty is a condition on the trajectory, and that it cannot be checked from the data before training. We use Theorem 5.2 to isolate what the factor geometry contributes when the gradient moves arbitrarily. Before the theorem, the revision explains what positive separation η(t)\eta(t) means (rev. l.329–330), and after the theorem it gives two sufficient conditions (rev. l.359–360): η(t)≥η0>0\eta(t)\ge\eta_0>0 for large tt, or 2∫t0∞∣Z′∣op<λmin⁡(G(t0))−σh+1(Z(t0))2\int_{t_0}^\infty\\|Z'\\|_{\rm op}<\lambda_{\min}(G(t_0))-\sigma_{h+1}(Z(t_0)) (Eq. (C.9)). They make the assumption concrete, and they are still trajectory conditions. Section 6 then works with local conditions, and Appendix F.2 gives a convergent example with Ξ(∞)<∞\Xi(\infty)<\infty.

3. About the local assumptions in Section 6. We understand the concern that these assumptions place training in a good regime, and we want to separate two kinds of conditions. The certificate conditions (MM), (SC), and (ND) are finite-dimensional checks once a candidate dual solution p⋆p^\star and its active set are available: an SVD of Δ⋆\Delta^\star, the margins of W⋆=polarh(Δ⋆)W^\star=\mathrm{polar}_h(\Delta^\star), and the Hessian of p↦∣Δ(p)∣(h)p\mapsto\\|\Delta(p)\\|_{(h)} at p⋆p^\star. The initialization requirement (a small initial energy and a large enough logit scale) is a substantive restriction, and we do not give an a priori guarantee that an arbitrary initialization meets it. A global softmax-attention solution with positive prefactors, uniformly regular base weights, and Ξ(t)→∞\Xi(t)\to\infty eventually meets these thresholds under (MM) (end of Appendix E), and Theorem 6.3 then still requires (TC), (SC), (ND), and ∣S∣≥2|S|\ge2. The revised remark after Theorem 6.2 states that this is a local guarantee (rev. l.411–412), and Section 7 states that the results give no computable basin of attraction.

4. About code. We apologize for the misleading wording in the original Reproducibility Statement. Our initial submission contained only the manuscript PDF, without the experimental code or feature tensors. We have now uploaded the experimental code as a zip file in the supplementary material. Its README and RUN_PAPER.md provide instructions for reproducing the figures and tables. In particular, python reproduce.py verify checks the saved results, python reproduce.py smoke runs short fresh integrations, and python reproduce.py run --experiment all reruns the synthetic experiments. For the frozen-feature examples (Appendix G.3), the archive includes the inputs of case A and the certificate matrices of all three cases, together with a script to reconstruct cases B and C from public BERT and SST-2 releases. We have updated the Reproducibility Statement and Appendix G.3 accordingly.

Thank you again for the helpful suggestions and we are happy to discuss any further question.

AuthorsOpenReview ↗

Our response to Reviewer 3skY (2/2)

Our references

- Ataee Tarzanagh, D., Li, Y., Thrampoulidis, C., and Oymak, S. Transformers as support vector machines. NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023a.

- Ataee Tarzanagh, D., Li, Y., Zhang, X., and Oymak, S. Max-margin token selection in attention mechanism. NeurIPS, 2023b.

- Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.

- Du, S. S., Hu, W., and Lee, J. D. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. NeurIPS, 2018.

- Kang, C., Yun, J., Shin, B., Cho, Y., and Yun, C. Uniform spectral growth and convergence of Muon in LoRA-style matrix factorization. arXiv:2602.06385, 2026.

- Rhee, M., Simon, J., and Karkada, D. Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamics. arXiv:2606.30509, 2026.

- Sheen, H., Chen, S., Wang, T., and Zhou, H. H. Implicit regularization of gradient flow on one-layer softmax attention. arXiv:2403.08699, 2024.

- Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. EMNLP, 2013.

Thank you for clarifying the theoretical motivation. The example showing that two factorizations of the same key--query product can give different update directions helped readers understand why the factors need to be studied separately. I also appreciate the code and reproduction instructions, which should make it easier for others to check and build on the results.

As a practitioner outside this area of theory, my remaining reservation concerns what the results let us predict before training.

  • For a new problem with a changing gradient, Theorem 5.2 still gives me no way to tell from the data and initialization whether its trajectory condition will hold.
  • For Theorems 6.2 and 6.3, the new experiments show that randomly initialized key and query factors can approach the certified target on two instances with frozen BERT features. This strengthens the evidence beyond initialization near the target. It remains open whether, when training a causal language model from scratch, with representations evolving throughout training, it would enter the regime described by the local theory.

I can confirm my concerns are partially resolved and would maintain my original score, while leaning more toward acceptance if the AC judges the theoretical advance sufficient in its own right.

AuthorsOpenReview ↗

Our follow-up to reviewer 3skY

We appreciate your thoughtful follow-up and understand your remaining concerns. Our work aims to establish a theoretical foundation for understanding the implicit bias of factorwise spectral optimization in attention. Extending these results to practical Muon training with evolving representations and arbitrary initializations remains an important open problem that we leave for future work.

Thank you again for your constructive feedback and consideration.

Reviewer_F1W5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the implicit bias of factorwise spectral optimization in self-attention. The authors primarily focus on how independently updating the key and query matrices affects attention. It introduces a continuous-time, factor-wise polar flow model for per-head Muon and characterizes the resulting max-margin geometry using the dual Ky Fan norm. The authors establish convergence to leading singular directions under fixed gradients and prove conditional max-margin convergence under time-varying gradients. For exact softmax attention, they derive local convergence guarantees for the attention product and, under assumptions. Synthetic experiments support these theoretical findings while highlighting the importance of initialization, spectral alignment, and rank-deficient dynamics.

Critical strengths
  1. The paper studies the optimization bias introduced due to the spectral updates in Self-attention. The study of training dynamics when query and key projection parameters updates separately, while model depends on their product is relevant and can lead to better understanding of inner workings of transformers.

  2. The paper provides convergence guarantees for factorwise spectral optimization, establishing connections between attention dynamics, implicit bias, and max-margin solutions characterized by the dual Ky Fan norm.

  3. The cross-Gram matrix and factorwise polar flow provide new insights into how independently updating key and query matrices influences convergence.

  4. The paper supports its theoretical findings through controlled synthetic experiments, initialization sensitivity studies, and rank-boundary analyses.

Critical weaknesses
  1. While the paper presents an interesting and highly relevant theory, its impact would be significantly strengthened by addressing two key areas. First, the authors should discuss how the proposed theory affects downstream task performance. Second, it would be highly valuable to include a discussion on the specific conditions under which the learned model shifts toward a more interpretable/non-interpretable mode from attention perspective.

  2. The convergence guarantees rely on strong assumptions, including spectral separation, favorable initialization, integrated coercivity, and dual nondegeneracy, limiting their applicability to general attention optimization.

  3. The experiments primarily use low-dimensional synthetic attention tasks, with limited evaluation on realistic Transformer architectures or large-scale datasets.

  4. The analysis uses a simplified continuous-time model that excludes momentum, discrete updates, and interactions between attention heads. This limits its applicability to real-world Transformer training.

  5. The notation is complex in some sections, and the theoretical sections (4 and 5) would benefit from more intuitive insights and discussion.

Questions for authors
  1. Can the authors clarify how restrictive the convergence assumptions are in practice?

  2. How does the experimental evaluation extend to higher-dimensional attention models and realistic Transformer architectures?

  3. How well do the theoretical results extend to practical Muon optimization with discrete updates, momentum, and approximate orthogonalization?

5 comments
AuthorsOpenReview ↗

Our response to Reviewer F1W5 (1/2)

We want to thank you for the positive assessment of the theory and for the concrete questions. We answer them in order below.

1. About how restrictive the assumptions are in practice. We see three levels.

1. The condition Ξ(t)→∞\Xi(t)\to\infty in Theorem 5.2 is a trajectory condition. After the theorem, the revision gives two sufficient conditions (rev. l.359–360): η(t)≥η0>0\eta(t)\ge\eta_0>0 for large tt, or 2∫t0∞∣Z′∣op<λmin⁡(G(t0))−σh+1(Z(t0))2\int_{t_0}^\infty\\|Z'\\|_{\rm op}<\lambda_{\min}(G(t_0))-\sigma_{h+1}(Z(t_0)) (Eq. (C.9)).

2. The certificate conditions of Section 6 are finite-dimensional checks once a candidate dual solution is available: (MM) requires an SVD of Δ⋆\Delta^\star and the margins of W⋆=polarh(Δ⋆)W^\star=\mathrm{polar}_h(\Delta^\star), and (ND) requires the Hessian of p↦∣Δ(p)∣(h)p\mapsto\\|\Delta(p)\\|_{(h)} at p⋆p^\star on the tangent space of the active simplex. Within the construction of Appendix F.3, (ND) holds for almost every invertible M0M_0.

3. The strongest assumption is the local initialization (a small initial energy and a small inverse logit scale), and we do not give an a priori guarantee that an arbitrary initialization meets it. A global solution with Ξ(t)→∞\Xi(t)\to\infty eventually meets these thresholds under (MM) (end of Appendix E). In the coupled example of Appendix G.4 (d=2d=2, h=1h=1), every run with rinit≤0.75r_{\rm init}\le0.75 meets the target-error criterion at T=10T=10.

2. About higher dimensions and realistic Transformers. Our theory is for a single head with fixed representations, values, and readouts, so we did not train full Transformers. Our closest step is Appendix G.3: three constructed instances from two SST-2 sentences (Socher et al., 2013), with BERT states (Devlin et al., 2019) projected to d=64d=64, h=4h=4, and four queries with causal masks. Cases A and B use prescribed utilities (target 1, competitor 0), case C uses a frozen task readout, and the factors start at the numerically certified target. At t=6t=6, the product errors are 1.40×10−41.40\times10^{-4} to 5.04×10−45.04\times10^{-4}, and the weight errors are 8.77×10−68.77\times10^{-6} to 1.36×10−41.36\times10^{-4} (Table 8). We read this as evidence that the certificate conditions can hold on pretrained features, and as finite-horizon behavior consistent with the local theory. End-to-end training and the behavior of typical random initializations remain open. Interacting heads and learned representations are the main next step (Section 7).

3. About discrete updates, momentum, and approximate orthogonalization. For approximate orthogonalization, Table 5 (Appendix G.3) replaces the exact polar factor by the cubic Newton–Schulz iteration in the fixed-matrix task. At the tested depths 5 and 8, the endpoint product error at t=20t=20 equals the exact-polar value (5.83×10−45.83\times10^{-4}) to the displayed precision, whereas depth 1 gives a negative separation parameter and a product error of 0.5720.572. This test uses the cubic iteration, and the tuned quintic iteration of practical Muon (Jordan et al., 2024) remains to be tested. Our analysis does not cover discrete steps or momentum, which are studied in separate lines of work (Kim & Oh, 2026; Choudhury et al., 2026; Li et al., 2026b). Combining them with the factorwise analysis is a natural next step.

4. About downstream performance and interpretability. An implicit-bias result describes which solution the optimizer selects, so our claims concern the selected solution and leave downstream accuracy open. Under (MM), (TC), and initialization near the target, Theorem 6.2(c) gives 1−αir,kir⋆(t)≤(∣Mir∣−1)e−c0/ε(t)1-\alpha_{ir,k^\star_{ir}}(t)\le(|\mathcal M_{ir}|-1)e^{-c_0/\varepsilon(t)} for every supervised query and all t≥t0t\ge t_0. This is concentration of attention on the prescribed target key, which is a weaker property than semantic or causal interpretability. A related point concerns the gradient weights: product convergence need not identify how the limiting normalized gradient weight is allocated among the tight comparisons. In the commuting example of Appendices F.2 and G.3, the product converges to I2I_2 while p1p_1 converges to 1/21/2, 11, or 00 depending on the initialization. We think this distinction is a useful starting point for the question you raise.

5. About notation. Following your comment and Reviewer G5EN's, each term flagged in the reviews is now explained in the main text, and we added short explanations after Theorem 4.1 (rev. l.276–277), around Theorem 5.2 (rev. l.329–330, 359–360), and after Assumption 6.1 and Theorem 6.2 (rev. l.387–388, 411–412).

Thank you again for the thoughtful review and we are happy to discuss any further question.

AuthorsOpenReview ↗

Our response to Reviewer F1W5 (2/2)

Our references for above comment

- Choudhury, S., Cheng, X., Takáč, M., Na, S., and Kolar, M. Muon with Nesterov momentum: Heavy-tailed noise and (randomized) inexact polar decomposition. arXiv:2605.06884, 2026.

- Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.

- Jordan, K., Jin, Y., Boza, V., You, J., Cesista, F., Newhouse, L., and Bernstein, J. Muon: An optimizer for hidden layers in neural networks. Blog post, 2024.

- Kim, G. Y. and Oh, M. Convergence of Muon with Newton–Schulz. arXiv:2601.19156, 2026.

- Li, X., Zhang, Z., Liu, W., and Bao, H. Denoise first, orthogonalize later: Understanding momentum in Muon via spectral filtering. arXiv:2606.03899, 2026b.

- Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. EMNLP, 2013.

Reviewer_F1W5OpenReview ↗

Response to Rebuttal

I thank the authors for their careful response to the review. The authors have responded to most of my concerns. The following are my comments based on the reviews and the authors' response:

  1. The trajectory assumption is highly restrictive. The authors should show that this assumption is satisfied, or at least that it holds closely under the corresponding optimizer and standard initializations.

  2. I agree with reviewer 3skY that the setting and message are not practical (Response 4). The paper needs a clear, practical message relevant to researchers in theory and application. Even though implicit bias results concern which solution the optimizer selects, the paper's message should be clear about what effect that selected solution has in practice.

  3. Could the authors expand on the meaning of the "negative separation parameter" in response 3?

Reviewer_G5ENOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the limiting directions of the key-query matrices in training a single attention head with spectral updates (an idealized version of muon) for a fixed target loss. This is an extension of previous work for other optimization approaches, other parameterizations, and other objectives (other combinations of spectral updates, gradient updates, regularization paths, with factorized and linear - not factorized - parameterizations, and with various objectives).

This is a natural extension of past work in what is a novel combination of settings, so it is an easy-to-state problem to present to a strong AI model (generalize xxx results in yyy directions). So the conceptual contribution is unclear and it appears that the problem-solving part of the contribution comes from an AI. We should expect the presentation to be a significant part of the contribution. But the presentation is at the same time beautifully fluent and littered with undefined terms, missing explanations, and other obfuscating features. I compiled a list from the first three sections and gave up after that.

Critical strengths

The paper studies the limiting directions of the key-query matrices in training a single attention head with spectral updates (an idealized version of muon) for a fixed target loss. This is an extension of previous work for other optimization approaches, other parameterizations, and other objectives (other combinations of spectral updates, gradient updates, regularization paths, with factorized and linear - not factorized - parameterizations, and with various objectives).

Critical weaknesses

This is a natural extension of past work in what is a novel combination of settings, so it is an easy-to-state problem to present to a strong AI model (generalize xxx results in yyy directions). So the conceptual contribution is unclear and it appears that the problem-solving part of the contribution comes from an AI. We should expect the presentation to be a significant part of the contribution. But the presentation is at the same time beautifully fluent and littered with undefined terms, missing explanations, and other obfuscating features. I compiled a list from the first three sections and gave up after that:

line 075: the definition of sym isn't provided (until p5)

l083: What does 'separable comparisons with positive regular base weights' mean?

l085: What 'convex relaxation'?

l087: What 'trajectory-wide condition'?

l090: What is 'dual nondegeneracy'?

l095: The sentence containing "a survey of 1,536 initializations" is remarkable. Who writes that?

l134: softmax_M(a): tell us what an index set is. e.g., say a\in R^n, M\subset[n].

l134: The author thinks the reader has never seen notation for operator norms, but knows what 'softmax restricted to M' means.

l150: "c_ir = Y_iv for a label Y_i and a linear head v" - what is Y_i? Real? Matrix? What is a linear head?

l162: "Each query has a prescribed target key" - having just defined loss, shouldn't you tell us how this is defined in terms of that objective? And what conditions on the data (c,V) you are assuming that will ensure the uniqueness of the target key?

l163: "visible" means in M_ir?

l167: "We enumerate the comparisons as \Delta_1,..." - what does this mean? This is vague and imprecise.

l172: "Gradient prefactors rho_j" - this terminology and notation has never been used before.

l181: "the natural norm is the dual Ky Fan norm" Why?

l188: "the rank-constrained problem and its convex relaxation ..." Why is this the convex relaxation?

l194: "maximizers of gamma_h" -> "maximizers in the definition of gamma_h"

l200: The update rule is some kind of factorwise polar flow, but there is no explanation in the main paper of where it comes from (I couldn't find an explanation until Appendix A).

l206: Why are you normalizing dynamics? Where does this come from?

l224: "The factor balls are forward invariant" What does this mean? What is a factor ball? What does forward invariant mean?

Questions for authors

n/a

1 comment
AuthorsOpenReview ↗

Our response to Reviewer G5EN

We really appreciate your careful reading of our paper and the detailed list of comments. We also apologize for the difficulties you may have encountered while reading it.

1. About the writing problems. We apologize for the writing problems you met in the first three sections. Indeed, there are terms that are used not defined before: for example, sym appears at l.075 but is defined only at l.232, and (ND) appears at l.090 but is defined only at l.366. We also had to move some of important explanations in the appendix due to 9-page limit: where the update rule (3.6) comes from (Appendix A.1, "Relation to Muon", l.764–782), how the normalized flow (3.7) and the invariance of the factor ball follow from it (Appendix A.1, "Normalized flow", l.784–800), why the dual Ky Fan ball is the convex hull of the feasible products (Appendix A.1, "Product geometry", l.819–838), and the formula for ρj\rho_j (Appendix D.1, Eq. (D.1), l.1198–1205). We have addressed these issues in the revised manuscript (please see the changes marked in blue). Below, we respond to each point you raised, with the corresponding line numbers in the revision (*l.*: original line; *rev. l.*: revised line).

- l.075 (sym): defined where it first appears (rev. l.077).

- l.083, 087, 090 (base weights, trajectory-wide condition, dual nondegeneracy): each term is now explained in the Contributions (rev. l.085–087, 089–091, 093–094), and the conditions are summarized immediately after Assumption 6.1 (rev. l.387–388).

- l.085, 188 (convex relaxation): the Contributions point to Eq. (3.5) (rev. l.087), and Section 3 explains that the relaxation replaces the rank-constrained feasible set with its convex hull (rev. l.189–197).

- l.095 (1,536 initializations): this number comes from the initialization study in Section 7 and Appendix G.4, where 512 initial factor pairs are each tested at three initial inverse logit scales in the coupled softmax-attention example (1,536 runs). We have rewritten the statement as “including a sweep over initializations” (rev. l.097–099) and provided the design and outcomes in Section 7 (rev. l.472–479) and Appendix G.4 (rev. l.2216–2227, 2267–2274).

- l.134 (index set, masked softmax): explicitly defined for a∈Rna\in\mathbb R^n and M⊆[n]\mathcal M\subseteq[n] (rev. l.137–139).

- l.150 (YiY_i, linear head): Yi∈±1Y_i\in\\{\pm1\\} is a label, v∈Rdv\in\mathbb R^d is a fixed linear head, and z=xirz=x_{ir} is the query token (rev. l.154–157).

- l.162, 163 (targets, "visible"): as with the optimal tokens of Att-SVM (Ataee Tarzanagh et al., 2023a), the targets are fixed in advance, and (TC) in Section 6 requires each target to be the unique key of largest utility in its mask (rev. l.167–170). We also clarify that the utilities are the token scores of Att-SVM (rev. l.151–153). "Visible competitor" is replaced by "competitor key k∈Mir∖kir⋆k\in\mathcal M_{ir}\setminus\\{k^\star_{ir}\\}" in Section 3 and in (TC) (rev. l.169–170, 379).

- l.167 (enumeration): the triples (i,r,k)(i,r,k) are indexed in a fixed order by j∈[M]j\in[M], with Δj=Δirk\Delta_j=\Delta_{irk} (rev. l.174–175).

- l.172 (ρj\rho_j): the formula ρj=β[−∂rℓi(ui)],αir,kir⋆(uir−girk)\rho_j=\beta[-\partial_r\ell_i(\mathbf u_i)]\\,\alpha_{ir,k^\star_{ir}}(u_{ir}-g_{irk}) now appears immediately after Eq. (3.3), and the first mention points to it (rev. l.179, 184).

- l.181 (why dual Ky Fan): the normalized factors satisfy ∣K∣op,∣Q∣op≤1\\|K\\|_{\rm op},\\|Q\\|_{\rm op}\le1, so their products lie in W:rank(W)≤h, ∣W∣op≤1\\{W:\mathrm{rank}(W)\le h,\ \\|W\\|_{\rm op}\le1\\}, and the dual Ky Fan ball is the convex hull of this set (rev. l.189–192).

- l.194: clarified that WW solves (Att-SVM$_h$) if and only if γhW\gamma_hW attains the maximum in the definition of γh\gamma_h (rev. l.203–204).

- l.200 (update rule): we now explain how Eq. (3.6) idealizes per-head Muon: removing momentum, replacing the Newton–Schulz iteration with the exact polar factor, and passing to continuous time gives a factorwise steepest-descent flow in the operator norm (rev. l.208–210). We also clarify that Eq. (3.6) is a differential inclusion (rev. l.214–215).

- l.206 (normalization): as in implicit-bias analyses, we track the direction of W~\widetilde W. Since both factors grow at most linearly, we normalize them by this growth (rev. l.216–217).

- l.224 (factor ball, forward invariant): both terms are now defined explicitly in the main text (rev. l.238–239).

2. About the problem formulation. We want to position the story behind it, state our viewpoint, and please see our explain in general response (points 2 and 3).

3. About AI use. We disclosed our AI use in the paper according to the ICLR policy, and we checked every claim and proof ourselves. We would be grateful if you could take another look at the revised version, especially Sections 4–6, which contain the main results.

Thank you again for your careful analysis, and we are happy to discuss any further question.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response: Our motivation, key idea, and model (1/3)

We want to thank all three reviewers for their time and their careful feedback. We have uploaded a revised PDF for the reviewers' convenience (the changes are marked in blue).

And in this public comment, we summarize the main changes and answer the concerns that more than one reviewer raised, and especially we want to provide our view on the reason for investigating this learning problem.

1. About presentation. Reviewer G5EN pointed out that many terms were not defined and that we left the explanation in the appendix. We have fixed this writing error in the new revision of the submission. Each term flagged in the reviews is now explained where it first appears. And to keep the main text at nine pages, we made Figure 1 smaller and moved the initialization-survey figure to Appendix G.4 (please see the blue passages in the new revision, mostly in Sections 1 and 3, with short additions in Sections 4–6).

2. Our motivation for this problem (informal). We started from two lines of work: the implicit bias of training (Soudry et al., 2018; Gunasekar et al., 2018; Ji & Telgarsky, 2020; Lyu & Li, 2020), and the recent use of the Muon optimizer (Jordan et al., 2024) in LLMs (Liu et al., 2025; Kimi Team, 2025), where some models (GLM-5, Kimi K3) orthogonalize the momentum of the key and query projections head by head (GLM-5 Team, 2026; Kimi Team, 2026). Many implicit-bias results study a linear model or a directly parameterized matrix (Soudry et al., 2018; Vasudeva et al., 2025; Fan et al., 2025), and factorwise spectral updates have been analyzed for squared-loss matrix factorization, where the data fix the target product (Kang et al., 2026; Rhee et al., 2026). A product of two trained matrices, such as KQ⊤KQ^\top, behaves differently, and it appears in many successful methods in deep learning (low-rank adaptation (Hu et al., 2021), for example, and attention (Vaswani et al., 2017), which is the case we study here).

In attention, the logits depend on the product KQ⊤KQ^\top, and some observed models update its factors separately, like the per-head Muon variant. This is why we think the problem is worth studying, both as an implicit-bias question for a product of two factors and as a first step toward understanding how these models change in practice.

We list three key points that make this question non-trivial:

1. Euclidean gradient flow conserves K~⊤K~−Q~⊤Q~\widetilde K^\top\widetilde K-\widetilde Q^\top\widetilde Q (Du et al., 2018; Arora et al., 2018), so balanced factors remain balanced and the product follows an autonomous flow. Factorwise spectral updates, in our momentum-free and exact-polar idealization, do not conserve this quantity, and two factorizations of the same product can induce different product velocities under the same gradient (Section 1, Appendix F.1). We therefore track both factors, and we use the cross-Gram matrix G=sym(K⊤ZQ)G=\mathrm{sym}(K^\top ZQ) as an alignment certificate for the pair.

2. The factorized margin problem has KKT points whose margins are arbitrarily worse than the optimum (Appendix F.1). These points are not local minima when h≥2h\ge2, but they show that stationarity alone cannot certify optimality. Hence a result stating that every limit direction is a KKT point, as shown for homogeneous models (Tsilivis et al., 2026; Gronich & Vardi, 2026), does not identify the selected product here, and we follow the trajectory instead.

3. Earlier work on factorized attention assumes that the product converges in direction (Sheen et al., 2024). Under (MM) and (TC), we prove that the product converges for initializations near the target (Theorem 6.2), and we show that the product can converge while the gradient weights converge to initialization-dependent limits (Appendix F.2).

3. Our key idea. Since KK and QQ move separately during training, we track the cross-Gram matrix G=sym(K⊤ZQ)G=\mathrm{sym}(K^\top ZQ), where ZZ is the normalized negative gradient. For a unit v∈Rhv\in\mathbb R^h, v⊤Gv=(Kv)⊤Z(Qv)v^\top Gv=(Kv)^\top Z(Qv) measures how well the key direction KvKv and the query direction QvQv align with the gradient. For a fixed gradient, this alignment never decreases, so, when σh(Z)>σh+1(Z)\sigma_h(Z)>\sigma_{h+1}(Z), a head whose initial alignment exceeds σh+1(Z)\sigma_{h+1}(Z) in every latent direction converges, up to a common rotation, to the leading hh singular vectors of ZZ (Theorem 4.1, a sufficient condition). When the gradient moves with the softmax weights, a related dissipation argument bounds the gap to the convex-relaxation value γh=min⁡p∣Δ(p)∣(h)\gamma_h=\min_p\\|\Delta(p)\\|_{(h)} by C0e−Ξ(t)C_0e^{-\Xi(t)}, and when Ξ(t)→∞\Xi(t)\to\infty a product of rank at most hh attains γh\gamma_h (Theorem 5.2). And in exact softmax attention under (MM) and (TC), the normalized product converges to the optimal one, and the factors approach its common-rotation orbit, from initializations near the target and at a large enough logit scale (Theorem 6.2).

AuthorsOpenReview ↗

Our motivation, key idea, and model (2/3)

4. About reproducibility. We uploaded the complete code as a zip file in the supplementary material. (please see our reply to Reviewer 3skY for details).

5. Scope of our model. Our guarantees hold for a single head with fixed representations, values, and readouts, trained by exact polar flow without momentum, and the softmax-attention results are local (they require a sufficiently small initial error and inverse logit scale). Section 7 states this explicitly.

We want to note that a single softmax-attention layer on fixed token embeddings, with the analysis focused on the key–query weights, is an established model in implicit-bias theory for attention (Ataee Tarzanagh et al., 2023a; 2023b; Sheen et al., 2024; Vasudeva et al., 2025; Julistiono et al., 2026). With a single query per sequence, our model reduces to that of Ataee Tarzanagh et al. (2023a) (Section 3). Keeping the rest fixed isolates the one new ingredient, separate spectral updates of KK and QQ in attention, a gap that these works do not cover. In the same spirit, the exact polar step is steepest descent in the operator norm (Bernstein & Newhouse, 2024), the core of Muon (Jordan et al., 2024), while momentum and the Newton–Schulz approximation are analyzed in separate lines of work (Kim & Oh, 2026; Choudhury et al., 2026; Li et al., 2026b). In our fixed-matrix task, the cubic Newton–Schulz iteration at the tested depths 5 and 8 gives the same endpoint product error as the exact polar factor, to the displayed precision (Appendix G.3, Table 5).

Discrete Muon with momentum, several interacting heads, and learned representations are the natural next steps.

AuthorsOpenReview ↗

Our motivation, key idea, and model (3/3)

Our references for comments above

- Arora, S., Cohen, N., and Hazan, E. On the optimization of deep networks: Implicit acceleration by overparameterization. ICML, 2018.

- Ataee Tarzanagh, D., Li, Y., Thrampoulidis, C., and Oymak, S. Transformers as support vector machines. NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023a.

- Ataee Tarzanagh, D., Li, Y., Zhang, X., and Oymak, S. Max-margin token selection in attention mechanism. NeurIPS, 2023b.

- Bernstein, J. and Newhouse, L. Old optimizer, new norm: An anthology. OPT 2024: 16th Annual Workshop on Optimization for Machine Learning, 2024.

- Choudhury, S., Cheng, X., Takáč, M., Na, S., and Kolar, M. Muon with Nesterov momentum: Heavy-tailed noise and (randomized) inexact polar decomposition. arXiv:2605.06884, 2026.

- Du, S. S., Hu, W., and Lee, J. D. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. NeurIPS, 2018.

- Fan, C., Schmidt, M., and Thrampoulidis, C. Implicit bias of spectral descent and Muon on multiclass separable data. NeurIPS, 2025.

- GLM-5 Team. GLM-5: From vibe coding to agentic engineering. arXiv:2602.15763, 2026.

- Gronich, E. and Vardi, G. The implicit bias of Adam and Muon on smooth homogeneous neural networks. ICML, 2026.

- Gunasekar, S., Lee, J., Soudry, D., and Srebro, N. Characterizing implicit bias in terms of optimization geometry. ICML, 2018.

- Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. arXiv:2106.09685, 2021.

- Ji, Z. and Telgarsky, M. J. Directional convergence and alignment in deep learning. NeurIPS, 2020.

- Jordan, K., Jin, Y., Boza, V., You, J., Cesista, F., Newhouse, L., and Bernstein, J. Muon: An optimizer for hidden layers in neural networks. Blog post, 2024.

- Julistiono, A. K., Ataee Tarzanagh, D., and Azizan, N. Optimizing attention with mirror descent: Generalized max-margin token selection. JMLR, 27(36), 2026.

- Kang, C., Yun, J., Shin, B., Cho, Y., and Yun, C. Uniform spectral growth and convergence of Muon in LoRA-style matrix factorization. arXiv:2602.06385, 2026.

- Kim, G. Y. and Oh, M. Convergence of Muon with Newton–Schulz. arXiv:2601.19156, 2026.

- Kimi Team. Kimi K2: Open agentic intelligence. arXiv:2507.20534, 2025.

- Kimi Team. Kimi K3: Open frontier intelligence. arXiv:2607.24653, 2026.

- Li, X., Zhang, Z., Liu, W., and Bao, H. Denoise first, orthogonalize later: Understanding momentum in Muon via spectral filtering. arXiv:2606.03899, 2026b.

- Liu, J. et al. Muon is scalable for LLM training. arXiv:2502.16982, 2025.

- Lyu, K. and Li, J. Gradient descent maximizes the margin of homogeneous neural networks. ICLR, 2020.

- Rhee, M., Simon, J., and Karkada, D. Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamics. arXiv:2606.30509, 2026.

- Sheen, H., Chen, S., Wang, T., and Zhou, H. H. Implicit regularization of gradient flow on one-layer softmax attention. arXiv:2403.08699, 2024.

- Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., and Srebro, N. The implicit bias of gradient descent on separable data. JMLR, 2018.

- Tsilivis, N., Gronich, E., Kempe, J., and Vardi, G. Flavors of margin: Implicit bias of steepest descent in homogeneous neural networks. JMLR, 2026.

- Vasudeva, B., Deora, P., and Thrampoulidis, C. Implicit bias and fast convergence rates for self-attention. TMLR, 2025.

AuthorsOpenReview ↗

Closing Remarks from Authors

Dear Area Chair and Reviewers,

As the discussion period draws to a close, we would like to sincerely thank you for your time, careful reading, and constructive feedback. We have carefully addressed the concerns raised in our responses and revised the manuscript accordingly, with changes highlighted in blue.

We would be grateful for any further comments or suggestions and would be happy to clarify any remaining questions before the discussion period closes. Thank you again for your valuable feedback and consideration.

Best regards,

Authors of Submission 13402

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.