acceptodds
Under review as a conference paper at ICLR 2027

Vanishing Initialization Turns Diagonal Tensor Gradient Flow into Greedy Matching

Abstract

We study gradient-flow (GF) dynamics of canonical polyadic (CP) tensor factorization from vanishing initialization. For order- models with and targets diagonal in mode-wise orthonormal bases, we prove that the limiting training trajectory is governed by greedy bipartite matching. Each atom–sector pair acquires a random activation clock determined by its microscopic initialization; processing these clocks in increasing order, subject to atom and sector exclusivity, determines the successive learning events. Away from activation times, the tensor output converges to the corresponding staircase trajectory. Complex modular addition is an exact equal-sector instance after Fourier diagonalization, so the activation clocks are i.i.d. Combining this reduction with known Random Edge asymptotics yields an explicit distribution-free large-system loss curve in the edge-exposure parametrization; we further derive its asymptotic conversion to physical training time. For real weights, conjugate Fourier modes instead form two-dimensional sectors. We characterize their rank and border-rank cascade and conjecture that the full GF dynamics is organized by the corresponding interleaved cascades.

Primary area:
learning theory
Keywords:
gradient flow, tensor learning, modular addition, greedy matching, loss trajectory, small initialization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qf2eOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper studies gradient flow for asymmetric CP factorization of orthogonally diagonal tensors of order ν≥3\nu\geq3, starting from small Gaussian initialization. Its main result describes the limiting dynamics through greedy matching between model atoms and target sectors, with activation times determined by random clocks. For fixed dimensions, convergence is uniform on compact slow-time sets away from accepted activation times. For equal-amplitude sectors, taking the initialization limit before the large-system limit connects the loss to the known Random Edge matching law. The complex modular-addition application further gives explicit training-time asymptotics. The paper also establishes geometric results for real Fourier blocks, while leaving the corresponding gradient-flow cascades conjectural. The results are supported by proofs and numerical illustrations.

Critical strengths
  1. The matching description identifies which atom learns which sector and when throughout the activation sequence. Two aspects make this result particularly interesting: surviving atom–sector pairs retain their original clocks after earlier captures, and cancellation of the target contribution suppresses further activation in claimed sectors. These mechanisms explain how ordinary asymmetric CP gradient flow produces the matching process. This explicit account of the full event sequence is the main reason I support acceptance.

  2. The complex modular-addition analysis connects the matching description to quantitative training-time predictions. The Random Edge loss law is known, and its distribution-free form applies after transforming time by the clock CDF. The additional contribution is the derivation of the cubic activation-clock tail and the resulting conversion to physical time, including the location and width of the learning window.

Critical weaknesses

Lemma 7 needs a correction in its product-error estimates. Equation (414) bounds the forcing by C(σν+E⊥,σν−1)C(\sigma^\nu+E_{\perp,\sigma}^{\nu-1}), but the self-interaction expansion in equation (124) contains terms quadratic in the off-sector residuals. With the main coordinates bounded away from zero, these generally give nonzero contributions of order E⊥,σ2E_{\perp,\sigma}^{2}, which the stated bound does not cover for ν>3\nu>3. Equation (420) also omits the decaying contribution of the initial transverse error; this issue remains for ν=3\nu=3.

This matters because Lemma 7 supplies the O(σν)O(\sigma^\nu) accuracy used to restart the induction in Proposition 9. Retaining the quadratic forcing and its transient contribution appears compatible with the logarithmic locking time, so I regard this as a fixable proof issue. However, the corrected estimate should be given explicitly.

Questions for authors

Can you provide a corrected estimate for equations (414)–(420), accounting for the quadratic off-sector forcing and its transient contribution, and show that the same logarithmic locking interval yields O(σν)O(\sigma^\nu) product error under the stated hypotheses for every ν≥3\nu\geq3?

2 comments

We thank the reviewer for the positive feedback and for identifying two genuine errors in our proof of Lemma 7. We acknowledge both mistakes and have revised the proof accordingly, providing additional details to make the argument easier to verify.

First, the original proof incorrectly used the stronger bound E⊥,σν−1E_{\perp,\sigma}^{\nu-1} instead of the correct E⊥,σ2E_{\perp,\sigma}^{2}. The weaker bound is nevertheless sufficient: the transverse error decays exponentially, and the equation governing the product error is itself exponentially contracting. Combining these two estimates shows that the resulting contribution to ∣πr,σ−λir∣|\pi_{r,\sigma}-\lambda_{i_r}| becomes O(σν)O(\sigma^\nu) over our chosen logarithmic locking interval.

Second, we had omitted the contribution of the initial transverse error. We have restored this term explicitly and shown that it also decays to the required O(σν)O(\sigma^\nu) scale during the locking interval.

Consequently, both corrections preserve the original conclusions of Lemma 7.

Thank you for addressing both points. I checked the revised proof of Lemma 7. Equation (419) now includes the quadratic off-sector forcing, and equation (424) retains the contribution of the initial transverse error. The subsequent choice of logarithmic locking time correctly brings these terms to the required O(σν)O(\sigma^\nu) scale, preserving the estimates used in Proposition 9.

This resolves my original soundness concern. I shall update my soundness assessment to 3, and maintain my positive recommendation.

Reviewer_SCLDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, the authors study the gradient-flow dynamics of "canonical polyadic" (CP) tensor factorization, from vanishing initialization.

Claims:

  • The authors "prove" that, in a particular setting (order-ν\nu models, with ν≥3\nu \geq 3, and targets diagonal in mode-wise orthonormal bases) the training dynamics are governed by greedy bipartite matching (Theorem 1).
  • Complex modular addition is an instance of this setting after a Fourier transform (Section 4, Corollary 1). Combined with a known random matching result (Proposition 1, Arnosti 2022), this gives a loss curve in the limit of many atoms and sectors (Theorems 2 and 3, Figure 2).
  • For real weights, the paper characterizes the rank and border-rank cascade of a two-dimensional Fourier block (Proposition 2) and conjectures the full dynamics from one simulation (Section 6, Figure 3).
Critical strengths
  • In Theorem 1, the authors give the sequence of learning events for ordinary CP gradient flow.
  • The paper states its limits (lines 478-485) and separates proven results from the conjecture for real weights (lines 435-440).
  • Figure 2 shows good agreement between experiments (gradient flow at p=128p = 128) and the loss curve predicted from theory.
Critical weaknesses

High-level: The writing is unpolished, hard to read, jam-packed with jargon that is difficult to parse and extract intuition or meaning from.

Detailed comments:

  • You use the term "activation clock" in the abstract (line 15) but this is not a standard phrase and is not defined until later (lines 119-135) so it makes the abstract hard to read on its own. The same holds for "atom" and "sector" (abstract line 14, introduction line 51; defined at lines 100 and 113).
  • Lines 15-17: "processing these clocks in increasing order, subject to atom and sector exclusivity, determines the successive learning events." This sentence seems meaningless? (It only becomes clear after the definition of the greedy process at lines 146-149.)
  • Line 37, "exhibit nonlinear near-origin dynamics": what does it mean that tensors "exhibit" dynamics?
  • Lines 72-74, "our results rigorously establish the pipeline": how is this a pipeline? I don't see how this "pipeline" makes sense. Also, the last step relies on a known result (Proposition 1, lines 279-283).
  • Figures 1 and 3 both seem unpolished and low-effort.
  • In Section 6, "sector" is used for two-dimensional objects (lines 22-23, Figure 3), but in line 373 they are referred to as "blocks". Please define the term once and use it consistently.
  • The comparison to prior work is only in Appendix A (lines 838-877); it should be covered in the main text.
Questions for authors
  1. Who is the intended audience of this paper? Is it deep learning researchers? Or applied mathematicians? What is the motivation for caring about this particular task/setting?
  2. In the abstract, you say "targets diagonal in mode-wise orthonormal bases". This is a bit tortuous wording, what does it mean exactly?
  3. Line 18: "Complex modular addition is an exact equal-sector instance after Fourier diagonalization". What does "equal-sector" mean here?
  4. Lines 20-21: what is "edge-exposure parametrization"?
  5. Line 20: what is "large-system loss"? Is this a standard term?
  6. What do you mean by "physical training time"? What is "physical" about it?
  7. Over how many seeds and values of pp did you observe the interleaved cascade of Figure 3?
  8. Can you add gradient flow runs for α=1/2\alpha = 1/2 and α=2\alpha = 2 to match Figure 2(c)?
3 comments
AuthorsOpenReview ↗

Author Response (Part 1/2)

We thank the reviewer for the constructive feedback, particularly regarding the readability and accessibility of the paper. We agree that several concepts and terms were introduced too abruptly, making the presentation unnecessarily difficult to follow.

Readability and terminology. We have substantially revised the manuscript to address these concerns. In particular, we have rewritten the abstract to make it more self-contained, clarified the introduction of specialized terminology throughout the main text, and added a glossary at the end of the appendices.

We have retained terms such as atom, sector, and activation clock because they denote specific mathematical objects that appear repeatedly in the analysis. We believe that consistently defined terminology ultimately makes the exposition clearer than repeatedly using more generic terms such as "component." However, we agree that these terms should be explained before they are extensively used, and have revised the manuscript accordingly.

Below we address several of the reviewer's specific questions.

Intended audience and motivation. The paper is intended primarily for researchers interested in the mathematical theory of neural network training dynamics. Our motivation is to identify nonlinear learning problems for which the entire gradient-flow trajectory can be rigorously characterized. Diagonal higher-order tensor factorization provides such a setting, while complex modular addition gives a concrete application connecting this theory to a widely studied machine-learning problem.

Scalar sectors, non-scalar sectors, and Fourier blocks. By a sector, we mean a tensor-product subspace associated with a component of the target. In real modular addition, each pair of conjugate Fourier frequencies gives a two-dimensional real subspace in each mode. Their tensor product is an eight-dimensional non-scalar sector. The target remains block-diagonal with respect to this coarser orthogonal decomposition. We have clarified this distinction in Section 6 and in the glossary.

Equal-sector terminology. We meant equal-amplitude target sectors, i.e., sectors with identical coefficients λi\lambda_i. We now use the more precise expression "equal-amplitude sectors."

Physical training time. This refers to the original time variable tt in the gradient-flow equations, as opposed to the rescaled slow time θ=σν−2t\theta=\sigma^{\nu-2}t used to describe the vanishing-initialization limit. We now introduce and explain this distinction explicitly in Section 2.

Large-system loss. This refers to the limiting normalized loss as the numbers of target sectors and model atoms grow, p,Hp→∞p,H_p\to\infty, with Hp/p→αH_p/p\to\alpha. We have clarified the limiting regime and normalization in Section 5.

Edge exposure. The edge-exposure parameter ss measures how many candidate atom--sector clocks have been reached. More precisely, s=pFν(λpσν−2t)s=pF_\nu(\lambda_p\sigma^{\nu-2}t), where FνF_\nu is the common CDF of the isolated activation clocks. Thus, ss is the expected number of exposed candidate edges incident to each atom. Importantly, an exposed candidate need not be accepted by the matching process. This parametrization connects the gradient-flow dynamics to the Random Edge process and yields a universal large-system loss curve. We have expanded its explanation in Section 5.

AuthorsOpenReview ↗

Author Response (Part 2/2)

The theoretical "pipeline" and the role of prior work. We have removed the term "pipeline" from the revised Introduction. It was intended to describe the sequence of mathematical reductions connecting gradient flow, activation clocks, greedy matching, and large-system loss dynamics.

We agree that the random-matching asymptotic is a known result, which we explicitly attribute to Arnosti (2022). Our novel contribution is the rigorous derivation of the greedy matching process from ordinary nonlinear CP gradient flow and its use to characterize the complete limiting learning trajectory.

In response to the reviewer's suggestion, we have also added a short discussion of related work and our contributions relative to previous results in the Introduction, while retaining the more detailed literature review and comparisons in Appendix A1.

Figures 1 and 3. We appreciate the reviewer's concern about the presentation of these figures. Both figures were carefully designed to illustrate the underlying learning mechanisms, rather than only the loss curves. Figure 1 connects the GF loss staircase, microscopic atom--sector candidate trajectories, and the resulting greedy matching. Figure 3 similarly connects the loss dynamics, sector-resolved atom amplitudes, and the successive assignments forming interleaved cascades.

Several visual choices are deliberate. For example, Figure 1 uses a logarithmic amplitude scale to display the microscopic competition between candidates, whereas Figure 3 uses a linear scale to emphasize the evolution of macroscopic atom contributions. Colors, event numbers, and line styles connect corresponding events across panels. We hope these features help the reader understand the mechanisms illustrated by the figures.

Additional numerical experiments for Figure 3. We have examined 40 Gaussian seeds for p=5,H=7p=5,H=7 at three initialization scales. At σ=0.005\sigma=0.005, 37/40 trajectories completed both three-stage cascades by slow time θ=2.5\theta=2.5, with 33/40 exhibiting interleaving. We also performed preliminary experiments for p=7,H=10p=7,H=10 and p=9,H=15p=9,H=15, observing completed interleaved cascades for all three tested seeds in each setting.

We are continuing these experiments, and so far we have found no evidence contradicting the proposed multi-stage cascade mechanism. We emphasize that Figure 3 is intended to illustrate this conjectured dynamics, rather than provide its comprehensive experimental verification.

Additional gradient-flow experiments for Figure 2(c). We have added Figure 4 in Appendix A5, showing GF experiments for α=1/2\alpha=1/2 and α=2\alpha=2, alongside the corresponding theoretical predictions. As for α=1\alpha=1, the observed loss trajectories agree well with the Random Edge limit.

We hope that these revisions and additional experiments address the reviewer's concerns and make the main results and their significance easier to appreciate.

Dear Reviewer SCLD,

Thank you again for your thoughtful review. As the discussion period draws to a close, we wanted to ask whether you have had a chance to look at our revised manuscript.

In particular, we would appreciate your feedback on whether the revised presentation has made the main results easier to follow and whether the new experiments have made our conclusions more convincing. We would also be happy to clarify any remaining questions.

Best regards,
The Authors

Reviewer_HefzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper focuses on the problem of learning a diagonal target tensor of order at least 3 using gradient flow, and in particular analyzing the dynamics of that gradient flow. They show that the problem reduces to a greedy bipartite graph matching algorithm which depends on the initialization of the tensor. They then study learning the tensor of modular addition and show that it exactly follows the regime of a diagonal target tensor.

Critical strengths

S1: The paper is able to clearly articulate how the tensor learning problem relates to a greedy bipartite graph matching.

S2: The example of modular addition is a nice illustration of their analysis.

Critical weaknesses

W1: If my understanding of the diagonal target tensor is correct (Q1), this seems like a large restriction. The authors do provide a relevant example (modular addition) where it is directly relevant, and they do acknowledge this as a limitation.

W2: The analysis is interesting and appears correct to me. However, it is a little unclear whether these insights are actionable and how. Does this allow us to learn tensors more efficiently or more accurately?

W3: I was not able to follow Section 5 as well as the previous sections. What exactly is the setting described in line 262 for H_p/p \to \alpha? Are we imaging that the target tensor p is growing to infinity and the H atoms are also growing to infinity, but at a constant ratio alpha? And then the random edge algorithm is an alternative to the gradient flow strategy? The theoretical results in the section are heavy on the notation, a plain language description and explanation of significance accompanying each result would be helpful for my understanding.

Questions for authors

Q1: I am slightly confused by the diagonal targets. Perhaps an explicit example would be helpful. If p=3, nu=3, and the orthonormal bases are the original coordinate bases, what does T look like? Are the scalar sectors lambda_1 [1 0 0] \otimes [1 0 0] \otimes [1 0 0], lambda_2 [0 1 0] \otimes [0 1 0] \otimes [0 1 0], and lambda_3 [0 0 1] \otimes [0 0 1] \otimes [0 0 1]?

Q2: On line 119, what is meant by order sigma? Of course sigma is the standard deviation of the g_k, but I am missing the connection.

Q3: What is the significance of Corollary 1?

I believe that many of the weakness I identified and questions I have stemmed from my understanding of your work, rather than its inherent quality or correctness. I am open to adjusting my score if you are able to address these.

3 comments
AuthorsOpenReview ↗

Author response (1/2)

Thank you for the positive assessment of our results, particularly the connection between tensor gradient flow and greedy bipartite matching, and for the thoughtful questions and suggestions. We address the concerns below and clarify several points that may not have been sufficiently explained in the original submission.

W1. Restriction to diagonal tensor targets.

We agree that the diagonal-target assumption is a significant restriction. Nevertheless, this is a mathematically important and well-studied class of tensors, known as orthogonally decomposable (odeco) tensors over R\mathbb R and unitarily decomposable (udeco) tensors over C\mathbb C. Such tensors have been investigated particularly in the context of tensor decomposition, spectral methods, and algebraic characterizations (e.g., Anandkumar et al., 2014; Boralevi et al., 2017). Our perspective is different: we study the complete learning trajectory of ordinary asymmetric CP gradient flow and characterize it through random activation clocks and greedy bipartite matching. We briefly discuss odeco/udeco tensors and their relation to prior work in Appendix A1 of the revised manuscript.

Moreover, related structures arise naturally in other arithmetic learning problems. For example, modular multiplication over a prime field Fp\mathbb F_p admits a closely related Fourier decomposition: besides p−2p-2 scalar sectors associated with nontrivial multiplicative characters, there is one two-dimensional sector involving the trivial character and zero. Thus, modular multiplication provides another example of a structured tensor-learning problem closely related to, but not directly covered by, our present theory.

We believe that diagonal tensors provide a natural starting point for developing a precise understanding of nonlinear tensor-learning dynamics. Their analysis can serve as a baseline for studying more general structured targets, including those with multidimensional sectors, as already suggested by our results for real modular addition.

W2. Practical significance and potential applications.

Our main contribution is a detailed analytical description of gradient-flow dynamics in a nonlinear, nonconvex learning problem. This description already reveals several important properties that are far from obvious a priori.

First, although the CP optimization problem is nonconvex, we prove that, when H≥pH\geq p, gradient flow from vanishing random initialization eventually learns all target components, reaching zero limiting loss rather than becoming trapped in a suboptimal state. Second, we determine the activation times of individual components and, for complex modular addition, derive an explicit asymptotic learning-time scale. Remarkably, we also find that a substantial decrease in loss occurs within a time interval that is asymptotically much shorter than the characteristic learning time (by a relative factor of order 1/log⁡p1/\log p). Thus, learning may appear almost sudden, following a long period of very little change in the loss.

Beyond these immediate results, we expect that this detailed and analytically tractable picture can serve as a starting point for studying more practical optimization methods, such as discrete gradient descent and gradient descent with momentum. This is analogous to how the exact solvability of quadratic optimization problems provides the foundation for a detailed spectral theory of gradient descent and its variants. Understanding which features of our dynamics persist for these methods is an interesting direction for future work.

AuthorsOpenReview ↗

Author response (2/2)

W3. Clarification of the large-system limit and Random Edge matching.

We agree that Section 5 was not sufficiently accessible. We have revised the manuscript to improve readability, clarified the introduction of several concepts, and added a glossary.

Your interpretation of the large-system setting is correct: we consider a sequence of models in which both the target dimension pp and the number of model components HpH_p grow to infinity, while their ratio Hp/pH_p/p approaches a fixed positive constant α\alpha. Such proportional scaling limits are common in the theoretical study of large machine-learning models, as they often yield nontrivial limiting dynamics that can be described analytically.

Importantly, Random Edge is not an alternative to gradient flow. Rather, we prove that the GF dynamics reduces to this random matching process in the vanishing-initialization limit. Intuitively, when the initial weights are very small, different atom--sector pairs initially evolve almost independently, with random activation times determined by their initialization. When an atom–sector pair activates, its atom learns the corresponding target sector, and neither the atom nor the sector can be selected again in subsequent learning events. This competition is described precisely by greedy bipartite matching. For equal-amplitude sectors, the random activation times can then be transformed into uniformly distributed edge priorities, giving the Random Edge process.

Taking the subsequent large-system limit yields an explicit deterministic loss curve, which depends only on the ratio α\alpha after a suitable time reparametrization. Thus, Random Edge provides a mathematical description of the original GF dynamics, rather than a different method for learning the target.

Q1. Example of a diagonal tensor target.

Yes, your understanding is exactly correct. For p=3p=3, ν=3\nu=3, and the standard coordinate bases in all three modes, the target is

+\lambda_2 e_2\otimes e_2\otimes e_2 +\lambda_3 e_3\otimes e_3\otimes e_3,$$ where $e_1=(1,0,0)$, $e_2=(0,1,0)$, and $e_3=(0,0,1)$. These three summands are precisely the scalar sectors. Our framework is somewhat more general: the orthonormal bases can differ between modes. For example, they can be independently permuted coordinate bases: $$T=\lambda_1 e_1\otimes e_2\otimes e_2 +\lambda_2 e_2\otimes e_1\otimes e_3 +\lambda_3 e_3\otimes e_3\otimes e_1.$$ More generally, our framework allows arbitrary orthonormal bases in each mode. This flexibility is important for modular addition, where the target becomes diagonal in suitable Fourier bases, even though it is not diagonal in the original coordinate bases. **Q2. Meaning of "order $\sigma$".** By "order $\sigma$," we meant $O(\sigma)$. Recall that the weights are initialized as $u_k^{(a)}(0)=\sigma g_k^{(a)}$, where the coordinates of $g_k^{(a)}$ are standard Gaussian random variables. Thus, the initial weight coordinates have standard deviation $\sigma$ and their typical magnitudes are proportional to $\sigma$. **Q3. Significance of Corollary 1.** Corollary 1 applies our general Theorem 1 to modular addition, which is the main concrete learning problem considered in the paper. It establishes that, for a CP model with complex weights, the vanishing-initialization GF dynamics reduces to a sequence of learning events in which individual Fourier sectors are learned one by one, according to the greedy matching process. In particular, it explicitly describes the limiting model output and loss trajectory. The corollary also provides the foundation for Theorem 3, where we obtain an explicit deterministic large-system loss curve and its asymptotic relation to physical training time. Importantly, the use of complex weights is essential for this reduction. As discussed in Section 6, with real weights the Fourier decomposition instead produces multidimensional sectors, for which we conjecture a more complicated multi-stage learning mechanism. We hope that these explanations, together with the revisions to the manuscript, have addressed your questions and clarified the scope and significance of our results.
Reviewer_HefzOpenReview ↗

Response to your Rebuttal

Thank you for thoroughly addressing my questions and comments, in particular around the significance of your results. I have raised my score from a 2 to a 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.