- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the gradient flow of matrix factorization problems, i.e. minimization problems of the form . They show exponential convergence under a Polyak-Lojasiewicz (PL) condition on the loss and a condition on the initialization. The rate is explicit in the PL constant and in the initialization.
Extensions are made to gradient descent, deep linear residual networks and the inertial gradient flow.
Critical strengths
The identification and the exposition of the problem structure in Section 2, was new to me, and I found it particularly elegant. It nicely prepares the exposition of Thm 3.1. However, it seems that there are similar results in the literature (see below).
Critical weaknesses
Core contribution lacks novelty, insufficient comparison to the related work. The paper of Min et al. (2021)- on the explicit role of initialization on the convergence and implicit bias of overparametrized linear networks - provide a result (Thm 1) remarkably similar to the Thm 3.1 of this paper. The only difference is that the former restricts to being quadratic (least-squares problem) while this paper proposes the generalization to the PL case. However, the paper of Min et al. is not cited, and there is no comparison to related work in the paper suggesting such a similarity with previous works.
More generally, the related work provides related papers with vague descriptions, but no comparison with related work is provided when the results are provided. Given the example provided above, this shows that the contributions of the paper are not sufficiently well identified and thus the paper can not be accepted.
Poor writing style. The writing is very dense, lacks some general perspective. In some parts I could not understand the jargon (see first paragraph of the intro, for instance). The paper enumerates many variations of a result (Thm 3.1), rather than explaining its core contribution. There is a surprising repetition between Remark 3.2 and Remark 3.5. These problems might be due to an excessive use of a LLM in writing.
Questions for authors
Explain the novelty (both in terms of setting and of technical difficulty) in Thm 3.1.
Minor suggestions:
- l.148: should be .
- Figure 2(b): I don't understand the caption
- l.330: what is a 'layer'? Is it one of the factors or two factors with a residual connection?
2 comments
Thank you for your answer. I understand your decision to withdraw and resubmit.
Response to Reviewer pzdj
The concern about overlap with Min et al. (2021) is addressed in the common responses above.
Thank you, we will write the argument explicitly to avoid ambiguity.
The panel starts from Σ0=Idu+dv and replaces one diagonal entry in each of the u and v blocks by r∈[0,1], leaving the other entries unchanged. Thus g(Σ0)=2r, and the certified loss-decay exponent is 4κr for r>0. The dashed curve shows the r=1 reference rate. At r=0, the support-gap guarantee vanishes. We will revise the caption to specify this initialization and explain the reference curve.
In the residual network setting, "layer" refers to the residual layer obtained by composition of the predictor (the product of two factors) and a residual connection. We realize this terminology clashes with the rest of the paper, where "layer" is used to refer to one of the factor matrices, and we will clarify this in future versions.
Thank you for this feedback. We will revise the exposition to give a clearer overview of the contributions of the paper. We take responsibility for all work and its presentation, including any AI-assisted material.