- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors analyse the gradient descent / flow dynamics of a symmetrised variant of phase retrieval. In particular, they focus on the ratio of sample size to dimension and the values of above which (i) success is more likely than failure (dynamical threshold) and (ii) the optimisation succeeds almost surely (trivialisation threshold). They claim that the success and dynamics of the optimisation process are determined by how well the initialisation is aligned with the target and from this, derive a sample complexity linear in dimension. They support their claim with empirical observations as well as theoretical considerations. They find that in finite dimensions, the two thresholds are distinct, while they coincide in the infinite-dimensional mean-field limit.
Critical strengths
- The clear writing style and structure make it easy to follow and understand the authors' argumentation.
- The paper provides a plausible and comprehensive explanation of the two-threshold phenomenon in the considered example.
- The problem is well motivated and the numerical findings provide a good exposition of the qualitative features of the optimisation properties.
Critical weaknesses
- The analysis is limited to a simple model using synthetic data and it is unclear if the techniques and findings are applicable outside of this controlled setting. It would be good to have some clarification regarding this (or some other justification for restricting to this setting) as otherwise the paper's impact seems fairly limited.
- The argument for the existence of a critical initial overlap in Section 3.2 (which I understand to be the central claim of the paper) is not entirely convincing (see Questions below).
- The derivation of the retrieval time distribution is based on an unfounded approximation and comparison to the empirical distribution is done using only histograms, which is not a reliable method (see also Questions below). The text (l. 417) claims a logarithmic dependence of on , but the curves in Figure 4(a) are not linear within the 75% interval marking the relevant range of .
Questions for authors
- Figure 3(a) contains a region reached by successful and unsuccessful trajectories (which could be a higher-dimensional phenomenon hidden by the projection onto the -plane). Does this contradict the claim that success is determined by ? What happens if a trajectory is started somewhere in this region, e.g. with ?
- Have you tried initialisations which do not lie on the curve ? Figure 3(a) suggests there may be some higher-dimensional manifold separating successful and unsuccessful initialisations whose intersection with the sphere of radius , on which the initialisations lie, is described by . In low dimensions, it may be possible to do a kind of grid search to get a comprehensive picture of which initialisations fail and which succeed. Have you done such an experiment?
- Do you claim validity of Eq. (12) for all ? The approximations and seem valid only close to initialisation and validity for seems necessary in order to derive the retrieval time.
- For comparing the empirical distribution of to Eq. (14), can you provide any other statistical evidence beyond the three histograms in the paper, like the empirical CDF, q-q-plots or statistical tests?
- For computing in Eq. (11), how is chosen? Does it vary over the samples?
- In Figure 4(a), why are only successful trajectories considered? How do you deal with the possibility that the optimisation fails in your analysis of ?
- At the end of Section 3.4, you state that the shape of the retrieval time distribution may provide evidence for some underlying multiplicative growth. Which distribution shapes would indicate something like this and how would you infer a multiplicative growth mechanism?
- Figure 2(a) shows the loss landscape in terms of and for some fixed . What does this picture look like for other , is this representative for the general shape of the loss landscape?
6 comments
Response to reviewer Ag24
W3, Q3 (validity of Eq. (12)). The reviewer caught an important weakness in our argument. Thank you. We do not claim that Eq. (12) holds for all , and we have rewritten Sec. 3.4 to state its regime of validity. The argument now has three steps.
(i) Fig. 4(a) shows that, at fixed , the retrieval time is a function of , up to fluctuations; for this function is given by DMFT.
(ii) Eq. (12) describes only the early escape from the symmetry plane, and only for initial overlaps well above the critical one, . It cannot hold near : a constant growth rate would imply that all trajectories retrieve, whereas there trajectories linger at the basin boundary. This is the origin of the non-linearity in Fig. 4(a) that the reviewer points out: is logarithmic in away from and steeper as is approached. Eq. (14) follows when most random initializations are in this regime, , as for at . Nor do we claim that the growth is exponential up to : stages whose duration does not depend on only add a constant to . The final approach to the teacher is such a stage: from to it takes a time of about at , with a standard deviation below between trajectories, so the choice of threshold only shifts . is therefore an effective rate; this is now clearly stated in Sec. 3.4, which was substantially edited. For comparison, Fig. 4(b) now also shows the exact full-population prediction (App. D), which has no fitted parameter.
(iii) When a significant fraction of the retrieving trajectories start near ( at ), these assumptions do not hold. Empirically, Eq. (14) with a fitted rate still describes the distribution, but we do not have a derivation for this regime.
Q4 (statistics of ). We have added App. F (Fig. 9), comparing the empirical CDF of with Eq. (14) at , . is the single fitted parameter (maximum likelihood). The Kolmogorov-Smirnov distances are , showing that Eq. (14) is a good description of the distribution (fitting in addition a time shift does not consistently improve the agreement).
Q5 ( in ). is the student's transverse direction at time , an -dimensional unit vector orthogonal to the teacher, shared by all samples and evolving in time. and contract this high-dimensional tensor to a scalar, so Eq. (11) is exact (App. B), but its coefficients change in time. In Fig. 2(a,b), is frozen at its value after training, for visualization purposes.
Q6 (successful runs only). is defined only for runs that reach , so its statistics are conditional on success; the failure probability is shown in Figs. 3(b) and 1(b). Eq. (14) neglects this conditioning, which is justified when so that almost all runs retrieve ( at ), and not for ; see point (iii) above.
Q7 (multiplicative growth). Eq. (12) gives Eq. (13), , so in Eq. (14) has an exponential right tail , from small , and a sharp left cutoff. Conversely, observing this shape points to multiplicative growth, and the tail's decay rate estimates , even when the growing coordinate is unknown. Since this remark was speculative and not supported by an analysis in the paper, we have removed it from the revision.
Q8 (other in Fig. 2(a)). Fig. 2(a) uses the reached by gradient descent on the original problem at . Other seeds give the same qualitative picture: the two teacher minima and one spurious minimum, whose position varies (with positive or negative ). For a random , is Gaussian and : no spurious minimum, only a saddle at , .
We believe that the revision addresses the concerns about soundness raised in the review. If the reviewer agrees, we would appreciate it if they would reconsider their score.
Thank you for the detailed response and revised manuscript. Could you maybe highlight the changes in a different colour to make them easier to identify?
W1
The clarification and helpful additional explanations have made the relevance much clearer. I share the sentiment that a comprehensive understanding of a simplified setting is a valuable contribution. However (as Reviewer w64F also mentions), the unclear applicability to the relevant general setting still remains a weakness and it would significantly strengthen the manuscript (for ICLR or future submissions) if more insight could be provided.
W2, Q1-Q2
I like the new Section E, thank you for providing this additional evidence and the explanation for why the -sphere is a natural choice for initialisation. I am still not convinced by the claim that "trajectories starting with retrieve the teacher, and those starting with converge to a nongeneralizing state", even when including the new dependence of on . Fig. 3(a) shows that there is some region (passed by the failing trajectories) and a region (passed by the successful ones) whose projection onto the -plane is the same (however far apart they may be in the full, -dimensional space). If a trajectory was started in region , say with and , it would continue along one of the orange lines of Fig. 3(a) and subsequently fail (a point visited in the intermediate stages of one of the orange trajectories could also be used as a starting point). But the rectangle in Fig. 8 would imply success almost surely. This suggests that whether a trajectory fails or succeeds has an additional dependence on the type of initialisation. In other words, if the probability of success was only determined by for all initialisations, projecting trajectories onto the plane should retain the boundary of Fig. 8. What your findings do seem to show is that for typical choices of initialisation (e.g. standard Gaussian with concentration on the -sphere), there is indeed such a threshold . I would be interested to get your thoughts on this and if this issue was resolved (e.g. by discussing this dependence on initialisation and adapting the claim in the paper) I would definitely raise my score.
W3, Q3
While the revised Section 3.4 is more sound, I find the new exposition a bit confusing. As I understand it, you claim validity of Eqns. (13) (up to a constant) and (14) in the setting where . If this is correct, it would be good to make this clearer and adjust the claims about the findings of this section in the introduction and conclusion (in line with the explanation "We note that ..." starting in l. 487). Regarding the exposition, I still have some open questions:
- If you do not claim that the growth is exponential up to , why should Eqn. (13) be valid (even up to a constant)?
- What is the motivation for replacing by ? What do you mean by your claim that this approximation "quantitatively captures the retrieval time distribution"? If you use your empirical findings to motivate this theory and then subsequently claim that the theory is supported by these findings, the argumentation seems circular.
- Can you provide more details about your experiment on the final approach to the teacher? Over which trajectories and ranges of did you compute these statistics? It would be good to include more details in the paper to support your claim that the final approach only adds a constant.
Q5
I unfortunately still do not understand how is chosen. By "the student's transverse direction at time ", do you mean the (normalised) projection of onto the orthogonal complement of ? If this is the case, how is shared by all samples, are these projections then averaged over all samples to obtain ?
Q4 & Q6-8
Thank you for the responses, which have resolved these questions.
Response to reviewer Ag24 (1/2)
We thank the reviewer for the detailed questions, which pointed to passages that were not clear enough; we have revised them as detailed below.
W1 (simple synthetic model). Phase retrieval is a notoriously hard problem [Fienup 1982; Shechtman et al. 2015; Fannjiang & Strohmer 2020; Candès et al. 2015; Sun et al. 2018; Chen et al. 2019; Barbier et al. 2019; Maillard et al. 2020b; Mignacco et al. 2021]. There is an extensive and currently active research effort devoted to understanding it even, and sometimes especially, in toy models with Gaussian data and a single quadratic neuron [Chen et al. 2019; Sarao Mannelli et al. 2020a; Maillard et al. 2020a; Mignacco et al. 2021; Bonnaire et al. 2025; Tsironis & Moustakas 2025; Maillard et al. 2026]. This is an open and important problem. As Reviewer w46F writes, recovery with gradient-based algorithms in phase retrieval "is still unknown", and a mechanism that carries over to it would be "an important step towards its comprehension". We make a substantial contribution to this problem by studying the simplest variant in which the question of generalization can be answered completely. This is in the spirit of Bonnaire et al. (2025), who analyzed a variant of the original problem. Symmetrization is such a variant: it only restores at finite sample size a symmetry of the population loss, yet makes the problem tractable. The history of this and related fields has many examples in which the study of simple solvable models produced insights that later proved general, both in learning theory [Seung et al. 1992; Saad & Solla 1995; Saxe et al. 2014; Advani et al. 2020; Goldt et al. 2020] and in high-dimensional non-convex optimization [Sarao Mannelli et al. 2019; Antenucci et al. 2019; Aubin et al. 2020; Maillard et al. 2020a].
W2, Q1-Q2 (existence of a critical overlap; Fig. 3(a); other initializations). We thank the reviewer for this question, which showed us that Fig. 3(a) was not presented clearly enough. We have revised the text and the figure accordingly. In short, Fig. 3(a) directly demonstrates that, for the parameters used there (N=1000, α=10), trajectories starting with R0>Rc retrieve the teacher, and those starting with R0<Rc converge to a nongeneralizing state. The two ensembles differ only in R0 (0.03 and 0.07, on either side of Rc≈0.052), so the outcome is set by R0. Fig. 3 makes this quantitative: as R0 is scanned, the retrieval probability rises from 0 to 1 around Rc, the rise sharpens as N−1/2 (Fig. 3(c)), and its location agrees with the DMFT prediction for all α studied (Fig. 3(d)). Together these imply a step at Rc(α) as N→∞, Eq. (8).
The region visited by both ensembles does not contradict this. The full dynamics also involves the transverse direction u^ and its relation to the data, and Fig. 3(a) shows only its projection on the (R,s) plane. Two trajectories can therefore pass through the same point of the plane with different futures (in the reviewer's words, "a higher-dimensional phenomenon hidden by the projection"). The region is shared only in the projection, not in the full state space. Failing trajectories reach it only later, once u^ has relaxed toward the data. We realize that the figure did not make the direction of time visible, so we have added arrows along the trajectories in Fig. 3(a) and refer to them in Sec. 3.2.
Following the reviewer's helpful suggestion (Q2), we have also added a grid search over the initial overlap and the initial transverse norm, (R0,s0), with 0.1≤s0≤2 and the transverse direction drawn at random, at N=1000 and α=10,15,20 (new App. E, Fig. 8). This does not change the picture: for every s0 there is still a sharp critical overlap, only at a different value Rc(s0). For the reviewer's example, R0≈0.1, s0≈0.5 (at α=10, as in Fig. 3(a)), we find Rc(s0=0.5)≈0.02, so a trajectory started there retrieves the teacher, while one started at the same s0 with R0=0.01 fails. Rc(s0) is almost proportional to s0, with slight deviations. Random initialization concentrates on s0≃1, which is the case studied in the main text.