- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies the sensitivity of bias-free two-layer linear network predictions to training-lable perturbation when a network is initialized close to zero. The main convergence theorem (Theorem 1) showing the sensitivity is established for both GF and GD. The authors further claims that the sensitivity phenomenon can be missed by Propagated GN approximation (Section 2).
Critical strengths
Their main convergence theorem (Theorem 1) may potentially contain interesting mathematical results. However, I have concern about their interpretation, which may affect its significance.
Critical weaknesses
- First of all, I find the submission hard to follow due to inconsistent and unclear technical notations in addition to the lack of motivation before introducing such notations and their main theorem. For example, Section 2 introduces A_\varepsilon and t_\varepsilon together before introducing \varepsilon. There is inconsistent use of A_t and A_\varepsilon in Section 2. You have to wait until Section 3 for the formal introduction of \varepsilon, where the motivation or the role of \varepsilon is not necessarily clearly stated. It seems that all of the notations in Section 2 can be introduced without \varepsilon, and it will help fix the inconsistency issue. The paper keeps introducing technical notations in (eq-6) without explaining what they are and go straight forward to state Theorem 1. (eq-7) seems to be their main mathematical finding but its interpretation is not given. If my understanding is correct, the authors seem to claim that as \varepsilon \to 0, the emergence time T_L gets larger while the perturbation size z/L gets smaller. Thus the convergence to the profile which depends on \tau and z shows the growing sensitivity to a small label perturbation as initialization gets smaller. However, this interpretation is not immediate from the current main text.
- In addition to the interpretability issue, I have a substantial concern with the mathematical significance of Theorem 1. While the authors seem to claim that (eq-7) shows the sensitivity to a small perturbation, it appears to me that it is merely the effect of change-of-variable. This makes me wonder whether the sensitivity interpretation is the right way to see this equation, and wonder how general their claim could be.
- The authors claim in Section 4 that the phenomenon observed in Theorem 1 can be missed with the propagated GN approximation. I wonder if this is actually surprising. Comparison with the GN approximation would make sense if the R_t term is negligible, but is it really the case here? If R_t is not negligible, then it is not surprising that the GN approximation exhibit a different behavior.
Questions for authors
- Can you provide a very clear, informative interpretation of Theorem 1 along with more detailed explanation of technical notations (I appreciate if you could correct inconsistency among notations altogether)?
- Can you clarify elaborate on your sensitivity interpretation? Is it different from the effect of change-of-variable?
- Can you elaborate on why the GN approximation is the right object of comparison?
6 comments
Revision uploaded: interpretation of Theorem 1 and the GN comparison
Thank you for identifying the notation and presentation problems and for questioning the interpretation and significance of Theorem 1. We have uploaded a revised manuscript addressing each of these concerns.
1. The theorem and its notation are now explained before use
The abstract and introduction now explain the prediction question, linear influence, and feature emergence in words. The introduction defines the initialization scale and . Section 2 distinguishes the time-dependent derivative from its checkpoint value . Section 3 defines the growth rate, emergence time, phase, and timing direction before Theorem 1, then adds “Reading the result” and “Why the nonlinear-network proof is needed.” Figure 1 illustrates the mechanism.
The theorem concerns smooth, bias-free nonlinear two-layer networks, with both layers trained and any fixed finite width, under gradient flow or fixed-step GD. The solvable scalar model is an illustration.
2. The sensitivity is a fixed-time prediction difference, not a coordinate artifact
No, the result is not merely a change of variables. Both runs start from the same parameters and are evaluated after the same training time. Changing the horizontal axis of a plot cannot remove the difference between the predictions recorded at that checkpoint. Stopping each run at its own learning stage changes the experiment.
Here is the distinction in the theorem. Let denote the limiting prediction curve through emergence, with measuring time relative to the nominal transition. A label perturbation produces a timing advance , which remains finite even as that perturbation vanishes at the critical scale of order . As initialization tends to zero, the actual change and linear forecast have the limits
\text{linear forecast}\ \longrightarrow\ \Phi'(\tau)d.$$ Linear influence follows the tangent; retraining follows the curve. Under the theorem's conditions, this distinction gives a strictly positive limiting relative RMS error for training predictions at the critical Gaussian noise scale. It is a statement about observable predictions in the original training procedure. The proof uses normalized time to control the dynamics, but the theorem returns to the original, common observation time. The difficult step is proving that this return is valid for both the nonlinear trajectory and its label derivatives. Ordinary smooth dependence over a fixed time interval does not supply uniform bounds over the waiting period, which grows as initialization shrinks. Lemma 4 establishes the required control; Appendix A.2 derives the growing timing term and bounded remainder; Appendix B establishes the necessary-and-sufficient noise boundary. Thus the result does not follow by applying the chain rule to an assumed limiting curve. The alternative observation rule is explicit: under the matched-progress conditions, allowing each run a different stopping time cancels the leading timing term (Eq. (11), Appendix E). **3. Why the GN comparison is informative** Residual curvature is **not negligible** here. During the long initial small-output phase, the network has not fitted the labels, and its residual approaches the negative label vector. Section 4 now states this directly. The point is not that omitting a non-negligible term can change an answer. We identify **which part of the response is lost, how its size scales, and a sufficient way to recover it**. Propagated GN follows the same nominal training trajectory and retains every feature update and the same label forcing. Under the theorem's conditions, the positive-semidefinite propagation in Section 4 remains $O(1)$ at emergence, while the exact response grows as $\Theta(L)$. At critical noise of order $1/L$, the GN prediction therefore vanishes while the actual prediction change remains nontrivial. Its relative RMS error tends to one, the error of predicting no change. The discrete result requires the stable-propagation conditions stated in Appendix D. This separation has a constructive counterpart: under gradient flow, Corollary 2 retains residual-curvature action along the parameter velocity using a correction of rank at most two and recovers the leading response with bounded error. It identifies a sufficient curvature component to restore, rather than requiring the full residual Hessian. In bias-free B1, the median operator norm of the exact response divided by $L$ changes from 1.28 to 1.23 as $L$ increases from 5 to 40, while the unscaled GN norm stays between 1.59 and 1.65 (Section 6). For reference, original Eq. (7) is now Eq. (5), original Eq. (15) is Eq. (11), and original Table 1 is Table 2. Theorem 1 and Corollary 2 retain their numbering. We would appreciate your assessment of these clarifications.Thanks for your revised manuscript and I appreciate the authors effort. However, the revision contains extensive changes, including substantial rewriting and restructuring as well as noticeable shift in the positioning of the paper. While I appreciate the authors effort to clarify our concerns, I am not sure reviewers are expected to conduct a new review of the revised manuscript during this short period.
As for my concerns,
- I don't think the authors made explicit my interpretation of equation (5) is correct or not directly.
- I was not asking whether the proof requires technically difficult argument or not. I was asking whether the mechanism can be largely reduced to cancellation of scaling: T_L scales by L while z/L scales by 1/L and they cancels out. It seems to me the equation preceding Theorem 1 reinforces my point. Since the network remain close to zero until it reaches near , I'm not sure if this is something to be surprised at. Again I want to make it clear that I'm not saying proving Theorem 1 is easy. I recognize the technical difficulty of proving it rigorously. My concern is solely about the significance. "Sensitivity is a fixed-time prediction difference" seems to reinforce my point. It seems to me the choice of fixed time is the key of equation (5).
- I think your correction made my point for the propagated GN approximation. Since is not negligible, it is not surprising the approximation exhibit a different behaviour.
Thank you for identifying the clarity problems. We will clarify the abstract, introduction, and explanation of Theorem 1.
Q1. What does Theorem 1 mean, including Eq. (7)?
The theorem concerns smooth, bias-free nonlinear two-layer networks, with both layers trained and any fixed finite width. Label changes shift when a feature is learned, so two runs evaluated at the same time can be at different learning stages.
Let ϵ be the initialization scale and L=log(1/ϵ). Under gradient flow, the leading emergence time is TL(y)=L/λ(y), where λ(y)>0 is the label-dependent growth rate. The transition occurs over an O(1) time interval. For δy=z/L, Eq. (7) states:
Fϵ(TL(y0)+τ;y0+z/L)=Φ(τ+q⊤z)+o(1).
Here Fϵ(t;y) denotes predictions at time t for labels y, and y0 denotes nominal labels:
The remainder vanishes as ϵ→0 for fixed τ,z. The evaluation time is fixed using nominal labels, not adjusted for each perturbed run.
Linear influence replaces the shifted curve by its tangent. Under the stated assumptions and isotropic Gaussian label noise of standard deviation σ, its relative RMS error vanishes as ϵ,σ→0 if and only if σL→0. At σ≍1/L, the label change vanishes but the timing shift remains finite.
We will introduce ϵ at the start of Section 2, distinguish the time-indexed derivative At from its checkpoint evaluation Aϵ, and define every symbol in Eq. (6) before use.
Q2. Is this merely a change of variable?
No. The result is not merely a change of variables, for two reasons.
First, the comparison is fixed and observable. Both runs share their initialization and training time. Changing coordinates does not remove their prediction difference. Comparing runs at their own emergence times changes the question; under the stated retiming conditions, the order-L term cancels and the response remains bounded (Eq. 15, Appendix E).
At L=40, phase 0, and critical noise level c=1, Table 1 reports median fresh-output relative RMS errors of 0.374, 0.430, and 0.128 for exact linear influence in B1–B3, versus 0.088, 0.088, and 0.025 for the rate-clock correction, over five paired initialization seeds. B1 matches the theorem's bias-free architecture; B2 and B3 train biases and test beyond that assumption.
Second, we prove the nonlinear emergence profile and control its label derivatives. The chain-rule calculation is elementary once a sufficiently regular profile exists. The required proof establishes:
Appendices A.1 (Lemma 4) and A.2 establish derivative control and the response expansion; Appendix B proves the noise boundary. A nominal trajectory limit alone does not justify differentiation with respect to labels.
First emergence is covered for gradient flow and fixed-step GD; GD's Gaussian statement requires stability and moment conditions. Later-transition results require the diagonal model's separated activations and no deactivations; they do not cover general MLPs.
Q3. Why GN? Is residual curvature negligible?
No. It is not negligible for the leading response. During the long small-output phase, residuals approach −y0; in this model, residual curvature generates the unstable growth responsible for timing amplification. We ask: does following evolving features suffice to recover that response?
Propagated GN uses the same nominal trajectory and label forcing as exact sensitivity, including every feature update. Under the theorem's assumptions, exact sensitivity is Θ(L), whereas positive-semidefinite curvature propagation in this setup remains O(1). GD additionally requires stable propagation. Table 1 gives GN errors of 0.959–0.961 across B1–B3; predicting no change gives 1.
This curvature choice is relevant to attribution: SOURCE uses approximate unrolling with an EK-FAC approximation to Gauss–Newton curvature (Bae et al., NeurIPS 2024, Section 3.3).
Corollary 2 and Appendix D.1 identify a sufficient repair. Under gradient flow, a rank-at-most-two correction retaining residual-curvature action along parameter velocity recovers the leading order-L response with bounded error remaining. Its action requires a residual-Hessian–vector product.