acceptodds
Under review as a conference paper at ICLR 2027

Logarithmic Activation Clocks in Diagonal Linear Networks: When SGD Does and Does Not Follow Lasso

Abstract

Under a path-monotonicity condition, the from-start average of gradient flow in a diagonal linear network, rather than its raw output, recovers the Lasso regularization path. We ask what survives for stochastic gradient flow (SGF) and finite-step stochastic gradient descent (SGD). Integrating the square-factor log dynamics yields the Lasso KKT system plus coordinatewise Itô variance clocks and a self-normalized martingale. In an exactly solvable least-squares model, fixed SGF noise and a fixed SGD step produce different weighted-Lasso paths; ordinary Lasso therefore need not survive in either regime. In weak noise, only clock mass accumulated before future support motion enters the comparison. Under the stated nonexplosion, localization, and path-variation conditions, the averaged path returns to Lasso without requiring . For the standard signed product parametrization, an exact rotation produces two nonnegative channels driven by the same minibatch, with distinct finite-step clocks and . When the positive and negative Lasso channels have zero downward variation, the stopped, averaged actual-SGD path converges in prediction norm and signed-Lasso objective as and , again without an condition. For correlated row sampling, our positive results retain explicit tail-control assumptions; a universal off-face excursion bound remains open.

Primary area:
optimization
Keywords:
Hadamard reparametrization, diagonal linear networks, implicit regularization, Lasso path, stochastic gradient descent, sparse optimization, early stopping

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mAiCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies SGD and Stochastic Gradient Flow of quadratic objectives and an overparametrized Hadamard model (two layer linaer diagonal network, i.e. replace one parameter vector with the element wise product of two parameter vectors) in order to determine if the paths of the 'from start average' (over these algorithms trajectories) correspondence to a Lasso path. Previous work showed the Gradient Flow (not stochastic) yields a correspondence with a 'deterministic Lasso path' (i.e. the set of solutions to the Lasso problem with regularization 1/t, plotted against increasing t). This paper aimed to test if the same can be shown for SGF and SGD. Their first main result shows that for 'fixed noise' the paths recovered by both SGD and SGF correspond to that of a Lasso path with a coordinate wise weighting, with distinct weighting for SGF and SGD. Hence both objects yield solutions which are distinct from eachother and non-stochastic gradient flow. The second main results considers the 'weak noise' setting where the initialization and step size tend to 0. They show that as η→0\eta \to 0 (or δ→0\delta \to 0 for SGF) both SGD and SGF recover the deterministic Lasso path, even if Lαη↛0L_\alpha \eta \not\to 0 (or Lαδ↛0L_\alpha \delta \not\to 0). The reason is a 'future-active clock' argument: only the noise/variance accumulated on a coordinate before its next future move in the true Lasso path actually contributes to the final error, so the total accumulated noise over the whole horizon is not the relevant quantity. This recovery is conditional on a localization/stopping-time argument (whose failure probability vanishes) and, for the actual-SGD case, on the true path never deleting or reversing a coordinate's sign.

Critical strengths

The paper extends previous analysis comparing gradient flow to deterministic Lasso paths to include both SGF and SGD in order to demonstrate that these processes do not all behave the same in the fixed noise case but also demonstrates conditions under which they all recover the deterministic Lasso Paths. This is a clear improvement from past results as far as I undestand.

Critical weaknesses

The entire paper is very difficult to read. It feels like on is expected to already be an expert in this precise specific sub-field in order to understand the terminology that is casually thrown around from the very start of the paper. This may or may not be a purposeful design choice from the authors but for a conference like ICLR it feels like the writing is not well aligned to the target audience. I did my best to understand the main intent behind the main theorems in order to understand the contribution.

The scope of the results is extremely limited in that it is one very specific toy model being considered. I will acknowledge their results continue an existing line of work and in fact make it more general, but nevertheless I want to emphasize that the work focuses on a very specific toy problem.

Questions for authors

The first main result (Theorem 4.1) makes use of the 'EXACTLY SOLVABLE AXIS SAMPLER'. Do you expect the results to hold more generally? If so what would be the main bottleneck in doing so? As disclosed in the paper the proofs came from a conversation with ChatGPT-5.6 Sol, so if the majority of the technical details are outsourced, then I am curious why it was not feasible to prove things at a greater level of generality. My assumption being that: if more generality came at a cost of making things 'messier' or more tedious, then this cost could be paid with the LLM, whereas if there is a greater cost (such as the core approach or mechanisms used in the proofs do not extend) then can one believe these results actually contribute to understanding a phenomenon more general?

2 comments

1. Difficulty reading the paper and suitability for the ICLR audience

Common response: C1 and C4.

C1 summarizes the presentation changes, and C4 explains why the path characterization matters. We revised the introduction to make this motivation explicit. The revision distinguishes the instantaneous network predictor from its average along a training run and identifies where balanced or signed factors are used. The numerical results are presented as tests of these mechanisms on specified datasets.

2. Scope limited to a specific toy model

Common response: C2; C3, especially point 5; C4.

The revised introduction clarifies the motivation discussed in C3, point 5, and C4: a linear predictor and quadratic loss let us isolate the effects of sampling and finite updates. This is still a two-layer diagonal linear network; the data extensions do not change the architecture or loss.

Within this model, Section 5 treats correlated data, signed predictors and rank-deficient designs, including d>nd>n. Under the stated trajectory hypotheses, it compares averaged predictions with averaged ISS and gives an exact criterion for Lasso prediction and objective recovery. Section 5.4 also gives a correlated nonnegative SGD class for which all assumptions are proved.

Question. Does Theorem 4.1 hold more generally, and what is the main bottleneck?

Common response: C3, point 4 (fixed noise and fixed steps); C2 (general averaged dynamics).

Theorem 4.1 uses a dataset where each observation has only one nonzero feature. The logarithmic identities extend to correlated designs, but they do not automatically give the same fixed weighted-Lasso description. For the balanced nonnegative SGF model, Theorem A.1 assumes H≻0H\succ0 and joint uniform convergence in probability of the integrated predictor and accumulated variance to deterministic limits on a compact interval of positive rescaled time as initialization vanishes. It also assumes the limiting optimality conditions stated there: the integrated predictor and its dual slack are nonnegative and have zero product in each coordinate. Under these hypotheses, the limiting average solves a weighted Lasso problem constrained to nonnegative coefficients, with penalty coefficients (1+Ai(s))/s(1+A_i(s))/s. Here Ai(s)A_i(s) is the limiting accumulated variance correction, so the weights can vary with rescaled time ss.

Proposition A.2 separately proves an SGF limit for a symmetric correlated example with four observations sampled in two equiprobable blocks. Its scalar Lasso penalty is 1/s+δσ2/21/s+\delta\sigma^2/2, where δ\delta is the SGF noise level and σ2\sigma^2 is the constant coordinate gradient variance in that example. This gives a proved extension beyond the dataset in Theorem 4.1.

For a general correlated SGF extension, the additional task is probabilistic control of accumulated gradient variance, including time near inactive coordinates and repeated departures from and returns to those regions. The estimates in Appendix A.4 do not bound this total contribution. More algebraic bookkeeping alone does not supply the required global estimate, and the disclosed AI assistance did not supply it. A correlated fixed-step SGD result would also require control of its exact finite-step logarithmic corrections.

The general comparisons in Theorem 5.1 and Corollary 5.2 address a different limit. As initialization shrinks, the SGD step size tends to zero, or the product of the SGF noise level and the localized variance bound tends to zero. Under the respective trajectory hypotheses, the averaged predictions then approach those of averaged ISS. This comparison establishes the general averaged reference when the stochastic or finite-step corrections vanish.

Thank you for the response. I want to note that it is not clear how much was rewritten to address the commonly identified issue of the text being poorly written (which is now identified with AI usage) and it is unreasonable to expect reviewers under the tight deadlines of ICLR to completely re-read and re-review the entire paper to determine if it was well written this time. I appreciate the additional results but I do not believe they bring the significance of the results up to the acceptable level and so I will maintain my score.

Reviewer_LG6QOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies the implicit regularization induced by early stopping in diagonal linear networks. Previous works had studied the implicit regularization of the convergence point of the gradient flow, of gradient descent and in the presence of noise, as well as the implicit regularization of the early-stopped gradient flow. This article builds upon these to study the implicit regularization in the combination of early stopping and the use of gradient descent or a noisy gradient flow.

The paper provides a machinery for this analysis through a KKT identity involving additional terms due to noise. This machinery reveals how the analysis differs from the deterministic case. For orthogonal designs, the results are made explicit: a weighted Lasso penalty appears, where weights depend on the elapsed time and the noise.

Critical strengths

Relevance. This paper aims at extending the previous works on implicit regularization of diagonal linear networks to a setting with early stopping and the use of stochastic gradient descent. This is a significant step towards practice. However, the goal of the implicit regularization line of work is to provide intuitions on the resulting effect of the different mechanism. While the technical effort of this paper is significant, it lacks this effort of exposing clearly and discussing the result to build some intuitions (see below).

Critical weaknesses

The weaknesses listed below are typical of an excessive use of LLMs to generate the paper.

Clarity. The work makes excessive use of some jargon, that has not been defined earlier in the paper and that is unclear to me, even though I am familiar with the related work. This can be seen from the abstract and the introduction, for instance. Similarly, many mathematical notations are used before they are defined (and they are not classical).

Examples:

  • In the abstract and l.68, what is δα\delta_\alpha?
  • l.121: what is an "affine-quadratic" function? How is this different from the word "quadratic" used in l.114?
  • Where is the derivation of Eqs. (3.3)?
  • Paragraph starting l.143: it is not clear from the exposition that the discussion now focuses on the balanced case. Overall, the result could make much more clear which sections assume the balanced case.
  • Ll. 185-188: What is the problem discussed here? This is very implicit.
  • l.271: Why is it interesting that no condition of the form δαLα→0\delta_\alpha L_\alpha \to 0 is required?
  • l.324: what is ρα\rho_\alpha?

Technical rather than digested exposition. This paper is a technical account of attempts at solving the problem, rather than a digested global picture that emerges. The result is composed mostly of technical derivations and technical jargon. Little is left for pedagogy and simulations. The setting varies constantly from one section to the next, from one result / example to the next, sometimes not even explicitly. The paper does not come with a clear take-home message. This could be the output of an LLM that has not been digested by a human writer. As it is, the paper would be of little use for most readers. A few specialists could find some computations useful for follow-up work.

Superficial related work. Some related work is missing, and some is inappropriately cited. See below.

Examples:

  • l.43: Pesme et al., 2021 tackles the stochastic case, whereas this paragraph is about the deterministic setting. Instead, one could mention Woodsworth et al. (2020) for the l1 penalty at convergence. For the regularization of the whole path, the earliest contributions are Berthier (2023, JMLR), Pesme and Flammarion (2023) and, as already appropriately cited, Berthier (2025).
  • Related work, first paragraph: Suggala et al., 2018 - "connection optimization and regularization paths" could be cited by Ali et al., 2019.
Questions for authors

Detail significantly the AI use statement, in particular in terms of writing the paper.

3 comments

1. Clarity: undefined jargon and mathematical notation

Common response: C1.

We sincerely apologize for the confusion caused by our excessive reliance on AI when drafting the initial submission under time constraints. Although this use complied with ICLR policy, it contributed substantially to the confustion here. We have since manually rewritten almost all previously AI-generated prose, improving the paper's structure and clarity. We further address the seven examples individually.

  1. What is δα\delta_\alpha in the original abstract and introduction? The rewritten opening no longer uses this symbol. Section 2 defines δ\delta as the effective SGF sampling-noise level, including batch-size scaling, and writes δα\delta_\alpha when that level depends on the initialization scale α\alpha. Appendix B.1 states the factor SGF.
  2. What does "affine-quadratic" mean? Section 2 now states the precise property: each entry of the sampled least-squares gradient covariance is a polynomial of degree at most two in β\beta. This describes the covariance; the loss remains quadratic. Equation (2.3) defines the covariance Σ\Sigma and coordinate variances ViV_i.
  3. Where is the derivation of the original equation (3.3)? The factor-SGF model is now equation (B.1) in Appendix B.1. Its construction uses the sampled factor gradient (v⊙G,u⊙G)(v\odot G,u\odot G), where GG is the sampled predictor gradient with conditional mean gg and covariance Σ\Sigma. Let KK be the matrix formed by stacking Diag⁡(v)\operatorname{Diag}(v) above Diag⁡(u)\operatorname{Diag}(u). The factor gradient has conditional mean KgKg and covariance KΣK⊤K\Sigma K^\top. The SGF therefore uses drift −Kg-Kg and diffusion δKΣ1/2\sqrt{\delta}K\Sigma^{1/2} driven by one Brownian motion, preserving the covariance between the factor updates. Appendix B.1 also derives the predictor and logarithmic SDEs, equations (B.2)-(B.3). On the balanced slice it applies dβi=2ui dui+d⟨ui⟩d\beta_i=2u_i\,du_i+d\langle u_i\rangle, then the logarithmic Ito formula, obtaining the term −δVi(βt) dt-\delta V_i(\beta_t)\,dt in (B.3).
  4. Which sections assume the balanced case? The general prediction comparisons and Lasso criterion in Section 5 allow H⪰0H\succeq0 with r∈range⁡(H)r\in\operatorname{range}(H), under the stated stochastic hypotheses. Balanced initialization means u0=v0>0u_0=v_0>0. The shared sampled-gradient updates preserve u=vu=v, so βi=uivi=ui2≥0\beta_i=u_iv_i=u_i^2\ge0. Sections 3 and 4.1 use this balanced nonnegative model. Section 4.2 first derives the logarithmic identities in that model and points to Section 5.1 for signed updates. Section 5 permits different initial factors and signed predictors and states the verified nonnegative class separately. Appendix B specifies the scope of its nonnegative SGF results locally.
  5. What problem was discussed in the original lines 185-188? Section 2 now distinguishes the instantaneous network predictor from its average over training since initialization. Equation (2.7) shows the difference in the one-dimensional deterministic example: away from the activation time, the instantaneous predictor has a step-function limit, while its average has the continuous Lasso soft-threshold form. The path comparisons concern the latter estimator.
  6. Why is it useful not to require δαLα→0\delta_\alpha L_\alpha\to0? Put Lα=log⁡(1/α)L_\alpha=\log(1/\alpha). Section 3 gives the example δα=Lα−1/2\delta_\alpha=L_\alpha^{-1/2} or ηα=Lα−1/2\eta_\alpha=L_\alpha^{-1/2}. The corresponding recovery conclusion holds although the product with LαL_\alpha diverges. Section 4.2 and Appendix B.4 explain why: the logarithmic equation is normalized by LαL_\alpha over Tα(s)=sLα/2T_\alpha(s)=sL_\alpha/2. Error in the unscaled logarithmic equation can grow while the error relevant to the average vanishes. The sharper diagonal SGF bound in Appendix B.4 retains its nonexplosion and bounded-variance assumptions.
  7. What is ρα\rho_\alpha? The failure-probability parameter in Appendix B.4 is now written pfail∈(0,1)p_{\mathrm{fail}}\in(0,1). Appendix B.5 chooses a sequence pfail,α↓0p_{\mathrm{fail},\alpha}\downarrow0 with log⁡(1/pfail,α)=o(Lα)\log(1/p_{\mathrm{fail},\alpha})=o(L_\alpha). This is the confidence notation corresponding to ρα\rho_\alpha in the original submission. It is distinct from the coordinate growth rates ρiF,ρiD\rho_i^{\mathrm F},\rho_i^{\mathrm D} in Section 4.1 and from the correlation parameter in the example of Section 5.4.

2. Technical rather than digested exposition

Common response: C1 and C2.

The revised introduction motivates the averaged-path question; C1 and C2 summarize the main result order and technical additions. Section 4.2 defines and explains the accumulated corrections before presenting the correlated example and numerical checks. Section 5 then treats correlated data and signed predictors under the stated localization hypotheses, using averaged inverse-scale-space (ISS) dynamics, and gives the exact condition for Lasso agreement. The scalar example explains why averaging matters, and Section 2 states how the positive-square and signed models are related. Supporting estimates and full numerical specifications are in the appendices.

3. Superficial related work and citation placement

Common response: C4.

The introduction now cites Woodworth et al. (2020) for the deterministic small-initialization ℓ1\ell_1 connection and discusses Pesme et al. (2021) with stochastic endpoint results. Section 6 includes Berthier (2023, JMLR) for incremental activation in its underparametrized anti-correlated class, Pesme and Flammarion (2023) for the raw trajectory, and Berthier's averaged Lasso-path work. Suggala et al. (2018) is cited alongside Ali et al. (2019) for optimization and regularization path comparisons. Section 5 credits the deterministic ISS reference to Pesme and Flammarion and to Berthier and Pillaud-Vivien.

Question. Detail the AI use statement, particularly writing the paper

Common response: C1.

The disclosure names GPT-5.6 Sol and GPT-6.1 Sol and distinguishes the drafting assistance with the initial submission from subsequent human rewriting. It describes literature review, mathematical stress tests and audits, proof refinement and numerical implementation, and GPT-6.1 Sol's role in discussing and finalizing the more general Section 5 results. It specifies that the authors supplied the problem setup, target result forms, preliminary derivations and technical references, reviewed the materials, and take responsibility for the final text, claims, proofs, code and numerical outputs.

I thank the authors for their response. I can not review a major rewrite of the paper in such a short time. Moreover, the excessive use of AI, even under "time constraints", should not be tolerated, as it puts a lot of pressure on the reviewing system. I therefore maintain my grade.

Reviewer_jt2kOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2(−1)
Soundness
3
Read review
Paper summary

The paper studies whether averaging the training trajectory of a diagonal linear network recovers the Lasso path under stochastic gradient flow (SGF) and finite-step SGD. It tracks accumulated stochastic and discretization effects through coordinate-dependent activation clocks. In a solvable coordinate-sampling model, SGF and SGD approach different weighted-Lasso paths. Under weak noise or vanishing step size, together with trajectory-control and path-monotonicity assumptions, the averages approach ordinary Lasso. Synthetic experiments test these path comparisons.

Critical strengths

The paper gives a clean algebraic decomposition of stochastic and discrete deviations from the averaged Lasso path. Expressing the from-start average through a perturbed KKT system clarifies how quadratic variation in SGF and logarithmic remainders in discrete SGD act as coordinatewise penalties. The solvable axis-sampling model demonstrates that moment-matched SGF and finite-step SGD induce different effective Lasso weights. The paper is also transparent about which trajectory bounds are assumed and where general correlated-design guarantees remain open.

Critical weaknesses
  1. The theory excludes an important regime for sparse implicit regularization. The standing assumption is H≻0H\succ0 (Section 3). For empirical least squares, where H=X⊤X/nH=X^\top X/n, this requires a full-column-rank design and d≤nd\leq n. It therefore excludes regression with more variables than observations. This is a restriction relative to Berthier's deterministic averaged-path result, which allows a positive-semidefinite Hessian. The singular two-channel representation in Section 6 does not remove the original assumption on HH. Lasso remains meaningful when d≤nd\leq n, but this exclusion limits the relevance of the stochastic extension to high-dimensional sparse learning.

  2. The general SGD convergence result leaves substantial conditions to be verified. Corollary 6.2 assumes bounded iterates, gradients, and noise moments up to a stopping time, together with a vanishing probability of exiting that region over the growing training horizon. It also requires zero downward variation of the positive and negative parts of the scaled Lasso path. These conditions are not established for general correlated datasets. The paper gives useful special cases and clearly states these limitations, but the result does not yet provide a broadly applicable guarantee for ordinary minibatch SGD. Verifying the trajectory assumptions is a substantive unresolved part of the problem.

  3. The remaining advance over the closest literature is limited in scope. The averaged-path correspondence and logarithmic time scaling come from Berthier. Accumulated stochastic corrections to implicit bias appear in Pesme et al. (2021), and the exact discrete logarithmic corrections already appear in Even et al. (2023). The authors acknowledge these connections. Their addition is to retain these effects in a uniform averaged-path comparison, which does not follow automatically from the earlier results. However, the explicit fixed-noise limits concern a coordinate-sampling model, while the broader convergence result remains conditional. The experiments support the path diagnostics, but do not resolve these restrictions or establish a further consequence for sparse recovery. I find the resulting advance a bit limited.

  4. The significance of recovering the Lasso path is not sufficiently established. The main target is a correspondence between averaged training trajectories and penalized optimization solutions. This is different from establishing sparse recovery, prediction accuracy, or a beneficial implicit bias. The paper makes the correspondence more precise for stochastic and discrete training, but it does not show what the resulting coordinate-dependent distortions imply for statistical performance. The experiments mainly measure agreement with the proposed paths, so they validate the correspondence without establishing its broader significance. Given that the deterministic averaged-path relation and the stochastic and discrete bias corrections have close predecessors, the paper needs a more consequential implication to justify the remaining extension.

Questions for authors
  1. Which conclusions, if any, remain valid when the original Hessian is singular, as in empirical regression with d>nd>n? Please distinguish this case from the singularity introduced by the two-channel representation and clarify the restriction relative to Berthier's result.

  2. For ordinary signed minibatch SGD on correlated data, is there an explicit class for which both the localization assumptions and the scaled-path condition in Corollary 6.2 follow from checkable assumptions on the data? Clarifying what is already proved would help assess the scope of the main SGD result.

5 comments

1. The theory excludes an important regime for sparse implicit regularization

Common response: C3, point 1.

The revised standing model in Section 2 allows H⪰0H\succeq0 with r∈range⁡(H)r\in\operatorname{range}(H), replacing the original requirement H≻0H\succ0 on the data Hessian. For empirical least squares, H=X⊤X/nH=X^\top X/n and r=X⊤y/nr=X^\top y/n satisfy this condition for every fixed XX and yy, including d>nd>n and noninterpolating data. Under the stated stochastic hypotheses, the prediction and Lasso-objective comparisons and the exact recovery criterion apply in this setting.

For the objective comparison, Appendix D.5 first verifies that dual increments lie in range⁡(H)\operatorname{range}(H) before applying the pseudoinverse H†H^\dagger. The constant is γH=(r⊤H†r)1/2\gamma_H=(r^\top H^\dagger r)^{1/2}. The regularity needed for the chosen Lasso path is justified separately using its optimality and monotonicity properties. It is not inferred from a bound on fitted values.

2. The general SGD convergence result leaves substantial conditions to be verified

Common response: C2; C3, points 2 and 3.

The general SGD result in the revised main text is Theorem 5.1. It compares averaged predictions with averaged ISS without the Lasso-path condition. The trajectory assumptions are treated separately in Section 5.2. On a fixed finite dataset, a bound on the coefficient vector βk\beta_k bounds the sampled gradients and their conditional variances. For sufficiently small step size it also gives the required stability margin for each update. Section 5.4 and Proposition D.5 prove these bounds over the whole training horizon for a correlated nonnegative class. The general signed theorem retains localization because it needs the same coefficient bound, with probability tending to one, over the growing horizon. Section 5.2 and Appendix D.2 explain the roles of the assumptions and why the earlier boundedness results do not automatically verify this stronger requirement.

The path condition has a different role. Under the stated stochastic hypotheses, with H⪰0H\succeq0 and r∈range⁡(H)r\in\operatorname{range}(H), uniform Lasso prediction recovery and a uniformly vanishing Lasso objective gap hold exactly when there is a choice of Lasso minimizers βs\beta_s for which both (sβs)+(s\beta_s)_+ and (−sβs)+(-s\beta_s)_+ are nondecreasing, starting from zero. Recovery is on every fixed [s0,S][s_0,S] with 0<s0<S0<s_0<S. The general averaged-ISS comparison allows deletion and sign reversal without this path condition, while retaining localization. See Theorem 5.3.

3. The remaining advance over the closest literature is limited in scope

Common response: C2 and C4.

The revision credits Berthier's deterministic averaged-path theory, Pesme et al.'s stochastic endpoint corrections, Even et al.'s exact discrete logarithmic remainders, the established ISS description and the standard positive-cone criterion. The revised introduction makes the distinction discussed in C4 explicit: our comparisons concern averaged SGF and actual SGD predictions under the stated trajectory hypotheses. C2 summarizes the singular-design extension, exact Lasso criterion, verified correlated class and signed full-batch GD example.

4. The significance of recovering the Lasso path is not sufficiently established

Common response: C4.

We discuss the significance of the averaged-path characterization in C4 and have revised the introduction to make it explicit. Example 5.4 and its full-batch signed-GD realization show persistent prediction and Lasso objective gaps even as initialization and step size vanish. The fixed-noise example also distinguishes SGF and actual SGD penalty paths. Prediction-risk and support-recovery guarantees would require a separate statistical analysis; they do not follow from path agreement alone.

Question 1. Which conclusions remain valid when the original Hessian is singular?

Common response: C3, point 1.

Theorem 5.1 compares averaged SGD fitted values with those of averaged ISS when H⪰0H\succeq0 and r∈range⁡(H)r\in\operatorname{range}(H). Under the respective stochastic hypotheses, the exact Lasso prediction and objective criterion also applies: there must exist a choice of Lasso minimizers whose scaled positive and negative parts are nondecreasing. These conclusions include fixed empirical designs with d>nd>n and do not require a unique coefficient vector. See Theorems 5.1 and 5.3 and Corollary D.4.

Question 2. Is there an explicit class for ordinary signed minibatch SGD on correlated data?

Common response: C3, point 3.

The fully verified class is the balanced nonnegative special case, βi=ui2\beta_i=u_i^2. The construction in Section 5.4 uses dd observations with one nonzero feature each and one observation involving every feature, with responses generated by positive coefficients. It gives H=(1−ρ)I+ρ11⊤H=(1-\rho)I+\rho\mathbf{1}\mathbf{1}^\top for 0<ρ<10<\rho<1. Proposition D.5 and Appendix D.6 verify a fixed invariant box, and the standard positive-cone criterion verifies the required Lasso-path geometry. Both requirements therefore follow from the data.

This proof uses nonnegative coefficients, feature entries and labels. It does not establish the same bound for a signed initialization with two nonzero initial sign channels, so the general signed-minibatch theorem keeps localization as a hypothesis. The full-batch signed example has its own proved trajectory bound and establishes Lasso disagreement in that noiseless setting.

Thank you for your response. However, I still feel that the progress on the problem this paper made and the significance of recovering the Lasso path is not sufficiently enough to justify the paper to be accepted, I would like to maintain my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Common Response

Thank you for all the constructive comments on presentation, generality, assumptions and significance. We summarize the shared changes here and address each reviewer's specific comments in individual responses.

C1. Rewriting and AI use

It seems that a major objection comes from the fact that the reviewers found the exposition too technical and at times misleading with shifting setups, unclear purpose, and inconsistent notations. We sincerely apologize for the confusion caused by our excessive reliance on AI when drafting the initial submission under time constraints. Although this use complied with ICLR policy, it contributed substantially to the confustion here. We have since manually rewritten almost all previously AI-generated prose, improving the paper's structure and clarity.

Specifically, we rewrote the introduction and reorganized the main text to present clean Lasso recovery results first, followed by fixed-noise and fixed-step effects, general averaged dynamics, and the exact condition for Lasso agreement. Supporting estimates and proofs are collected in the appendices. Given the extent of the revision, explanatory rewrites appear in black, while new technical changes are marked in red. We hope the revised presentation addresses the exposition concerns raised by reviewers LG6Q and mAiC, and respectfully ask them to reconsider the paper in light of these changes.

We also changed the title from Logarithmic Activation Clocks in Diagonal Linear Networks: When SGD Does and Does Not Follow Lasso to Averaged SGD and the Lasso Path in Diagonal Linear Networks because the added general results in current section 5 address a different issue relevant to characterizing the Lasso path, beyond "logarithmic activation clocks".

The expanded AI disclosure names GPT-5.6 Sol and GPT-6.1 Sol and distinguishes GPT-5.6 Sol's initial drafting assistance, later human rewrites, and GPT-6.1 Sol's role in revising the general results. It also describes the task-level literature, mathematical and numerical assistance, and the authors' review and responsibility.

C2. Technical additions and extensions

The main technical changes are:

  • Allowing singular designs (jt2k, weakness 1 and question 1; mAiC, weakness 2). The general prediction and Lasso-objective comparisons now allow H⪰0H\succeq0 with r∈range⁡(H)r\in\operatorname{range}(H), including fixed empirical designs with d>nd>n.
  • General averaged dynamics beyond scaled-path monotonicity assumption (jt2k, weaknesses 2 and 3; LG6Q, point 2; mAiC, weakness 2). Averaged SGD and SGF predictions converge to those of averaged inverse-scale-space (ISS) dynamics under the stated trajectory bounds, allowing deletion and sign reversal. Initialization tends to zero, together with the SGD step size or the product of the SGF noise level and its variance bound (Theorem 5.1 and Corollary 5.2).
  • Exact Lasso criterion and a failure example (jt2k, weaknesses 2 and 4; LG6Q, point 2). Under the same stochastic hypotheses, prediction and objective recovery are equivalent to the existence of a Lasso minimizer path with nondecreasing scaled positive and negative parts, including when minimizers are nonunique (Theorem 5.3). The accompanying counterexample proves a persistent prediction and objective gap for signed full-batch GD, including an underdetermined design (Example 5.4 and Appendix D.7).
  • A correlated SGD class with verified assumptions (jt2k, weakness 2 and question 2; mAiC, weakness 2). The revision gives a finite test for the Lasso-path condition and a correlated balanced nonnegative SGD class with both trajectory bounds and Lasso-path geometry proved (Section 5.4; Propositions D.3 and D.5).

C3. Justification of assumptions and generality of results

  1. Design rank. The reviewers noted that the original full-rank assumption excluded regression with more variables than observations. The general prediction and Lasso-objective comparisons now allow H⪰0H\succeq0 with r∈range⁡(H)r\in\operatorname{range}(H), including fixed empirical designs with d>nd>n.

  2. Path monotonicity. The reviewers asked whether the scaled-path monotonicity assumption is essential. Under the stated stochastic hypotheses, Theorem 5.3 shows that Lasso prediction and objective recovery hold exactly when one can choose Lasso minimizers βs\beta_s, at penalty 1/s1/s, whose scaled positive and negative parts (sβs)+(s\beta_s)_+ and (−sβs)+(-s\beta_s)_+ are coordinatewise nondecreasing. The averaged-ISS comparison permits deletion and sign reversal without this condition.

AuthorsOpenReview ↗

Common Response (Part II)

  1. Trajectory control. The reviewers asked when the trajectory bounds follow from the data. On a fixed finite dataset, a bound on the coefficient vector bounds the gradients and their noise variances. We prove trajectory control for the orthogonal example and for a correlated class in the balanced nonnegative model βi=ui2\beta_i=u_i^2. For general signed predictors, the comparison retains localization: the same coefficient bound must hold with probability tending to one throughout the growing training horizon. A bound that depends on initialization or applies only over a fixed time interval does not establish this condition.

  2. Fixed noise and fixed steps. mAiC asked whether Theorem 4.1 extends beyond the dataset where each observation has only one nonzero feature. Proposition A.2 proves a correlated fixed-noise SGF example. The general SGD and SGF comparisons in Section 5 address a different limit, in which initialization and the step size or effective noise vanish. At fixed noise or step size, accumulated noise and finite-update effects can persist in the average. A general correlated-data extension therefore requires global control of these effects along the training trajectory; the local moment matching used to construct SGF does not provide it.

  3. Choice of model. The reviewers questioned the value of studying a simple diagonal network. It is a starting model for understanding how SGD develops sparsity without an explicit sparsity penalty. Keeping the predictor linear and the loss quadratic lets us study how sampling and finite updates change the training trajectory. The introduction and C4 explain this motivation.

C4. Contribution and significance

The reviewers asked what goes beyond prior work and why the path characterization matters. Sections 1 and 6 credit Berthier's averaged-path theory, Pesme et al.'s endpoint geometry, Even et al.'s discrete corrections, the established ISS description and the standard positive-cone criterion. Many of these earlier works focus on endpoint results that describe where training converges, but do not determine the predictors obtained by stopping earlier. Berthier addresses this trajectory question and its Lasso connection in the deterministic flow setup.

We extend that averaged-path analysis to SGF and actual SGD, characterizing averaged predictions and exactly when they agree with Lasso under the stated hypotheses. LG6Q calls the early-stopping/SGD extension a "significant step towards practice"; jt2k highlights the explanation of stochastic and discrete penalties; and mAiC confirms the advance beyond deterministic paths. We appreciate the recognition of our contribution from all reviewers.

Stochastic effects matter because minibatch noise can change the regularization induced by training. Its influence must be controlled over a horizon that grows logarithmically as initialization shrinks. Our SGF results show that fixed noise can produce coordinate-dependent Lasso penalties even on orthogonal data. Conversely, in the same orthogonal example, vanishing noise restores ordinary Lasso without requiring noise times the training horizon to vanish.

The actual SGD result concerns the algorithm used in training. SGF is continuous-time, whereas SGD uses finite random updates; matching the local means and covariances of SGD and forward-Euler SGF factor updates does not determine whether continuous SGF and actual SGD have the same long-time averaged predictor path. Our exact example shows that, even on the same orthogonal dataset, SGF and SGD can converge to different weighted-Lasso paths when the SGF noise level is matched to the SGD step size. A result for SGF therefore does not supply the corresponding SGD result.

More broadly, the paper advances our understanding of how training algorithms induce regularization. Characterizing the averaged SGD path connects sampling, step size and stopping time to the regularization produced during training. For diagonal linear networks, our paper extends deterministic trajectory theory to stochastic and finite-step training, filling an important gap in understanding how optimization and overparameterization interact to favor sparse solutions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.