acceptodds
Under review as a conference paper at ICLR 2027

On the stability of learning

Abstract

Learning algorithms are usually analyzed under the assumption that informative gradients are continuously available. But this is not always true—learning signals in biological neural networks, for example, may be sparse, local, and only intermittently available at any given synapse. In this work, we show that if gradient information is noisy and only intermittently available, learning may be unstable. In particular, even if average learning dynamics converge to the correct optimum, variance in learning may be asymptotically infinite. We explore this point in a number of analytically tractable settings, including single-layer and multi-layer linear networks, and derive explicit conditions under which learning is stable. These conditions complement, but are conceptually distinct from, existing conditions related to the stability of stochastic gradient descent. Our work demonstrates that unbiased credit assignment is not enough for reliable learning; sufficiently frequent and informative feedback is also required, which yields an additional constraint on models of biologically plausible learning.

Primary area:
learning theory
Keywords:
learning dynamics, gradient descent, stability, linear networks, theory

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_tiyoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks and studies an interesting question on the stability of learning process when gradient information is recieved only intermittently and with noise. As a concrete modeling, each coordinate wiw_i receives gradient arrivals from an independent Poisson process of rate ηρ\eta\rho. At each arrival, wiw_i jumps by a gradient step with zero-mean Guassian noise scaled by 1/ρ1/\rho. The scaling 1/ρ1/\rho is introduced to make the same averaged steps for every ρ\rho. Several applications including a toy example, linear regression, nonlinear networks, and deep scalar linear network are considered and discussed when the learning process is stable and unstable. Two strategies including pooling parameters and averaging gradients are provided to improve stability of learning.

Critical strengths
  1. The question is clean and clearly motivated, and the presentation is well organized, with many informative figures provided that help understanding.

  2. The model cleanly separates the roles of arrival frequency (with update sizes compensated by 1/ρ1/\rho), observation noise and continuous regularization. In particular, the paper provides closed-form mean-square stability thresholds for several linear settings, including continuous regularization, pooling and accumulation.

  3. The costs of the mitigation mechanisms, namely bias for pooling and slower convergence for accumulation, are made explicit.

Critical weaknesses
  1. My major concern is at the motivation of scaling gradients by 1/ρ1/\rho at each arrival. While I understand this is for balancing the rare arrivals from Poisson process of rate ηρ\eta\rho, it seems this is also a key reason of instability demonstrated in the paper. As shown for the toy example, learning becomes less stable with larger s/ρs/\rho, which corresponds to small ρ\rho. This seems not surprising as smaller ρ\rho may yield overly large step sizes. On the other extreme case, one can learn without scaling despite the rare gradient arrivals. This would enhance stability, but at the cost of slowing down learning process. So this reveals a trade-off between stability and learning speed, which the authors should discuss and explain explicitly.

  2. Another side concern is on the applicability of the modeling. The current study focuses on specific modeling such as gradient arrivals following Poisson process, zero-mean Guassian noises. I wonder if these requirements are essential to the analysis and if it can be extended to broader and more general settings. For instance, zero-mean noise with bounded variance.

Questions for authors

See the weakness discussion above. Additionally, I would also like to ask the following questions:

  1. Is the compensation by 1/ρ1/\rho essential? What happens with a fixed gain per arrival?

  2. Can you provide some explanation on the two-layer network example? In particular, how was ρc\rho_c computed, and why do the runs at 1.2ρc1.2\rho_c and 1.3ρc1.3\rho_c appear unstable, with smoothly decreasing loss around 2ρc2\rho_c?

  3. A minor question: should the coefficient of cic_i be 2γ+2si−si2/ρ2\gamma+2s_i-s_i^2/\rho instead in Eq. (12)?

4 comments
AuthorsOpenReview ↗

Rebuttal (part 1)

Thank you for your close reading of our paper and for your helpful comments. We hope to convincingly address your concerns below, but please let us know if we fall short of doing so!

On gradient scaling

My major concern is at the motivation of scaling gradients by 1/ρ1/\rho at each arrival. While I understand this is for balancing the rare arrivals from Poisson process of rate ηρ\eta \rho, it seems this is also a key reason of instability demonstrated in the paper. As shown for the toy example, learning becomes less stable with larger s/ρs/\rho, which corresponds to small ρ\rho. This seems not surprising as smaller ρ\rho may yield overly large step sizes. On the other extreme case, one can learn without scaling despite the rare gradient arrivals. This would enhance stability, but at the cost of slowing down learning process. So this reveals a trade-off between stability and learning speed, which the authors should discuss and explain explicitly.

Is the compensation by 1/ρ1/\rho essential? What happens with a fixed gain per arrival?

This is a very astute observation. Thinking hard about this comment has made us decide to reframe the paper slightly: indeed, the 1/ρ1/\rho compensation is not essential, and we will address the more general case explicitly in the revised version of the paper.

Your intuition is exactly right, but for completeness' sake, we will recount it and add our own thoughts. First, as you say, we introduced the 1/ρ1/\rho compensation for a very specific reason: we wanted to hold average dynamics fixed, and we wanted the steady state solution of average dynamics to correspond to the minimizer of our objective (e.g., Eq. 8). We thought that this would allow us to isolate what we consider the 'interesting' effect of Poisson gradient arrivals, which is very different behavior at the level of covariance dynamics.

However, you and Reviewer i2MH make a good point. Why would the system ever be in this regime if it yields instability? Doesn't the 1/ρ1/\rho compensation unfairly constrain the system? After some thought, we mostly agree. For the reasons just described, we still think the fixed mean dynamics case is interesting and worth discussing, but the more general case should also be discussed.

Suppose we do not include the 1/ρ1/\rho compensation. Exactly as you observe, there is a 'no free lunch' issue. While one can avoid instability (i.e., exploding variance), two other problems appear: the convergence of mean dynamics is generally slower, and its steady state solution is generally biased. The more the coefficient of the Poisson arrival term differs from 1/ρ1/\rho, the more this is true. Hence, we absolutely agree that a better 'story' may be the following. Intermittent gradient arrivals present a problem for learning. Rare enough arrivals either produce instability OR slow and biased learning.

We think this story is still quite interesting. We also don't think the layout of the paper needs to change; we just ought to address the tradeoff between speed and stability in the respective sections. The original version of the paper already discussed this in Sec. 6 (we note that the bias is spatial-frequency-dependent) and Sec. 7 (we note that gradient-averaging can really slow down learning).

There are two kinds of plots we can add in a few places that could help illustrate the speed-accuracy tradeoff. We can plot the maximum learning speed versus some parameter of interest (e.g., the Poisson arrival coefficient, or the regularization coefficient γ\gamma) and the minimum bias versus some parameter of interest (kind of like Fig. 5c).

AuthorsOpenReview ↗

Rebuttal (part 2)

(continued)

On generalizing the gradient arrival noise model

Another side concern is on the applicability of the modeling. The current study focuses on specific modeling such as gradient arrivals following Poisson process, zero-mean Guassian noises. I wonder if these requirements are essential to the analysis and if it can be extended to broader and more general settings. For instance, zero-mean noise with bounded variance.

This is a great question and we thank you for bringing this up. Indeed, the analysis can be made much more general. Looking over our derivations, the only aspects of the gradient noise term ξ\xi we use are its first two moments. We will rewrite our analysis to make this more explicit.

Meanwhile, the Poisson assumption can be generalized to other renewal processes. For example, the stability condition for our simplest model becomes (assuming for simplicity the 1/ρ1/\rho compensation) (1−sρ)2E[e−2ηγτ]<1\left( 1 - \frac{s}{\rho} \right)^2 \mathbb{E}[ e^{- 2 \eta \gamma \tau} ] < 1 where the expectation is over the associated 'waiting time' distribution, which is exponential in the Poisson case. Since, in particular, it is exponential with rate ηρ\eta \rho, this becomes (1−sρ)2ρρ+2γ<1\left( 1 - \frac{s}{\rho} \right)^2 \frac{\rho}{\rho + 2\gamma} < 1 which, after some algebra, is equivalent to the original paper's Eq. 13. We will discuss this generalization explicitly in a new appendix and consider one or two other waiting time distributions for illustration.

Other concerns

A minor question: should the coefficient of cic_i be 2γ+2si−si2/ρ2 \gamma + 2 s_i - s_i^2/\rho instead in Eq. (12)?

Good catch. Sorry about the typo; it is fixed now.

Can you provide some explanation on the two-layer network example? In particular, how was ρc\rho_c computed, and why do the runs at 1.2ρc1.2 \rho_c and 1.3ρc1.3 \rho_c appear unstable, with smoothly decreasing loss around 2ρc2 \rho_c?

Sorry for the brevity here, we will add a bit more explanation to the revised paper.

First, we consider a student network initialized near a teacher network with the same architecture. By construction, the system is initialized near a fixed point, so we can approximate the system as linear around that fixed point by numerically computing its Hessian.

Then we use the covariance dynamics discussed in Sec. 3 (Eq. 11) to determine ρc\rho_c. The important point not really discussed in the original text (for space reasons...sorry about this) is that, since the Hessian HH might not be diagonal or circulant, we can't use Eq. 13 or Eq. 16; we have to determine ρc\rho_c in a more general way. We have to numerically check when the right-hand side of Eq. 11 becomes unstable. Since Eq. 11 is linear in CC (in particular, one has a Lyapunov-like equation), this amounts to determining the ρ\rho value at which the matrix that determines the right-hand side develops an eigenvalue above zero.

Regarding the weird-looking behavior of 1.2ρc1.2 \rho_c and 1.3ρc1.3 \rho_c, we found that the precise shape of those curves varied from seed to seed. Generally we found that values above but near the critical ρ\rho value could have large excursions above the dashed line, but would never blow up to infinity. This was also affected a bit by the time step we used; we think the effect is a little smaller if we use a smaller Δt\Delta t. This is a subtle but important point, so we will also add some discussion of it to the revised version (although probably this will appear in an appendix).

Thank you for the detailed reply. Most of my questions are addressed, and I appreciate the candid discussion.

Gradient scaling. I agree with the proposed reframing in terms of the trade-off between instability and slow, biased learning. This also resolves my question about a fixed gain per arrival.

Generality. The clarification that only the first two moments of the noise are used addresses my concern about the noise model. The renewal-process condition is correct and reduces to Eq. (13) in the Poisson case.

Two-layer network. The explanation of how ρc\rho_c was computed is helpful. Since, as you explain, the runs at 1.2ρc1.2\rho_c and 1.3ρc1.3\rho_c do not diverge, they are better described as stable with large fluctuations. Such fluctuations are expected near the threshold, because the linear theory predicts a stationary variance of order 1/(ρ−ρc)1/(\rho-\rho_c) (Eq. (14) and Figure 2(c)). An event-driven simulation would remove the dependence on Δt\Delta t, and results over several seeds would make Figure 3(b) more informative.

As these changes are substantial and a revised manuscript seems not uploaded yet, I will keep my current score. I expect the revised version to be considerably stronger.

Thank you for engaging us. We are working on revising the paper now and should post an updated version before the end of the day.

Reviewer_i2MHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes models of learning dynamics in the brain through continuous processes with jumps. The jumps correspond to random coordinate gradient steps on a loss. The paper discusses the stability of learning depending on the stepsize, the rate of stochastic gradient updates, regularization, parameter pooling and time-averaging.

Critical strengths

This paper is easy to follow and seems correct. However, the concerns I have below dominate my final decision.

Critical weaknesses

Modeling issue. I have a central concern about the model, that can be easily discussed from Section 2. ρ\rho denotes the arrival rate of coordinate gradients, and the gradients updates are applied with a learning rate 1/ρ1/\rho to correspond in order to a flow of rate 11 on average. However, any reasonable learning system would in fact tune the learning rate to stay stable. It would not increase the learning rate if the arrival rate ρ\rho decreases; in fact, it would be more natural to keep the learning rate independent of the arrival rate ρ\rho.

This issue is present in all the subsequent sections.

In fact, the main statement of the paper (ll.12-14: 'if gradient information is noisy and only intermittently available, learning may be unstable') relies on this poor scaling of the learning rate. Instead, I think that with intermittent gradient information, the learning rate should be kept fixed and learning would be possible (but slower). This main message of the paper would be totally different.

Insufficient related work and lack of novelty. As a consequence of the above, the analyses in the paper are in fact similar to classical analyses of the stability of (stochastic) gradient descent and of the choice of the stepsize. Slight variations are due to the mixed continuous / discrete setting, but are not especially surprising in my perspective.

The authors seem unaware of the large body of work on this topic. In the conclusion, the citations for 'standard analyses of SGD' are from 2018 and 2022. I suggest taking a look at the recent book 'learning theory from first principles' by Francis Bach (for instance) and the references therein to discover much earlier and seminal references.

In particular, the distinction that the authors make between 'noise due to using stochastic gradients' and 'sparse gradient arrivals' is usually called 'additive' and 'multiplicative' noise respectively. Stochastic gradients can include multiplicative noise; the fact that there is only additive noise is due only to the specific model of the authors.

This is only a representative issue of the superficial related work. The citations list in l.81-82 also is very arbitrary.

Questions for authors

Minor suggestions:

  • l.97: clarify that rr is the combination of multiple independent Poisson processes.
  • In Sec. 2, the dynamics are not written clearly. The authors could use the framework of 'piecewise deterministic Poisson processes' - see Davis. Piecewise-deterministic markov processes: a general class of non-diffusion stochastic models. Journal of the Royal Statistical Society: Series B (Methodological), 46(3):353–376, 1984.
5 comments
AuthorsOpenReview ↗

Rebuttal (part 1)

Thank you for your close reading of our paper and for your helpful comments---you've clearly thought deeply about our paper's technical content, which makes us excited to see if we can convince you there's something interesting here. We will do our best to respond to the core concerns (i.e., the 1/ρ1/\rho issue, insufficient discussion of related work, and perceived lack of novelty) below. If we are unconvincing, please let us know and we will try to do better! Either way, we are exceedingly grateful for your time and energy.

On gradient scaling

Modeling issue. I have a central concern about the model, that can be easily discussed from Section 2. ρ\rho denotes the arrival rate of coordinate gradients, and the gradients updates are applied with a learning rate 1/ρ1/\rho to correspond in order to a flow of rate 11 on average. However, any reasonable learning system would in fact tune the learning rate to stay stable. It would not increase the learning rate if the arrival rate ρ\rho decreases; in fact, it would be more natural to keep the learning rate independent of the arrival rate ρ\rho.

This issue is present in all the subsequent sections.

In fact, the main statement of the paper (ll.12-14: 'if gradient information is noisy and only intermittently available, learning may be unstable') relies on this poor scaling of the learning rate. Instead, I think that with intermittent gradient information, the learning rate should be kept fixed and learning would be possible (but slower). This main message of the paper would be totally different.

This is a very astute observation. Thinking hard about this comment has made us decide to reframe the paper slightly: indeed, the 1/ρ1/\rho compensation is not essential, and we will address the more general case explicitly in the revised version of the paper.

Your intuition is exactly right, but for completeness' sake, we will recount it and add our own thoughts. First, we introduced the 1/ρ1/\rho compensation for a very specific reason: we wanted to hold average dynamics fixed, and we wanted the steady state solution of average dynamics to correspond to the minimizer of our objective (e.g., Eq. 8). We thought that this would allow us to isolate what we consider the 'interesting' effect of Poisson gradient arrivals, which is very different behavior at the level of covariance dynamics.

However, you and Reviewer tiyo make a good point. Why would the system ever be in this regime if it yields instability? Doesn't the 1/ρ1/\rho compensation unfairly constrain the system? After some thought, we mostly agree. For the reasons just described, we still think the fixed mean dynamics case is interesting and worth discussing, but the more general case should also be discussed.

Suppose we do not include the 1/ρ1/\rho compensation. Exactly as you observe, there is a 'no free lunch' issue. While one can avoid instability (i.e., exploding variance), two other problems appear: the convergence of mean dynamics is generally slower, and its steady state solution is generally biased. The more the coefficient of the Poisson arrival term differs from 1/ρ1/\rho, the more this is true. Hence, we absolutely agree that a better 'story' may be the following. Intermittent gradient arrivals present a problem for learning. Rare enough arrivals either produce instability OR slow and biased learning.

We think this story is still quite interesting. We also don't think the layout of the paper needs to change; we just ought to address the tradeoff between speed and stability in the respective sections. The original version of the paper already discussed this in Sec. 6 (we note that the bias is spatial-frequency-dependent) and Sec. 7 (we note that gradient-averaging can really slow down learning).

There are two kinds of plots we can add in a few places that could help illustrate the speed-accuracy tradeoff. We can plot the maximum learning speed versus some parameter of interest (e.g., the Poisson arrival coefficient, or the regularization coefficient γ\gamma) and the minimum bias versus some parameter of interest (kind of like Fig. 5c).

AuthorsOpenReview ↗

Rebuttal (part 2)

(continued)

On the novelty of our model

As a consequence of the above, the analyses in the paper are in fact similar to classical analyses of the stability of (stochastic) gradient descent and of the choice of the stepsize. Slight variations are due to the mixed continuous / discrete setting, but are not especially surprising in my perspective.

Is our model novel? Are the 'twists' one obtains by considering, e.g., the mixed continuous/discrete setting sufficiently interesting? Although this is subjective, we would like to argue that the answer is yes.

We apologize for not better explaining the distinctive features of our model in the original paper. Clearly it is important to convince the reader that there is something new here---otherwise, as you and reviewer tiyo point out, it feels like the paper makes a big deal about existing learning theory results.

In brief, we see the distinctive features of our model as follows.

  • First, the dual discrete-continuous nature of the problem is something that we think is theoretically interesting. Intermittent gradient arrivals occur against a background of other processes (e.g., weight decay) happening in continuous time.
  • Second, gradient arrivals are asynchronous at different synapses. This makes it harder to view our model as just finite-step GD disguised (the timeline is different for each synapse) and it also generally destroys our ability to simplify it by rotating into a basis where the covariance is diagonal. Incidentally, this technical issue is why we highlight the special cases of diagonal and circulant covariance matrices.
  • Third, it is easy to connect to known learning-associated neurobiology. Although it may be possible to transform the problem into a slightly different-looking one, in its current guise it is (arguably) easier to view as a simple model of neurobiological synapse updates. The extensions of the model we consider in Sec. 6 and Sec. 7 also have nice biological interpretations, which we consider a virtue.

It is pretty difficult to claim that nowhere in previous learning theory was a model like this considered. Indeed, you have encouraged us to look through the literature more systematically, and there are results on mixed discrete-continuous learning, asynchronous learning, and so on. We think the primary theoretical merit of our contribution is combining these things in a model which strikes a good balance between simplicity/tractability on the one hand, and interesting phenomenology on the other. We also think the connection to neurobiology is exciting, but we acknowledge that not all learning theory readers may be interested in this.

As a side note, we absolutely agree that the instability of discrete-time gradient descent (GD) for sufficiently large step sizes is well understood. We apologize for not emphasizing this enough in the original version. In Sec. 2, especially in the "useful intuition" subsection, we thought starting with this well-known feature of GD would help situate the reader before we present the more complex (dual discrete-continuous, asynchronous) model we care about. For brevity, we didn't really communicate that this is well-known and not meant to be a contribution. We will fix this in the revised version.

AuthorsOpenReview ↗

Rebuttal (part 3)

(continued)

A comment about the relationship between our model and finite-step GD

Building on our previous comment regarding the perceived lack of novelty, we think it could be helpful to clarify what's different between finite-step GD and our model with two illustrative examples.

First, we'd like to point out that the effect of regularization on stability differs depending on whether it occurs on the discrete or continuous timeline. For simplicity, assume that there is only a single weight that needs to be updated, and let ss denote the associated input variance.

Consider the difference between including the regularization term (whose coefficient is γ\gamma) in the objective (so that its effect is only felt intermittently) and having it be a separate continuous process. Recall from the paper that in the latter case, the stability condition reads (Eq. 13) ρ>s22(s+γ)\rho > \frac{s^2}{2 (s + \gamma)} where ρ\rho is what we call the (non-dimensionalized) arrival rate. In the former case, we end up with a different condition ρ>s+γ2\rho > \frac{s + \gamma}{2} In other words, for finite-step GD, regularization decreases stability, while in our joint discrete-continuous setting it increases stability. (Also note that, if we set things up this way, this is true despite the fact that both cases have the same average dynamics.)

Similar claims are true if we try to absorb other model extensions (see Sec. 6 and Sec. 7) into finite GD steps: the qualitative conclusions may be a bit different. This is especially consequential if we're trying to link the model to neuroscience (although we acknowledge not all learning theory readers may be interested in this).

Second, let us return to the general case of many weights that need to be updated. We would like to point out that the asynchronous nature of updates twists the problem in an interesting way. The fact that different synapses are (randomly) updated at different times destroys some of the basic structure of the covariance dynamics. If it were not there, we could easily rotate into a space where covariance dynamics become diagonal; with this feature, we generally cannot do this. At least in our view, what this implies for the stability condition is pretty dramatic. In one case, stability depends on the maximum eigenvalue of SS (Eq. 13), and in another it depends on the average over eigenvalues of SS (Eq. 16). In general, the critical arrival rate depends on an expression somewhere between those two extremes (see our answer to Reviewer jDtN for a related new result).

The asynchronous nature of updates also makes it harder to view the model as just a 'change in timeline' relative to finite-step GD. If one really wanted to reparameterize it in that way, it would correspond to something more like random coordinate descent, with one choosing different parameters to update at each step. But we think viewing things this way both obscures some of the fundamental structure and makes it harder to link the model to neurobiology.

Reviewer_jDtNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes gradient descent when discrete gradient updates occur on a Poisson process. Under scaling assumptions that keep the mean update per time fixed, learning is found to be unstable when the Poisson rate is too slow (few, large updates). Critical rate values are derived for simple models and numerical experiments are reported for multilayer networks.

Critical strengths

Introduces a new perspective on the complications of biological learning

Critical weaknesses

This appears to be an old problem in disguise. It is well understood how discrete-time gradient descent becomes unstable when the learning rate is too large (including the intermediate regime of damped oscillation). Nesting this discrete process in a Poisson process just locates the update events on an external timeline.

Questions for authors

Reparameterize your setup with a learning rate α=ρ−1\alpha = \rho^{-1} and a rate constant λ=ηρ\lambda = \eta\rho. Then we have standard discrete-time gradient descent with learning rate α\alpha and timescale λ−1\lambda^{-1}. Furthermore λ\lambda (and hence η\eta) is spurious because it just scales tt, which is not used. This is the well-studied problem of stability of discrete-time gradient descent as a function of the learning rate (α\alpha) and the shape of the loss landscape (∂J∂w\frac{\partial J}{\partial w} and γ\gamma). Is there something else to the setup that I’m missing?

The main counterargument I see to my previous question is that your updates are asynchronous. Does anything novel result from this assumption? For example, the diagonal and circulant cases (sec 3.1-3.2) are nearly identical up to rotation. The only aspect of the problem that changes under rotation is the update asynchrony (i.e., this is the only part tied to the coordinates), and the results end up different: a max bound (over eigenvalues) in eq 13 and a mean bound in eq 16.

Why do you need the circulant assumption in sec 3.2? Any covariance matrix can be diagonalized.

Is the gradient smoothing method in section 7 different (under the mapping I give above) from momentum?

Sec 8 points out the interplay between discrete and continuous dynamics (gradients and regularization). Under my mapping, the regularization effect is random (it turns out to have a truncated power-law distribution). Does this entail any interesting differences from the standard setup?

7 comments
AuthorsOpenReview ↗

Rebuttal (part 1)

Thank you for your close reading of our paper and for your helpful comments---you've clearly thought deeply about our paper's technical content, which makes us excited to see if we can convince you there's something interesting here. We will do our best to respond to the core concern (i.e., that this is just existing work "in disguise") below. If we are unconvincing, please let us know and we will try to do better! Either way, we are exceedingly grateful for your time and energy.

Is this just a well-known model/phenomenon in disguise?

This appears to be an old problem in disguise.

This is a great question, and we should have better explained the distinctive features of our model in the original paper. Clearly it is important to convince the reader that there is something new here---otherwise, as you and reviewer i2MH point out, it feels like the paper makes a big deal about existing learning theory results.

In brief, our model is not just a well-known one in disguise. We see its distinctive features as follows.

  • First, the dual discrete-continuous nature of the problem is something that we think is theoretically interesting. Intermittent gradient arrivals occur against a background of other processes (e.g., weight decay) happening in continuous time.
  • Second, gradient arrivals are asynchronous at different synapses. This makes it harder to view our model as just finite-step GD disguised (the timeline is different for each synapse) and it also generally destroys our ability to simplify it by rotating into a basis where the covariance is diagonal. Incidentally, this technical issue is why we highlight the special cases of diagonal and circulant covariance matrices.
  • Third, it is easy to connect to known learning-associated neurobiology. Although it may be possible to transform the problem into a slightly different-looking one, in its current guise it is (arguably) easier to view as a simple model of neurobiological synapse updates. The extensions of the model we consider in Sec. 6 and Sec. 7 also have nice biological interpretations, which we consider a virtue.

It is well understood how discrete-time gradient descent becomes unstable when the learning rate is too large (including the intermediate regime of damped oscillation).

We absolutely agree that this feature of discrete-time gradient descent (GD) is well understood. We apologize for not emphasizing this enough in the original version. In Sec. 2, especially in the "useful intuition" subsection, we thought starting with this well-known feature of GD would help situate the reader before we present the more complex (dual discrete-continuous, asynchronous) model we care about. For brevity, we didn't really communicate that this is well-known and not meant to be a contribution. We will fix this in the revised version.

AuthorsOpenReview ↗

Rebuttal (part 2)

(continued)

Reparameterize your setup with a learning rate α=ρ−1\alpha = \rho^{-1} and a rate constant λ=ηρ\lambda = \eta \rho. Then we have standard discrete-time gradient descent with learning rate α\alpha and timescale λ−1\lambda^{-1}. Furthermore λ\lambda (and hence η\eta) is spurious because it just scales tt, which is not used. This is the well-studied problem of stability of discrete-time gradient descent as a function of the learning rate (α\alpha) and the shape of the loss landscape (∂J∂w\frac{\partial J}{\partial w} and γ\gamma). Is there something else to the setup that I’m missing?

Sec 8 points out the interplay between discrete and continuous dynamics (gradients and regularization). Under my mapping, the regularization effect is random (it turns out to have a truncated power-law distribution). Does this entail any interesting differences from the standard setup?

Nesting this discrete process in a Poisson process just locates the update events on an external timeline.

This is a very interesting reparameterization. We think that it doesn't quite work given the asynchrony assumption, except in the special case of a single weight that needs to be updated. But even in this case, there is an interesting qualitative difference between finite-step GD and the model we consider.

Consider the difference between including the regularization term (whose coefficient is γ\gamma) in the objective (so that its effect is only felt intermittently) and having it be a separate continuous process. Recall from the paper that in the latter case, assuming just one weight, the stability condition reads (Eq. 13) ρ>s22(s+γ)\rho > \frac{s^2}{2 (s + \gamma)} where ss is the associated input variance and ρ\rho is what we call the (non-dimensionalized) arrival rate. In the former case, we end up with a different condition ρ>s+γ2\rho > \frac{s + \gamma}{2} In other words, for finite-step GD, regularization decreases stability, while in our joint discrete-continuous setting it increases stability. (Also note that, if we set things up this way, this is true despite the fact that both cases have the same average dynamics.)

Similar claims are true if we try to absorb other model extensions (see Sec. 6 and Sec. 7) into finite GD steps: the qualitative conclusions may be a bit different. This is especially consequential if we're trying to link the model to neuroscience (although we acknowledge not all learning theory readers may be interested in this).

The main counterargument I see to my previous question is that your updates are asynchronous. Does anything novel result from this assumption? For example, the diagonal and circulant cases (sec 3.1-3.2) are nearly identical up to rotation. The only aspect of the problem that changes under rotation is the update asynchrony (i.e., this is the only part tied to the coordinates), and the results end up different: a max bound (over eigenvalues) in eq 13 and a mean bound in eq 16.

We completely agree here---we think this is a critical ingredient of what makes our model interesting. We were motivated by the biological fact that real synapses are unlikely to all receive updates at exactly the same time.

Indeed, the asynchrony is what destroys some of the basic structure of the covariance dynamics. If it were not there, we could easily rotate into a space where covariance dynamics become diagonal; with this feature, we generally cannot do this. At least in our view, what this implies for the stability condition is pretty dramatic. Depending on the maximum eigenvalue versus an average over eigenvalues is quite different. Of course, this is subjective.

It also, as we said above, makes it harder to view the model as just a 'change in timeline' relative to finite-step GD. If one really wanted to reparameterize it in that way, it would correspond to something more like random coordinate descent, with one choosing different parameters to update at each step. But we think viewing things this way both obscures some of the fundamental structure and makes it harder to link the model to neurobiology.

AuthorsOpenReview ↗

Rebuttal (part 3)

(continued)

Other concerns

Why do you need the circulant assumption in sec 3.2? Any covariance matrix can be diagonalized.

This is a great point; we are sorry for being a bit too brief in the original paper. Indeed, any covariance matrix can be diagonalized, and covariance dynamics (e.g., Eq. 11) are always linear. We need these extra assumptions only to derive simple closed-form solutions for the critical arrival rate ρc\rho_c.

The more general situation goes as follows. Define the symmetric and positive definite matrix Kij:=∫0∞([Se−(S+γI)t]ij)2 dt.K_{i j} := \int_0^{\infty} \left( \left[ S e^{- (S + \gamma I) t} \right]_{i j} \right)^2 \ dt . Through some clever algebra, one can show that the stability of covariance dynamics (e.g., Eq. 11) is determined by the maximum eigenvalue of KK. Explicitly, ρc=λ′(K)\rho_c = \lambda'(K) where λ′(K)\lambda'(K) denotes the aforementioned maximum eigenvalue. As one can see from quick computations, this reduces to our maximum eigenvalue (Eq. 13) and average eigenvalue (Eq. 16) expressions in the cases that SS is diagonal and circulant. But in general, we can't go further because the maximum eigenvalue of KK is (analytically) hard to compute.

But, in thinking about your question, we did notice one more interesting thing. We noticed that, although the maximum eigenvalue is hard to compute in general, we can bound the critical arrival rate above and below by expressions that look like Eq. 13 and Eq. 16. In particular, 1N∑qsq22(sq+γ)≤λ′(K)≤max⁡qsq22(sq+γ)\frac{1}{N} \sum_q \frac{s_q^2}{2 (s_q + \gamma)} \leq \lambda'(K) \leq \max_q \frac{s_q^2}{2 (s_q + \gamma)} This is a cool result because it shows that the diagonal and circulant cases are not as arbitrary as one might initially think. The diagonal case constitutes a sort of 'worst case' scenario, while the circulant case constitutes a sort of 'best case' scenario. We will add the details of this more general argument, and discussion of the general SS case, to the revised paper. Thanks for prompting us; this was a neat thing to figure out.

Is the gradient smoothing method in section 7 different (under the mapping I give above) from momentum?

It's pretty closely related---basically, the average dynamics look just like GD with momentum. This is something we should have addressed explicitly in the original version, although we chose not to due to limited space. It was definitely part of our motivation for including that section. The thinking went, if something like momentum were included, how does that change the basic stability picture?

The important twist is that the phenomenology of our gradient-averaging model is a bit different from finite-step GD with momentum. One gets different results depending on whether the momentum parts happen in continuous time, versus at the same time as the intermittent gradient updates. (This is essentially the same argument as the one we made above, when we discussed including or not including regularization in the discrete updates.) This is an interesting point which we will discuss in the revised version.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Revised version of paper now posted

Thank you again to all of the reviewers. In our experience, it is rare that all reviewers deeply engage with the technical and conceptual content of a paper; we are thankful that this was the case here, and we think that your comments and suggestions have helped improve the paper tremendously.

Before we describe the changes we have made, we would like to make a few high-level comments regarding the discussion so far.

Perceived strengths and weaknesses

While all initial scores were quite low, all reviewers seemed to agree on a few things: namely, that the paper is clear, mathematically correct, and well written. The issues that led to low scores seemed instead to be about framing, novelty, and significance. For example:

  • Reviewer i2MH felt that there was insufficient discussion of related work.
  • Both reviewers jDtN and i2MH felt that the relationship between standard (finite step size) gradient descent and the paper's model was unclear. This is important for evaluating the paper's novelty, since if the model is merely a reparameterization of (for example) standard gradient descent, then the results become uninteresting. In more colorful terms: one could worry that this model is merely a more familiar one 'in disguise'.
  • Reviewers i2MH and tiyo were concerned that the paper's main claim regarding instability may be an artifact of a specific parameterization choice---i.e., the choice to keep mean dynamics fixed while allowing the variance in dynamics to increase.
  • Reviewer tiyo wondered to what extent our results can be generalized. (Reviewer jDtN had similar questions regarding our covariance analysis.)

We think we have addressed all of these issues, and a few others, in the course of our discussion and in the revised version of the paper. To elaborate:

On related work

We significantly rewrote parts of the Introduction and Discussion to better situate our work in the context of other work on (for example) the convergence and stability of gradient descent. We also tried to explicitly point out where we are (in our view) doing something interesting and/or different from what is typically done. We think this recontextualization of our work helps make its novelty clearer.

On our parameterization

This required some thought, but we think we ultimately found an interesting solution (that fortunately didn't require major changes to the paper). It was pointed out that, if the gain is allowed to be independent of the gradient arrival rate, one can avoid instability by making the gain smaller when gradient arrivals become rare. This is true, but only at the cost of slower learning and increased bias, as pointed out by Reviewer tiyo.

Is instability just an 'artifact'? We think that the answer is no, for the following reason. Although instability can always be avoided by making learning slower and more biased, this may not be acceptable if the system has a target performance level. Hence, one can ask the question like this: given a target performance level (that is, a target learning speed and bias level), what is the minimum gradient arrival frequency for which learning is stable?

This is only subtly different from what we had before. What we do is allow a general gradient gain α\alpha which is not necessarily equal to 1/ρ1/\rho. Then, since all mean dynamics quantities (including learning speed and the bias associated with the fixed point) depend on α\alpha and ρ\rho only through the product β:=αρ\beta := \alpha \rho, we characterize stability assuming β\beta is held fixed. In this language, the original version of the paper always assumed β=1\beta = 1; in the updated version, we relax this requirement and allow arbitrary β>0\beta > 0. The point is that, if you don't want to go under a certain performance level, the stability conditions become relevant.

This change required us to modify a lot of formulas to include β\beta, but not really to change the structure of the paper. Also, we now explicitly discuss the speed and accuracy versus stability tradeoff in the Sec. 3.

On generality

We showed that several assumptions can be relaxed. The new paper does a stability analysis assuming a general SS (rather than only diagonal or circulant SS), which responds directly to a concern of jDtN. It also better motivates the diagonal and circulant cases as 'extreme' cases that saturate certain upper and lower bounds.

In a discussion with tiyo, we explained that our noise models can be substantially generalized (ξ\xi from Gaussian to any well-behaved distribution with the same first and second moments; Poisson -> arbitrary renewal process).

AuthorsOpenReview ↗

A final reassurance

And one last reassurance:

The changes are mostly localized

A general concern during the review process is this: did the paper change too much? If it did, the conventional wisdom is that it may be better to start the review process over.

We think that this concern doesn't apply here. The changes are mostly about framing rather than content. The biggest changes are the (requested) updates to the Introduction and Discussion. All of the figures and most sections are the same as before. The main content change (explicitly requested by reviewers i2MH and tiyo) is that we changed the gradient gain from 1/ρ1/\rho to α\alpha throughout, which made a lot of formulas more general.

Hence, we humbly ask the reviewers to evaluate the revision with an open mind. We think we made the minimal changes required to respond to all of your concerns, and that it should be easy for you to evaluate whether this claim is true.

We are sorry this took so long to deliver. All we can say is that we did our best!

AuthorsOpenReview ↗

Postscript

We would like to add that the updated PDF isn't quite what we hoped to finish; we planned a few additional short appendices. One will discuss the speed and accuracy vs stability tradeoff in more detail (to respond to reviewer tiyo), another will elaborate on the two-layer network experiment's details (also to respond to tiyo), and the final new appendix will provide some details regarding the generalization from Poisson processes to renewal processes.

We hope you agree that all of these would be easy to incorporate if there were more time, and hope you don't count this against us.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.