- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks and studies an interesting question on the stability of learning process when gradient information is recieved only intermittently and with noise. As a concrete modeling, each coordinate receives gradient arrivals from an independent Poisson process of rate . At each arrival, jumps by a gradient step with zero-mean Guassian noise scaled by . The scaling is introduced to make the same averaged steps for every . Several applications including a toy example, linear regression, nonlinear networks, and deep scalar linear network are considered and discussed when the learning process is stable and unstable. Two strategies including pooling parameters and averaging gradients are provided to improve stability of learning.
Critical strengths
-
The question is clean and clearly motivated, and the presentation is well organized, with many informative figures provided that help understanding.
-
The model cleanly separates the roles of arrival frequency (with update sizes compensated by ), observation noise and continuous regularization. In particular, the paper provides closed-form mean-square stability thresholds for several linear settings, including continuous regularization, pooling and accumulation.
-
The costs of the mitigation mechanisms, namely bias for pooling and slower convergence for accumulation, are made explicit.
Critical weaknesses
-
My major concern is at the motivation of scaling gradients by at each arrival. While I understand this is for balancing the rare arrivals from Poisson process of rate , it seems this is also a key reason of instability demonstrated in the paper. As shown for the toy example, learning becomes less stable with larger , which corresponds to small . This seems not surprising as smaller may yield overly large step sizes. On the other extreme case, one can learn without scaling despite the rare gradient arrivals. This would enhance stability, but at the cost of slowing down learning process. So this reveals a trade-off between stability and learning speed, which the authors should discuss and explain explicitly.
-
Another side concern is on the applicability of the modeling. The current study focuses on specific modeling such as gradient arrivals following Poisson process, zero-mean Guassian noises. I wonder if these requirements are essential to the analysis and if it can be extended to broader and more general settings. For instance, zero-mean noise with bounded variance.
Questions for authors
See the weakness discussion above. Additionally, I would also like to ask the following questions:
-
Is the compensation by essential? What happens with a fixed gain per arrival?
-
Can you provide some explanation on the two-layer network example? In particular, how was computed, and why do the runs at and appear unstable, with smoothly decreasing loss around ?
-
A minor question: should the coefficient of be instead in Eq. (12)?
4 comments
Rebuttal (part 2)
(continued)
On generalizing the gradient arrival noise model
Another side concern is on the applicability of the modeling. The current study focuses on specific modeling such as gradient arrivals following Poisson process, zero-mean Guassian noises. I wonder if these requirements are essential to the analysis and if it can be extended to broader and more general settings. For instance, zero-mean noise with bounded variance.
This is a great question and we thank you for bringing this up. Indeed, the analysis can be made much more general. Looking over our derivations, the only aspects of the gradient noise term we use are its first two moments. We will rewrite our analysis to make this more explicit.
Meanwhile, the Poisson assumption can be generalized to other renewal processes. For example, the stability condition for our simplest model becomes (assuming for simplicity the compensation) where the expectation is over the associated 'waiting time' distribution, which is exponential in the Poisson case. Since, in particular, it is exponential with rate , this becomes which, after some algebra, is equivalent to the original paper's Eq. 13. We will discuss this generalization explicitly in a new appendix and consider one or two other waiting time distributions for illustration.
Other concerns
A minor question: should the coefficient of be instead in Eq. (12)?
Good catch. Sorry about the typo; it is fixed now.
Can you provide some explanation on the two-layer network example? In particular, how was computed, and why do the runs at and appear unstable, with smoothly decreasing loss around ?
Sorry for the brevity here, we will add a bit more explanation to the revised paper.
First, we consider a student network initialized near a teacher network with the same architecture. By construction, the system is initialized near a fixed point, so we can approximate the system as linear around that fixed point by numerically computing its Hessian.
Then we use the covariance dynamics discussed in Sec. 3 (Eq. 11) to determine . The important point not really discussed in the original text (for space reasons...sorry about this) is that, since the Hessian might not be diagonal or circulant, we can't use Eq. 13 or Eq. 16; we have to determine in a more general way. We have to numerically check when the right-hand side of Eq. 11 becomes unstable. Since Eq. 11 is linear in (in particular, one has a Lyapunov-like equation), this amounts to determining the value at which the matrix that determines the right-hand side develops an eigenvalue above zero.
Regarding the weird-looking behavior of and , we found that the precise shape of those curves varied from seed to seed. Generally we found that values above but near the critical value could have large excursions above the dashed line, but would never blow up to infinity. This was also affected a bit by the time step we used; we think the effect is a little smaller if we use a smaller . This is a subtle but important point, so we will also add some discussion of it to the revised version (although probably this will appear in an appendix).
Thank you for the detailed reply. Most of my questions are addressed, and I appreciate the candid discussion.
Gradient scaling. I agree with the proposed reframing in terms of the trade-off between instability and slow, biased learning. This also resolves my question about a fixed gain per arrival.
Generality. The clarification that only the first two moments of the noise are used addresses my concern about the noise model. The renewal-process condition is correct and reduces to Eq. (13) in the Poisson case.
Two-layer network. The explanation of how was computed is helpful. Since, as you explain, the runs at and do not diverge, they are better described as stable with large fluctuations. Such fluctuations are expected near the threshold, because the linear theory predicts a stationary variance of order (Eq. (14) and Figure 2(c)). An event-driven simulation would remove the dependence on , and results over several seeds would make Figure 3(b) more informative.
As these changes are substantial and a revised manuscript seems not uploaded yet, I will keep my current score. I expect the revised version to be considerably stronger.
Thank you for engaging us. We are working on revising the paper now and should post an updated version before the end of the day.
Rebuttal (part 1)
Thank you for your close reading of our paper and for your helpful comments. We hope to convincingly address your concerns below, but please let us know if we fall short of doing so!
On gradient scaling
This is a very astute observation. Thinking hard about this comment has made us decide to reframe the paper slightly: indeed, the 1/ρ compensation is not essential, and we will address the more general case explicitly in the revised version of the paper.
Your intuition is exactly right, but for completeness' sake, we will recount it and add our own thoughts. First, as you say, we introduced the 1/ρ compensation for a very specific reason: we wanted to hold average dynamics fixed, and we wanted the steady state solution of average dynamics to correspond to the minimizer of our objective (e.g., Eq. 8). We thought that this would allow us to isolate what we consider the 'interesting' effect of Poisson gradient arrivals, which is very different behavior at the level of covariance dynamics.
However, you and Reviewer i2MH make a good point. Why would the system ever be in this regime if it yields instability? Doesn't the 1/ρ compensation unfairly constrain the system? After some thought, we mostly agree. For the reasons just described, we still think the fixed mean dynamics case is interesting and worth discussing, but the more general case should also be discussed.
Suppose we do not include the 1/ρ compensation. Exactly as you observe, there is a 'no free lunch' issue. While one can avoid instability (i.e., exploding variance), two other problems appear: the convergence of mean dynamics is generally slower, and its steady state solution is generally biased. The more the coefficient of the Poisson arrival term differs from 1/ρ, the more this is true. Hence, we absolutely agree that a better 'story' may be the following. Intermittent gradient arrivals present a problem for learning. Rare enough arrivals either produce instability OR slow and biased learning.
We think this story is still quite interesting. We also don't think the layout of the paper needs to change; we just ought to address the tradeoff between speed and stability in the respective sections. The original version of the paper already discussed this in Sec. 6 (we note that the bias is spatial-frequency-dependent) and Sec. 7 (we note that gradient-averaging can really slow down learning).
There are two kinds of plots we can add in a few places that could help illustrate the speed-accuracy tradeoff. We can plot the maximum learning speed versus some parameter of interest (e.g., the Poisson arrival coefficient, or the regularization coefficient γ) and the minimum bias versus some parameter of interest (kind of like Fig. 5c).