- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper theoretically investigates the behavior of Adam with stability parameter in separable linear classification. This paper characterize the transition of Adam's updates from sign-like to gradient-like behavior and analyze how the classifier evolves from near -margin optimality toward -margin optimality.
Critical strengths
- This paper studies the behavior of Adam with a small positive stability constant in separable linear classification.
- The paper provides a theoretical characterization of how Adam's implicit bias transitions from -margin optimality to -margin optimality.
Critical weaknesses
- The main weakness of this paper is its presentation, which makes the theoretical contributions unnecessarily difficult to understand. The paper introduces numerous auxiliary quantities and presents many results in terms of these quantities, rather than clearly explaining what the results imply. For example, Proposition 3.2 involves several layers of notation, such as , , , and . As a result, readers must repeatedly trace back through definitions and substitute multiple expressions to understand the actual dependence on and . More importantly, the paper does not sufficiently explain the implications of its theoretical bounds or how the various results fit together to support the main conclusions. I believe the paper needs a substantial reorganization that emphasizes the key theoretical insights and their interpretations, rather than simply presenting a sequence of technically detailed results.
- The claim in the Introduction that Adam's updates are nearly aligned with the sign gradient appears overstated. Theorem 2 only shows that the sign residual converges to zero, but since this residual is weighted by the magnitude of each gradient coordinate, i.e., , Theorem 2 does not immediatelly imply that the update direction of Adam is equivalent to sign gradient descent.
- is used in line 247, while it is defined later, i.e., in line 265.
Overall, the theoretical results appear interesting. However, the large amount of notation and the difficulty of understanding the relationships among the main results substantially reduce the readability of the paper. I strongly encourage the authors to improve the presentation.
Questions for authors
See the weakness section.
3 comments
We thank the authors for the response. I agree that there is insufficient time during the rebuttal period to fix the presentation issues. However, the current manuscript is extremely difficult to read due to notation issues. I cannot support accepting the paper in its current state, and therefore I would like to maintain my negative score.
We believe that our presentation can be made clearer. However, we respectfully disagree that our paper has significant notation problems or that the remaining presentation concerns outweigh its theoretical contribution. Could you clarify which notation issues remain unresolved after our revisions and which specific definitions, statements, or passages are still difficult to follow?
Thank you for your feedback on our paper. We address your concerns below.
1. Presentation (Weakness 1)
We have added Figure 1 to the revised file to help illustrate our main results. However, within the time available for this rebuttal, we have not yet fully addressed all presentation concerns. We will work on further improving the exposition.
2. Sign residual (Weakness 2)
Below, we distinguish between the motivating coordinatewise intuition and the guarantee established by Theorem 2.
The intuition can be seen directly in the memoryless case, where β1=β2=0. In this case, Adam’s update direction is exactly Dt[j]=∣gt[j]∣+ϵgt[j]. Thus, on coordinates where ∣gt[j]∣≫ϵ, the update is close to sign(gt[j]), whereas on coordinates where ∣gt[j]∣≪ϵ, it is close to gt[j]/ϵ. In particular, if every nonzero gradient coordinate dominates ϵ, the update is coordinatewise close to the sign-gradient vector. This is the mechanism underlying the introductory intuition.
With momentum, the update also depends on past gradients, so the exact coordinatewise identity above no longer holds in general. Moreover, Theorem 2 does not assume that every nonzero gradient coordinate dominates ϵ. We therefore motivate and define the sign residual through its connection to first-order loss decrease. A small sign residual means that Adam’s first-order loss decrease is close to that of sign descent at the same iterate and with the same step size. As the reviewer correctly notes, this does not, by itself, establish unweighted closeness of the update to the sign-gradient vector.
This is the sense in which our analysis characterizes the update as “sign-like.” The motivation for this residual and the detailed justification of its interpretation are provided in Section 2, immediately after the residual definitions.
We have revised the introductory wording to make this interpretation explicit and to distinguish it from the motivating coordinatewise intuition.
3. Placement of Λϵ (Weakness 3)
Thank you for pointing this out. We have revised Section 3 to introduce Λϵ before its first use in the definition of Tˉϵ. When introducing it, we state the required conditions Λϵ→∞ and ϵΛϵ→0 as ϵ↓0 and indicate that its specific choice will be given below.
We retain the explicit formula immediately after the preliminary error bound to explain why this choice is useful. Specifically, we choose Λϵ so that the two terms ℓϵ/Λϵ and ∑t<Tˉϵηt2 are both controlled by the common bound O(ωa(ℓϵ)). This preserves the motivation for the formula while ensuring that the parameter is introduced before it is used.