- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the implicit bias of steepest descent towards max-margin solutions. The paper first decomposes the parameter space into three components: An (update-irrelevant) invariant component, an escaping component, and a component associated with the real optimization variables. The paper then quotients out the symmetries and optimizes out the escape directions, yielding a coercive reduced objective that is strongly convex on compact convex subsets and admits a unique minimizer (Theorem 3.1). With these in place, the paper establishes risk convergence (Theorem 4.3) using an error bound for the risk objective (Lemma 4.2), and then derives results for margin convergence (Theorems 5.2 and 5.3). As an aside, the paper also presents interesting counterexamples showing that margin convergence doesn't necessarily guarantee directional convergence.
Critical strengths
- The paper introduces an insightful decomposition of the parameter space.
- I believe that the theoretical developments in this paper make a solid contribution to the field.
Critical weaknesses
None.
Questions for authors
I believe the analysis of directional convergence in this paper is closely connected to the theory of cosmic spaces in variational analysis; see "Rockafellar, R. T., & Wets, R. J. (1998). Variational analysis. Berlin, Heidelberg: Springer Berlin Heidelberg." In particular, I believe the notion of directional convergence considered here is equivalent to the cosmic convergence in variational analysis (given the norm divergence established Theorem 5.2); see Definition 3.1 of the monograph. Moreover, in this terminology, the accumulation set considered here can essentially be identified with the horizon outer limit of the unnormalized iterates; see Section 4.F of the same monograph. I think highlighting this connection would deepen the discussion and suggest new directions for future research. For example, even when directional convergence fails, it would be interesting to determine whether one can still establish rates at which the iterates approach their cosmic accumulation set, measured using the cosmic metric in Exercise 4.48 of the same monograph.
3 comments
Thanks for your response, and all the best with your submission :)
Thank you for your response. We appreciate your continued positive assessment of our work.
Thank you for your insightful suggestions. We answered your question below.
Q. Connection between cosmic spaces in variational analysis.
Thank you for specifically pointing us to cosmic spaces in variational analysis. When the iterate norms diverge, directional convergence corresponds to cosmic convergence to a direction point, and the normalized accumulation set corresponds to the unit-norm directions in the horizon outer limit. We added this connection in Section 5, highlighted in blue. This perspective may provide useful tools for studying implicit bias.
Establishing rates of approach to the cosmic accumulation set in the metric of Exercise 4.48 of the monograph requires further analysis beyond the limited rebuttal period. We plan to investigate these rates in the next revision.