- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This submission investigates the optimization dynamics and implicit bias of gradient descent under feature-mediated spurious correlations on linearly separable data. In contrast to the conventional label-mediated setting.
Critical strengths
-
Conceptual Framing of Feature-Mediated Correlations and Representation Linearisation: The paper tackles an important and under-theorized setting by distinguishing instance-level feature-mediated spurious correlations () from classic label-mediated dependencies (). The Sparse Autoencoder (SAE) empirical analysis on Colored-MNIST provides an intuitive perspective on representation learning, demonstrating that multi-layer neural networks can approximately linearize nonlinear input-level correlations within latent feature spaces.
-
Continuous-Distribution Population Implicit-Bias Formulation: Extending the separable implicit-bias framework of Soudry et al. (2018) from discrete support vectors to continuous distributions with strictly positive boundary margin densities is a substantive theoretical formulation. Formulating population gradient descent via Laplace-type integrals over continuous margin levels and controlling the transverse parameter drift via an orthogonal energy argument (Lemma I.8) offers an insightful mathematical explanation for the emergence of the residual correction.
Critical weaknesses
- Mathematical Breakdown at the Critical Threshold () and Invalid Asymptotic Derivations: The paper's central theoretical claim of a continuous phase transition at is mathematically flawed:
-
Failure at : Theorem D.4(ii) (Eq. 43) groups together and claims a minority rate of , which evaluates to at . However, the proof in Appendix E.5 explicitly relies on Eq. (90): . At , the leading coefficient vanishes identically, which invalidates the estimate and breaks the subsequent integration. Analyzing the underlying ODE system at reveals that the trajectory gap satisfies , which induces an additional logarithmic factor, yielding a true critical rate of rather than .
-
Invalid Application of Karamata's Theorem: Proposition H.5 applies Karamata’s theorem to under the assumption . This inference is mathematically invalid: does not guarantee slow variation (e.g., , yet oscillates unbounded), undermining the noisy asymptotic rates in Appendix F.2.
-
Flawed Asymptotic Exponent Notation: In Theorem 5.1 and Corollary G.1, the error bounds are expressed as . Stating an exact logarithmic power in the numerator while leaving an unrefined factor in the denominator is inconsistent, as can absorb or dwarf any polylogarithmic factor .
- Artificial 1D Dimensional Collapse, Generative Model Conflict, and Metric Misinterpretation:
-
Subspace Collapse: To obtain closed-form ODE dynamics, the analysis imposes simultaneous isotropy (Definition 5.2), zero orthogonal initialization (, Assumption D.2), and conditional zero-mean orthogonal features (, Assumption D.1). This artificially restricts parameter updates to a 1D scalar line , bypassing the multi-dimensional transverse optimization dynamics of neural network training. Under non-zero feature noise , updates inject non-zero orthogonal components via that are not bounded or tracked in Appendix F.2.
-
Generative Model Inconsistency: Definition 2.3 (Eq. 3) defines the mixture conditioned on with constant probabilities and (). This implies , forcing identical -margin distributions across both groups (), which directly contradicts the subsequent assumption of unequal group margins ().
-
Conflation of Surrogate Loss with 0–1 Error: Throughout the manuscript, the soft surrogate error is referred to as "classification error". On linearly separable distributions with positive margin, the actual 0–1 misclassification error becomes identically zero in finite time. Asserting minority learning bottlenecks or worst-group performance guarantees based purely on decaying soft loss tails overstates the practical robustness implications.
Questions for authors
-
Resolution of the Critical Phase Transition at : In Eq. (90), the bound requires , failing when . Analyzing the coupled drift equations at shows that , which implies an asymptotic minority rate of . How do you reconcile this additional logarithmic factor with the claimed rate of and the continuity claim at in Theorem D.4 and Theorem 5.3?
-
Generative Model Specification vs. Margin Discrepancy: In Definition 2.3 (Eq. 3), the group assignment is specified conditionally on with fixed probability , which forces and to be independent and implies identical conditional -distributions. How can this formulation support distinct margin supports and group-specific densities without reformulating the joint law as and ?
-
Rigorous Foundation for Proposition H.5: Proposition H.5 invokes Karamata's theorem based solely on the assumption that . Since does not guarantee that is slowly varying, what additional regularity conditions (such as bounded variation or explicit decay rates on ) are required to ensure that Eq. (187) and the resulting rates hold?
-
Transverse Drift under Feature Noise: In Eq. (122), the parameter updates include the term . Because the noise is arbitrary and bounded within a ball in , these gradient updates generally possess non-zero projections onto . Why does this transverse drift not invalidate the reduction to a 1D scalar ODE in Lemma F.5?
-
Practical Interpretation of Error Metric: Given that the 0–1 classification error reaches zero on linearly separable data, what exact conclusions can be drawn from the asymptotic decay of the surrogate loss regarding actual test-time worst-group classification accuracy?