- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the asymptotic behavior of regularized empirical risk minimization for linear regression with independent, identically distributed heavy-tailed feature entries. The authors consider the proportional high-dimensional limit, where the sample size and feature dimension tend to infinity with . review1 The main contribution is a replica-based derivation of functional fixed-point equations that describe a random one-dimensional optimization problem associated with an individual coefficient. Rather than retaining the entire training dataset, this characterization represents its effect through a random response function whose distribution satisfies the fixed-point equations. The authors use this description to predict the limiting coefficient-error distribution and the resulting test prediction error. review1 I am not sufficiently familiar with the replica method to assess the derivation in detail, so I have not independently verified it. Nevertheless, the close agreement between the theoretical predictions and the simulations provides encouraging empirical support for the proposed characterization in the tested settings. review1 The analysis also offers interesting insights into universality across heavy-tailed feature distributions and the scaling of typical and extreme prediction errors. In addition, the authors characterize a Bayes-optimal benchmark for Gaussian teacher coefficients and Gaussian label noise, and compare independent-entry and elliptical heavy-tailed feature models.
Critical strengths
- The paper provides a novel asymptotic characterization of empirical risk minimization with independent heavy-tailed features, reducing the problem to a one-dimensional random optimization described by functional fixed-point equations. This extends previous results beyond quadratic losses and elliptical models.
- The theoretical predictions agree closely with numerical simulations and lead to interesting insights into heavy-tail universality and the different scaling behaviors of typical and extreme prediction errors.
- The paper also derives an explicit characterization of Bayes-optimal estimation error and establishes an interesting comparison between independent and elliptical heavy-tailed feature models.
Critical weaknesses
-
The results are restricted to data with independent and identically distributed feature entries.
-
A large part of the theoretical framework is presented only in the appendix, including the definitions of the main quantities , , and in Appendix B.3. This makes the paper difficult to follow for readers who are not familiar with this literature. I understand that this is an editorial choice, but I think the main text would benefit from more mathematical details. As it stands, readers may find it difficult to identify the precise mathematical contributions without going through the appendix.
-
Most numerical experiments use Huber loss, ridge regularization, Gaussian teacher coefficients, and Gaussian noise. These experiments provide useful validation, but one of the main contributions of the paper is its ability to handle more general losses and penalties. An additional experiment with a nonsmooth loss or a nonquadratic strongly convex penalty would help demonstrate this broader scope.
Questions for authors
Could you state clearly which conclusions depend on the replica assumptions and which are proved independently of them? In particular, are any additional assumptions needed to identify the functional fixed point with the limiting coefficient distribution? Did you state the existence and uniqueness of the solution somewhere? Which precise result of Montanari and Saeed establishes Proposition 4 under your assumptions? Could you also distinguish universality of the coefficient-error moment from Gaussianity of the test prediction score? Could you include a small experiment with another loss or a nonquadratic strongly convex penalty? This would help demonstrate the scope of the functional characterization beyond the Huber–ridge setting used in the experiments. Could you extend your results to non iid data and more general models?
Reminder to engage in the discussion with Authors
Dear Reviewers,
Thank you for your effort in reviewing this submission. The authors’ responses to your comments have been available since November 10. Please take the time to read them and engage in the discussion. In particular, please indicate whether your main concerns have been adequately addressed and raise any remaining questions as soon as possible, so that the authors have an opportunity to reply before the rebuttal period ends on November 18.
Best regards
AC of Submission 20800