- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies high-dimensional empirical risk minimization with non-Gaussian features and convex regularization, including nonsmooth penalties such as Lasso. The authors propose a surrogate optimization problem driven by a single Gaussian vector, with its parameters calibrated using the actual joint distribution of features and labels.
The paper proves existence and uniqueness of the population calibration equations. It then uses a cavity argument and a Hamilton–Jacobi equation to relate the surrogate to the original estimator. Under an additional conjecture about the conditional Gaussian distribution of fresh prediction scores, the authors establish asymptotic agreement in normalized means, prediction variances, and prediction laws for Lasso and strongly convex penalties.
Experiments with correlated non-Gaussian features show close agreement between the surrogate and empirical support-recovery statistics, although this support agreement is not covered by the theoretical guarantee
Critical strengths
-
The population surrogate has a solid mathematical foundation. Theorem 1 proves existence and uniqueness of the calibration equations for a broad class of convex penalties, including nonsmooth ones. This is a useful result on its own, since it makes the proposed surrogate well-defined and gives a starting point for further analysis.
-
The proof approach is interesting and handles nonsmooth regularization carefully. The cavity and Hamilton–Jacobi argument avoids differentiating the penalty directly. The authors also develop separate cutoff-removal arguments for strongly convex penalties and pure Lasso. These are meaningful technical contributions, even though the final prediction comparison still relies on Conjecture 1.
Critical weaknesses
-
The main theoretical result depends on an unproved conjecture. Conjecture 1 assumes that fresh prediction scores are approximately Gaussian after conditioning on the fresh feature and averaging over training randomness. Establishing this under the stated general conditions, especially for nonsmooth penalties, is itself a difficult part of the problem. Theorem 5 and Theorem 2 both rely on this conjecture. As a result, the paper provides a conditional characterization of the estimator's prediction behavior, while the central Gaussian approximation remains unresolved. This matters particularly because related work has already studied Gaussian surrogates for correlated Gaussian Lasso and non-Gaussian ERM with smoother regularization. The new treatment of nonsmooth penalties is valuable, but the missing Gaussian approximation limits how much of the broader extension has been established.
-
The experiments do not directly test the main theoretical assumption and cover a limited setting. Section 3 focuses on one synthetic configuration with and . The support-recovery agreement is impressive, but the paper itself acknowledges that transferring support statistics from the surrogate to ERM remains an open problem. More importantly, Figure 2(b) compares unconditional score distributions. Conjecture 1 concerns the conditional distribution for a fixed fresh feature, which is a stronger claim. The experiments therefore give only indirect evidence for the conjecture. The regularization grid also uses strictly positive ridge strengths, so pure Lasso, which receives substantial attention in the theory, is not directly tested.
Questions for authors
-
Is there a nontrivial class of non-Gaussian feature distributions and nonsmooth penalties for which Conjecture 1 can already be proved? Even a result under stronger assumptions would help establish how broadly the proposed theory applies.
-
Could the authors directly measure the conditional Wasserstein distance in Equation (21), by fixing fresh features and repeatedly sampling training datasets, across increasing values of and ?
3 comments
Thank you for the detailed response and the new proof. The proof of the conditional Gaussian prediction principle addresses my main concern about the original submission. The new experiments also provide more direct evidence for the theoretical results. These changes significantly improve the paper, and I am willing to raise my rating from 2 to 3.
Thank you for your constructive feedback and for raising your rating. We are pleased that the new proof addresses your main concern, and we appreciate your recognition of the theoretical and experimental improvements.
Thank you a lot for your comments. For the conditional CLT and Wasserstein convergence, see A1; for real-data validation and comparisons with prior predictions, see A6; for the sufficient conditions ensuring (22) for logistic Lasso, see A4. The synthetic experimental model's Poincaré condition is addressed separately in A2.
A6. Numerical validation of the proved predictions and comparison with prior work
The original experiments assessed our predictions indirectly through support recovery, with close agreement between the surrogate and ERM. We have moved these exploratory support illustrations to Appendix J, Figures 4 and 5 since support transfer is not among our proved results. We now provide predicted score histograms and classification performance on a diamonds-based semisynthetic population.
The interior interpolation points in Figure 1(a) assess Theorem 2, formerly Conjecture 1; Figure 2(b,c) assesses the population-calibrated prediction law in Corollary 5, and the classification-error predictions in Figure 2(a) assess Corollary 6. These diamonds-based comparisons are presented in Section 3.7, with numerical details in Appendix I.2. We also directly assess the prediction-variance conclusion of the original Theorem 2 (now Theorem 3): Figure 3 in Appendix I.1 reports the maximum relative error in α2 along the calibrated path, which decreases overall as n,p increase at fixed p/n.
We also compare our predictions with:
Agreement with ERM supports the accuracy of our predictions. The comparisons illustrate the contribution of retaining both the original nonsmooth penalty and the actual feature–label law: reproducing the latter through a Gaussian-mixture model can require many components even when the original data have few features.