acceptodds
Under review as a conference paper at ICLR 2027

An Asymptotic One-Sample Surrogate for Non-Gaussian ERM with General Convex Regularization

Abstract

We study convex empirical risk minimization (ERM) with non-Gaussian features and nonsmooth penalties through a surrogate driven by a single Gaussian vector. The surrogate preserves the penalty and calibrates its coefficients using the actual joint law of the feature–label model. We prove that this population calibration exists and is unique. We conjecture a conditional Gaussian prediction principle for fresh scores when the data satisfy a Poincaré inequality. Conditional on this conjecture, standard concentration conditions on the data and regularity conditions on the losses imply agreement between the ERM minimizer and its surrogate in normalized means, prediction variances, and conditional fresh-score laws averaged over the fresh feature. The result covers general uniformly strongly convex penalties and, under projection anti-concentration, pure Lasso with regularization strengths bounded away from zero and infinity. Transfer of the surrogate's support statistics to the trained estimator remains open, although it is strongly supported by simulations. The core proof relies on a cavity Hamilton–Jacobi argument that compares the main statistics of the two minimizers without differentiating the penalty.

Primary area:
learning theory
Keywords:
high-dimensional ERM, non-Gaussian data, nonsmooth regularization, Hamilton-Jacobi, CGMT

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_jWL1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies high-dimensional empirical risk minimization with non-Gaussian features and convex regularization, including nonsmooth penalties such as Lasso. The authors propose a surrogate optimization problem driven by a single Gaussian vector, with its parameters calibrated using the actual joint distribution of features and labels.

The paper proves existence and uniqueness of the population calibration equations. It then uses a cavity argument and a Hamilton–Jacobi equation to relate the surrogate to the original estimator. Under an additional conjecture about the conditional Gaussian distribution of fresh prediction scores, the authors establish asymptotic agreement in normalized means, prediction variances, and prediction laws for Lasso and strongly convex penalties.

Experiments with correlated non-Gaussian features show close agreement between the surrogate and empirical support-recovery statistics, although this support agreement is not covered by the theoretical guarantee

Critical strengths
  1. The population surrogate has a solid mathematical foundation. Theorem 1 proves existence and uniqueness of the calibration equations for a broad class of convex penalties, including nonsmooth ones. This is a useful result on its own, since it makes the proposed surrogate well-defined and gives a starting point for further analysis.

  2. The proof approach is interesting and handles nonsmooth regularization carefully. The cavity and Hamilton–Jacobi argument avoids differentiating the penalty directly. The authors also develop separate cutoff-removal arguments for strongly convex penalties and pure Lasso. These are meaningful technical contributions, even though the final prediction comparison still relies on Conjecture 1.

Critical weaknesses
  1. The main theoretical result depends on an unproved conjecture. Conjecture 1 assumes that fresh prediction scores are approximately Gaussian after conditioning on the fresh feature and averaging over training randomness. Establishing this under the stated general conditions, especially for nonsmooth penalties, is itself a difficult part of the problem. Theorem 5 and Theorem 2 both rely on this conjecture. As a result, the paper provides a conditional characterization of the estimator's prediction behavior, while the central Gaussian approximation remains unresolved. This matters particularly because related work has already studied Gaussian surrogates for correlated Gaussian Lasso and non-Gaussian ERM with smoother regularization. The new treatment of nonsmooth penalties is valuable, but the missing Gaussian approximation limits how much of the broader extension has been established.

  2. The experiments do not directly test the main theoretical assumption and cover a limited setting. Section 3 focuses on one synthetic configuration with n=200n=200 and p=400p=400. The support-recovery agreement is impressive, but the paper itself acknowledges that transferring support statistics from the surrogate to ERM remains an open problem. More importantly, Figure 2(b) compares unconditional score distributions. Conjecture 1 concerns the conditional distribution for a fixed fresh feature, which is a stronger claim. The experiments therefore give only indirect evidence for the conjecture. The regularization grid also uses strictly positive ridge strengths, so pure Lasso, which receives substantial attention in the theory, is not directly tested.

Questions for authors
  1. Is there a nontrivial class of non-Gaussian feature distributions and nonsmooth penalties for which Conjecture 1 can already be proved? Even a result under stronger assumptions would help establish how broadly the proposed theory applies.

  2. Could the authors directly measure the conditional Wasserstein distance in Equation (21), by fixing fresh features and repeatedly sampling training datasets, across increasing values of nn and pp?

3 comments

Thank you a lot for your comments. For the conditional CLT and Wasserstein convergence, see A1; for real-data validation and comparisons with prior predictions, see A6; for the sufficient conditions ensuring (22) for logistic Lasso, see A4. The synthetic experimental model's Poincaré condition is addressed separately in A2.

  • Conditional-Gaussian theorem: see A1 in our reply to iqWV for Theorem 2 and Appendix C, and A3 in our reply to KSWM for the Hamilton–Jacobi mechanism.
  • Real-data validation and comparison with prior predictions: see A6 below for the diamond-data histograms, performance predictions, and smooth-penalty and Gaussian-mixture comparisons.
  • Logistic Lasso under non-Gaussian designs: see A4 in our reply to JfYx for the sufficient conditions ensuring (22), and A1 in our reply to iqWV for the conditional Gaussian theorem under the stated data assumptions.
  • Conditional Wasserstein distance as n,pn,p increase: Figure 1(a) shows decreasing errors toward Gaussian finite-sampling benchmarks as dimension grows. See A1 in our reply to iqWV and Appendix I.1.

A6. Numerical validation of the proved predictions and comparison with prior work

The original experiments assessed our predictions indirectly through support recovery, with close agreement between the surrogate and ERM. We have moved these exploratory support illustrations to Appendix J, Figures 4 and 5 since support transfer is not among our proved results. We now provide predicted score histograms and classification performance on a diamonds-based semisynthetic population.

The interior interpolation points in Figure 1(a) assess Theorem 2, formerly Conjecture 1; Figure 2(b,c) assesses the population-calibrated prediction law in Corollary 5, and the classification-error predictions in Figure 2(a) assess Corollary 6. These diamonds-based comparisons are presented in Section 3.7, with numerical details in Appendix I.2. We also directly assess the prediction-variance conclusion of the original Theorem 2 (now Theorem 3): Figure 3 in Appendix I.1 reports the maximum relative error in α2\alpha^2 along the calibrated path, which decreases overall as n,pn,p increase at fixed p/np/n.

We also compare our predictions with:

  1. The equations of Yaakoubi et al. (ICML 2026) using two smooth approximations to the ℓ1\ell_1 penalty. As explained in Section 3.7 and Appendix I.2, these zero-ridge comparisons are extrapolations outside their uniform strong-convexity assumptions; their discrepancies do not contradict the stated theorem.
  2. The predictions of Loureiro et al. (2021) for Gaussian-mixture models fitted to the diamond data. This comparison tests whether their mixture-based predictions reproduce those obtained from the original data law. Our plots show that even the fitted mixture with ten Gaussian components per class does not recover the diamond-data predictions.

Agreement with ERM supports the accuracy of our predictions. The comparisons illustrate the contribution of retaining both the original nonsmooth penalty and the actual feature–label law: reproducing the latter through a Gaussian-mixture model can require many components even when the original data have few features.

Thank you for the detailed response and the new proof. The proof of the conditional Gaussian prediction principle addresses my main concern about the original submission. The new experiments also provide more direct evidence for the theoretical results. These changes significantly improve the paper, and I am willing to raise my rating from 2 to 3.

Thank you for your constructive feedback and for raising your rating. We are pleased that the new proof addresses your main concern, and we appreciate your recognition of the theoretical and experimental improvements.

Reviewer_JfYxOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies high-dimensional convex ERM with non-Gaussian features and potentially non-smooth regularization. It introduces a surrogate whose only random input is a single Gaussian vector, with scalar and vector parameters calibrated from the actual feature label distribution.

The authors establish existence and uniqueness of the population calibration under stated assumptions. Conditional on Conjecture 1 and additional assumptions, they establish agreement of normalized means and prediction variances, with consequences for fresh-score distributions and classification error. The comparison uses a cavity–Hamilton–Jacobi argument.

The experiments compare the surrogate with finite-sample ERM under non-Gaussian and covariance-matched Gaussian designs.

Critical strengths

The main idea works through perturbations of the optimal value and associated responses, rather than differentiating the penalty. This could be a technical direction for extending asymptotic analyses.

Critical weaknesses

My main concern is that the incremental contribution is not convincingly distinguished from existing work, and the presentation does not help clarify it.

Signal-plus-Gaussian descriptions and proximal surrogates are established ideas, as the paper’s own related-work discussion and AMP consistency calculations acknowledge. The paper does not clearly explain which new mathematical obstacle it resolves or why this constitutes a substantial advance. A sharper comparison with Celentano–Montanari–Wei, Loureiro et al., and Yaakoubi et al. is needed, alongside discussion of recent alternative proof approaches by Asgari–Montanari–Saeed and Vilucchio et al., accounting for their different assumptions.

The introduction presents the surrogate before establishing this motivation, leaving unclear whether the principal contribution is a new proof technique, a new statistical characterization, or an application. This ambiguity is particularly problematic because the strongest intended extension remains conditional on a conjecture.

The main comparison result relies on Conjecture 1. Although the transport argument may be nontrivial, its significance as a new proof framework remains insufficiently demonstrated. The experimental support is also limited: much of the evaluation concerns support recovery, which lies outside the proved guarantees, while Figure 2(b)’s pooled, unconditional score histograms do not directly test the fixed-feature conditional approximation required by Conjecture 1.

Questions for authors

Is there a concrete non-Gaussian setting with nonsmooth regularization in which Conjecture 1 and the conditions in (22) can be justified? I am asking about the basis for the current claims, rather than requesting a complete solution to the conjecture as future work.

Is there direct numerical evidence for the fixed-feature conditional approximation in Conjecture 1, beyond the pooled histograms in Figure 2(b)?

5 comments

Thank you for your comments. A4 gives sufficient conditions for (22), and A5 compares the cited works. The former Conjecture 1 is proved in Theorem 2, Appendix C; see A1 in our reply to iqWV. A3 in our reply to KSWM explains the contribution, and A6 in our reply to jWL1 gives the numerical comparisons. For the Poincaré bound, see A2 in our reply to iqWV.

A4. Sufficient conditions for logistic Lasso to satisfy (22)

We have added Proposition 4, with its proof in Appendix A.5, establishing the population conditions (22) for logistic regression with pure Lasso under an additional condition on the labels. Suppose the training pairs are i.i.d., p/n→c∈(0,∞)p/n\to c\in(0,\infty), c‾xIp⪯Σx⪯KIp\underline c_x I_p\preceq\Sigma_x\preceq K I_p, and Assumptions 1–3 hold, including projection anti-concentration for the pure-Lasso branch. Take

\rho(w)=\lambda_1\lVert w\rVert_1,\qquad 0<\lambda_-\le\lambda_1\le\lambda_+<\infty.$$ If, for some fixed $\epsilon\in(0,1/2]$, $$\epsilon\le\mathbb P(y=1\mid x)\le1-\epsilon \qquad\text{almost surely},$$ then the population solution satisfies (22): $$\sup_n\left\lbrace\frac1n\mathbb E_G\lVert W_\ast\rVert^2+\kappa_\ast\right\rbrace<\infty, \qquad \liminf_{n\to\infty}\alpha_\ast^2>0.$$ Here $\lambda_-$ and $\lambda_+$ are fixed bounds; $\lambda_1$ may depend on the problem size. The label condition requires that labels retain a nonvanishing degree of uncertainty given the features. Together with **Theorem 2**, which proves the former Conjecture 1, this provides both the conditional Gaussian approximation and the population bounds needed for **Theorem 3** under the stated assumptions. Neither Gaussian features nor an additional ridge penalty are required. **The central theoretical contribution and the incremental-contribution concern.** Please see **A3 in our reply to KSWM** for the Hamilton–Jacobi unification. **A6 in our reply to jWL1** also answers the criticism that the result is incremental: it compares the real-data score and performance predictions with smooth-penalty predictions from Yaakoubi et al. and the Gaussian-mixture predictions of Loureiro et al.; even the fitted mixture with ten Gaussian components per class does not reproduce the diamond-data predictions. **A5 below** gives the detailed literature comparison. ### A5. Comparison with the specific works cited by the reviewers In response to the reviewers' questions about specific prior works, the table below states the main distinctions. **Section 2, “Main contributions and comparison with prior work,”** gives the detailed assumptions and comparisons. | Reference | Relevant result and distinction | | --- | --- | | Celentano–Montanari–Wei | Their Gaussian-sequence surrogate and support results concern squared-loss Lasso with Gaussian design and a linear response model. They do not directly characterize our non-Gaussian logistic or pseudo-Huber examples. | | Loureiro et al. (2021a,b), Gaussian covariates and mixtures | Their results allow nonsmooth convex penalties but prescribe jointly Gaussian teacher–student covariates (2021a) or Gaussian-mixture features (2021b). Our result permits non-Gaussian dependent features under the stated concentration assumptions, without prescribing Gaussian components. | | [Pesce et al. (2023)](https://proceedings.mlr.press/v202/pesce23a.html) | They characterize training and test errors for Gaussian-mixture features with single-index labels, and determine when a single Gaussian gives the same errors. Our result retains the actual feature–label law without prescribing Gaussian components or a single-index response model. | | [Dandi et al. (2023)](https://papers.neurips.cc/paper_files/paper/2023/file/abccb8a90b30d45b948360ba41f5a20f-Paper-Conference.pdf) | They prove universality relative to an equivalent Gaussian mixture under a within-cluster feature-projection CLT (Assumption 4), verified for specific random-feature models. We instead prove the fresh-score Gaussian approximation from Poincaré concentration (Theorem 2, Appendix C), retaining the actual signal projection. | | Yaakoubi et al. | They already retain the non-Gaussian signal projection. Their smooth-regularizer treatment does not retain exact Lasso/elastic-net thresholding. We identify the quadratic case recovered by our equations and the nonsmooth extension. | | Asgari–Montanari–Saeed | Their Kac–Rice analysis assumes Gaussian covariates and a three-times continuously differentiable regularizer in its stated general setting. These assumptions do not cover our exact nonsmooth, non-Gaussian setting. | | Vilucchio et al. | Their Gaussian min–max/AMP analysis permits nonconvex and nonsmooth objectives, subject to its conditions, but assumes a Gaussian design and a specified single-index output channel. Nonconvexity is not the reason it does not directly apply here. |
Reviewer_JfYxOpenReview ↗

Thank you for the revision

Thank you for the detailed response and the substantial revision. The new Section 2 makes the contribution clearer even if I still have some concerns:

1. Scope of the contribution. As now stated in Section 2, the main result extends Yaakoubi et al. (2026) to nonsmooth penalties, including pure Lasso. If I understood correctly the surrogate equations are of a known form and Theorem 2 is proved on the interior domain, where the positive probes provide strong convexity. The contribution is therefore a technical extension of existing results, and I think the paper should present it as such.

2. Comparison with Yaakoubi et al. (Figure 2). If I read Appendix I.2 correctly, the curves for Yaakoubi et al. are computed using the Hessian of the smoothed penalty at zero, ∇2ρε(0)=λ1/(n ε) Ip\nabla^2 \rho_\varepsilon(0) = \lambda_1/(\sqrt{n}\,\varepsilon)\, I_p. If so, this corresponds to a ridge penalty rather than to the smoothed Lasso, and part of the mismatch could come from this approximation rather than from their framework itself. I may have misunderstood the implementation, in which case a clarification would help.

3. Hamilton–Jacobi interpolation. Interpolation and Hamilton–Jacobi arguments are well established for mean-field and high-dimensional inference problems (e.g., Guerra's interpolation and the Hamilton–Jacobi approach of Mourrat and coauthors). Discussing this literature would help position the novelty of the proof strategy.

Given the size of the changes made during the rebuttal the revised paper would benefit from a rewriting. I nevertheless increased my score.

Thank you for your constructive follow-up and for increasing your score. We agree that the revised manuscript would benefit from a more cohesive presentation. We address the comparison first, since it also helps clarify the contribution.

2. Comparison with Yaakoubi et al. The red empirical markers in Figure 2(a) show that smoothed Lasso closely approximates pure Lasso in this experiment, whereas the red theoretical curves do not accurately predict even the smoothed estimator. You are right that the Hessian at zero is prescribed by Yaakoubi et al.’s original equations but their objective (\mathcal J), Eq. (4), also retains the full penalty through (\rho_\varepsilon(\mu)). In particular, their mean equation involves (\nabla\rho_\varepsilon(\mu)). Consequently, evaluating these equations does not amount to replacing the penalty by pure ridge. These are extrapolations outside their assumptions: although smooth, our smoothed Lasso is not uniformly strongly convex. Their quadratic-reduction argument is therefore not justified in this regime. We will clarify this distinction in Appendix I.2: the experiment illustrates a limitation of extrapolating their formulas, without contradicting their theorem.

1. Scope of the contribution. We explicitly acknowledge that our work builds on earlier surrogate approaches and Yaakoubi et al.’s treatment of non-Gaussian data. The contribution is to combine the Gaussian surrogate formulation characterictic to nonsmooth penalty problems with calibration to the original feature–label law, yielding the score description (x^\top\mu_+\alpha_Z). To our knowledge, neither the earlier Lasso results nor the smooth-penalty equations establish nor evoke this possible combination - if it is an extension, it was not trivial nor obvious. It was, nevertheless, necessary, since we show experimentally the limits of previous approaches. Although parts of the surrogate have familiar forms, establishing the appropriate calibration and its validity requires a completely new argument.

You are correct that Theorem 2 is proved on interior domains, where positive probes ensure strong convexity. However, Louart’s CLT for M-estimators (if that's the "existing result" you refer to) requires a quadratic regularizer and therefore cannot be applied to Lasso, whose penalty remains nonsmooth after adding the probes. We agree that the interior restriction leaves a gap in scope, although it suffices for our main result. We are working to establish the conditional CLT uniformly along the full calibrated interpolation path under Theorem 3’s assumptions, and expect the complete proof to be ready for the camera-ready version.

3. Hamilton–Jacobi interpolation. Thank you for suggesting these references. We agree that interpolation and Hamilton–Jacobi methods have substantial precedents, and will cite and discuss Guerra (2003) and Mourrat (2021). Our novelty claim concerns the particular cavity Hamilton–Jacobi construction connecting non-Gaussian ERM to its one-sample surrogate while preserving the nonsmooth penalty and transporting prediction moments. We will make this distinction explicit.

We appreciate your careful reading and constructive suggestions. If these clarifications resolve your remaining concerns, we would be grateful if you would consider a further increase in your score.

Reviewer_KSWMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a Gaussian-vector-driven surrogate for high-dimensional convex empirical risk minimization with non-Gaussian, potentially coordinate-dependent features. The surrogate preserves the original penalty and calibrates its coefficients using the actual joint feature–label distribution, rather than replacing that distribution by a covariance-matched Gaussian model.

The paper proves finite-dimensional existence and uniqueness of the population calibration. Conditional on a conjectured Gaussian approximation for fresh scores over training randomness, it establishes agreement between the ERM estimator and the surrogate’s normalized mean and prediction variance, with consequences for averaged prediction laws and classification error. The argument uses a cavity reduction and a Hamilton–Jacobi interpolation, including cutoff removal for uniformly strongly convex penalties and pure Lasso under projection anti-concentration. Simulations investigate support recovery and score distributions.

Critical strengths
  1. An important observation is that the surrogate retains information that covariance matching can lose. Keeping the actual feature–label law in the calibration is well motivated. The decomposition into a potentially non-Gaussian signal projection and a Gaussian training fluctuation provides a useful way to distinguish sources of non-Gaussian behavior. Preserving the nonsmooth penalty is a meaningful extension. The surrogate remains a convex optimization problem with the original regularizer. This permits sparse solutions and avoids replacing the penalty by a quadratic approximation that would erase thresholding behavior. This serves as one of the paper strong aspects.

  2. Another strength worth mentioning is that the conditional comparison involves {\em substantial technical work}. The scalar-response argument, finite-force change of measure, and gradient transport from a value residual address genuine difficulties. In particular, Appendix D avoids simply differentiating an uncontrolled asymptotic error. The pure-Lasso cutoff argument also requires more than the strong-convexity case: it combines a support gap, localization, curvature, and inactive-coordinate slack.

Critical weaknesses
  1. A major weakness - note that the principal ERM comparison depends on an unresolved central conjecture. Conjecture 1 is required uniformly over interior interpolation domains, for both full and cavity optimizers. It concerns conditional Gaussianity of the training-induced score fluctuation at a fixed fresh feature, with a deterministic averaged variance. This is a substantial probabilistic assertion, particularly for general nonsmooth penalties and dependent features. Poincaré concentration and quadratic-form concentration do not themselves establish it in the manuscript. The conditional theorem is valuable, but an unconditional result for a nontrivial nonsmooth subclass would considerably strengthen the contribution. The coordinate counterexample in Appendix G.4 usefully rules out a stronger formulation; it does not establish the proposed fresh-score formulation.

  2. Related to this - the experiments do not directly test the conjecture used in the proof.
    For example, a more informative experiment would hold several fresh features fixed, collect scores across independent training datasets, and compare their centered distributions and conditional variances with the proposed Gaussian laws. Testing positive interpolation probes would also connect the measurements to the actual conjecture.

Questions for authors

None

1 comment

Thank you for your comments. We address conditional Gaussianity and the proof strategy in A1 and A3, numerical validation of score and performance prediction for pure Lasso in A6, the synthetic-data assumption in A2, and the coordinate counterexample in A1.

Conditional Gaussianity and full/cavity coverage. Please see A1 in our reply to iqWV: the former Conjecture 1 is now Theorem 2, with its complete proof in Appendix C. The theorem uses a deterministic averaged prediction variance and holds uniformly on each fixed interior domain for full and unforced cavity optimizers.

A3. Mathematical contribution and Hamilton–Jacobi mechanism

We agree that our previous presentation may have been ambiguous. Section 2, “Main contributions and comparison with prior work,” now explicitly presents the main contributions:

  • A Hamilton–Jacobi equation under general assumptions.
  • Extension under general loss and ℓ1\ell_1 regularization to general models for (x,y).
  • A proved Gaussian approximation for fresh prediction scores.
  • A test-score law calibrated to the actual data distribution.

Several ingredients already appear separately in the literature. Yaakoubi et al. (ICML 2026) describe the test-score law as the convolution of the signal projection with Gaussian noise, under assumptions including smooth, uniformly strongly convex regularization. Surrogates of the form W∗W_\ast appear in Loureiro et al. (2021) and related work for specific models, notably Gaussian covariates and Gaussian mixtures. Our central theoretical advance is to unify these approaches through a Hamilton–Jacobi interpolation, retaining the actual feature–label law and the original penalty while allowing pure ℓ1\ell_1 regularization. The proof transports prediction moments without differentiating the penalty (Section 4, Theorem 7), with gradient transport proved in Appendix E, Lemmas E.2 and E.3. We refer to Section 2 for the precise scope and detailed comparisons.

Direct comparison on diamonds. Figure 2, Section 3.7, compares these two approaches with our predictions on the diamonds-based price-classification problem. Our predictions match the empirical score laws and classification errors, while the displayed competing predictions miss the observed behavior. Yaakoubi et al. do not cover exact nonsmooth Lasso; their smooth-penalty equations are extrapolated here beyond their strong-convexity assumptions. Loureiro et al. allow Lasso, but the fitted Gaussian mixtures change the data law, and even ten components per class leave a discrepancy. Richer mixtures can improve the approximation, but their fixed-cluster theory does not itself justify replacing the original distribution by an increasingly detailed mixture. A6 in our reply to jWL1 gives the comparison details.

A nonsmooth application. A4 in our reply to JfYx gives sufficient conditions for logistic regression with pure Lasso to satisfy the population conditions (22), as established by Proposition 4. Separately, A2 in our reply to iqWV explains the dimension-independent Poincaré bound and the purpose of the synthetic experiment. For the real-data validation and comparisons, please see A6 in our reply to jWL1.

Fixed features, conditional variance, and positive probes. Figure 1(a,b) tests Gaussianity with fixed fresh features, including positive-probe tests of Theorem 2. The comparison uses a common trace-based variance; Figure 3 checks it against the population prediction. Figure 2(b,c) instead pools independent training/fresh-feature pairs to test the calibrated score law. See A1 in our reply to iqWV and Appendices I.1–I.2.

Reviewer_iqWVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a one-sample surrogate for the high-dimensional convex ERM with non-Gaussian features and non-smooth penalties, while the surrogate only depends on single Gaussian random vector. The paper has several contributions with some proof. The author proves the calibration of the surrogates exists and is unique. The author also shows that the original ERM and the proposed surrogate are asymptotically same under the conjecture 1 using Hamilton–Jacobi approach. This paper also shows some corollaries and practical illustrations with the sampling model and numerical protocol.

Critical strengths

The paper successfully proves the theorems and corollaries under their assumptions and conjecture for this important hard question of ERM.

Critical weaknesses
  1. All the asymptotic theorems and corollaries rely on conjecture 1, which is never proved in the paper.
  2. The experiments do not test what the theory proves. Theorem 2 is never checked directly and Fig. 2(b) tests a different law which is not same as conjecture 1.
Questions for authors
  1. How did you propose the conjecture 1? To make it more sound, could you try to establish it under a more restrictive but meaningful setting, such as smooth strongly convex penalties with Gaussian mixture features?
  2. Can you show direct numerical evidence for Conjecture 1 at fixed fresh x, across n?
  3. Does your experimental design satisfy Assumption 2, and with what constant?
3 comments

Thank you for your comments. A1 explains the proof of the former Conjecture 1 and its numerical tests. A2 addresses the Poincaré constant. A6 in our reply to jWL1 gives the validation of prediction and classification formulas and comparisons with prior work.

A1. Conjecture 1 is now proved

Motivation. Conjecture 1 was motivated by Louart (2026), A Central Limit Theorem for Regularized M-Estimators, Theorem 2.1. That result establishes a Gaussian limit for standardized, nondegenerate deterministic projections with quadratic regularization, under Poincaré concentration and smoothness and curvature assumptions. We sought to extend this Gaussian-approximation principle to nonsmooth penalties. With pure Lasso, coordinate projections can retain an atom at zero, so Gaussianity cannot generally hold in every deterministic direction. We therefore formulated the extension for prediction along an independent fresh feature xx: the score law is conditioned on xx, and the approximation error is averaged over its distribution, as in (21).

We have now proved Conjecture 1; it appears as Theorem 2 in the revised paper, with the complete proof in Appendix C. The proof uses the scalar Stein-equation bounds from Chatterjee (2008, Lemma 4.2), formulated as a Wasserstein comparison in Lemma C.3. We verify this criterion using concentration of the optimizer's quadratic forms (Lemma C.1) and a Stein kernel for the fresh test feature x. The kernel exists by the Poincaré inequality, following Courtade, Fathi and Pananjady (2019, Theorem 2.4); this construction is stated separately in Lemma C.2.

Direct conditional Wasserstein measurements. Figure 1(a) directly tests the Gaussian comparison (21), including pure Lasso. We measure the score law over independent training samples with each fresh feature held fixed, then average the Wasserstein distances over features. As dimension grows, the errors decrease toward the finite-sampling error obtained from Gaussian scores. Appendix I.1 gives the protocol.

Positive probes and endpoints. Interior grid points test Theorem 2 with positive probes. The endpoints test the surrogate (t=0t=0) and original ERM (t=1t=1). Figure 1(a) uses estimated moments to assess Gaussianity; Figure 2(b,c), Section 3.7 and Appendix I.2, tests the population-calibrated law in Corollary 5.

Fixed-feature histograms. Figure 1(b) shows score distributions xTWx^TW and e1TWe_1^T W across repeated training samples with the test data xx held fixed. The scores are approximately Gaussian for both pure ℓ2\ell_2 and pure ℓ1\ell_1 regularization, while e1TWe_1^T W is non-Gaussian with lasso regularization ( Appendix H.4, Proposition H.5, gives a rigorous coordinate counterexample). A heavy-tailed example outside the Poincaré assumptions shows departures from Gaussianity. Appendix I.1 gives the protocol.

A2. Poincaré constants and generated data

Appendix K, “Simple criteria for a Poincaré inequality,” retains general criteria for obtaining dimension-independent bounds. In particular, the class of laws satisfying a Poincaré inequality with constant CC is stable under:

  • Tensorization: if X1,…,XnX_1,\ldots,X_n are independent copies of XX, each with constant CC, then (X1,…,Xn)(X_1,\ldots,X_n) has the same bound CC.
  • 1-Lipschitz transformations: if Φ:Rp→Rp\Phi:\mathbb R^p\to\mathbb R^p is 1-Lipschitz for the Euclidean norm, then Φ(X)\Phi(X) also has the bound CC.

These properties cover a broad class of generated data. An LL-Lipschitz generator applied to standard Gaussian latent variables gives a Poincaré constant at most L2L^2, independent of dimension when LL is uniformly bounded. This includes the Lipschitz Gaussian-generator framework for GAN-generated data studied by Seddik et al. (ICML 2020); the appendix derives the Poincaré bound from the stability properties above. The same argument applies to the joint pair (x,y)(x,y) when it is generated this way.

For the synthetic model in Appendix I.1, fixed-size blocks, tensorization and the uniformly Lipschitz response give a Poincaré bound independent of n,pn,p, as required by the theory. The elaborate block-mixture distribution was chosen to reveal the effect of dependence on score laws. Sharpening its numerical constant is a separate problem, outside the purpose of these direct Gaussian-approximation experiments.

Thank you for your response and paper revision. After briefly checking the correctness of the new proof of Theorem 2, it resolves my earlier concern regarding the reliance on the previous Conjecture 1. And the new experiments and author's replies are generally convincing and provide reasonable comparisons with existing work. I will increase my score.

Thank you for your thoughtful feedbacks and for raising your score. Your comments have helped us strengthen our work, and we truly appreciate your support.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Lean verification of Conditional CLT

We invite reviewers to consult the conditional_gaussian folder in our anonymous repository which contains the complete Lean formalization of the now proven Conjecture 1 (Theorem 2 proven in Appendix C proof), without sorry or custom axioms, together with a mapping to the manuscript results and instructions for independent verification:

https://anonymous.4open.science/r/ERM_non_smooth_penalty-F4F0/

It also contains the python scripts used to plot the figures we added

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.