acceptodds
Withdrawn from ICLR 2027

Convex Polynomial Regularization for Learning Representations with Degree-Ordered Power Growth

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Learning effective representations requires inducing structured inductive biases over function spaces, where polynomial degree serves as a natural hierarchy of structural complexity. While linear degree weighting enforces this ordering, standard degree-dependent power penalties inadvertently invert capacity control near zero. This study resolve this by introducing two shape-corrected regularizers that preserve global degree monotonicity including a linear-power penalty (LiP) and a shifted-power penalty (ShiP). These formulations directly constrain the learned coordinate representations within coercive, strictly convex optimization landscapes, guaranteeing unique global solutions and admitting exact coordinate-wise minimization. Across 09 regression benchmarks, regularizing representations via ShiP achieves competitive predictive performance that rivals non-parametric Gaussian process baselines. By bridging degree-ordered complexity constraints with non-linear power penalties, this framework ensures that expressive feature representations remain well-conditioned, interpretable, and computationally tractable without distorting the underlying capacity hierarchy.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
convex polynomial regularization, convex optimization, polynomial degree, learning representation, feature learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8hPPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes two regularizers for polynomial regression. They give larger penalties to higher-degree terms at the same coefficient magnitude while keeping nonlinear penalty growth. The paper provides optimization results and evaluates the methods on nine univariate regression datasets.

Critical strengths
  1. The paper explains the problem with pure-power penalties clearly and proposes two simple fixes. The resulting models remain explicit and easy to inspect.

  2. The experiments include several standard baselines and statistical tests. The authors clearly acknowledge that the proposed methods do not show a consistent prediction advantage.

Critical weaknesses
  1. The practical benefit over degree-weighted lasso is still unclear. That simpler method already penalizes higher-degree terms more, and the experiments do not show a consistent improvement over it. A concrete example showing when nonlinear penalty growth is useful would make the contribution more convincing.

  2. The experiments mainly evaluate prediction error, while the motivation also emphasizes model structure and interpretability. The reported coefficient counts do not clearly show whether the proposed methods produce simpler or more stable models than standard regularizers. A direct comparison of model complexity and coefficient stability would better support these benefits.

Questions for authors

Refer to the weaknesses above.

Reviewer_UFwpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how to regularize polynomial regression, in which a prediction is constructed from terms of increasing degree, such as linear, quadratic, and cubic terms. Its goal is to make higher-degree terms more expensive than lower-degree terms whenever their coefficients have the same magnitude. It identifies a problem with using degree-dependent powers as penalties: when coefficients are small, raising their magnitudes to higher powers makes the penalty smaller, reversing the intended preference for simpler terms. They propose two corrections: LiP and ShiP. The paper establishes conditions under which these penalties preserve the desired degree ordering. It also proves that the resulting regression problems have unique solutions, including when the design matrix contains redundant columns, and derives coordinate-wise optimization updates. These results concern coefficients in a fixed polynomial basis with a specified scaling convention.

Critical strengths
  1. The paper defines a clear design problem.
  2. The central claims are sound.
  3. The interpretation of the results is correct.
Critical weaknesses
  1. The practical value of the additional nonlinear penalty structure remains unclear. This paper establishes that the proposed combination of properties is achievable, but provides limited evidence explaining when that combination is useful.
  2. The comparisons do not fully isolate the contribution of higher-power growth. First, ShiP changes both the nonlinear penalty shape and the relative degree weights near zero. This paper’s expansion shows that its initial shrinkage depends on an additional degree-dependent factor, whereas the degree-weighted-lasso baseline uses weights proportional only to degree. Second, a degree-weighted elastic-net baseline would combine degree ordering, sparsity, and a quadratic penalty that helps ensure a unique solution. This paper includes ordinary elastic net and degree-weighted lasso separately, but not this combined control.
  3. Some claims in the abstract are overstated. The framing should remain aligned with the demonstrated setting.
  4. Some presentation corrections should be made. The abstract’s statement about strictly convex optimization landscapes is broader than Theorem 2’s guarantee of convexity and a unique minimizer. In degenerate cases, uniqueness can hold without strict convexity of the full objective. Appendix A4 contains a reference error.
Questions for authors
  1. Could you identify a setting in which the nonlinear terms improve coefficient stability, recovery of known structure, control of oscillation, or another meaningful property beyond degree-weighted lasso?
  2. How does ShiP compare with a weighted-lasso baseline that matches its relative shrinkage near zero? Would a degree-weighted elastic net, or a linear-and-quadratic approximation to ShiP, retain most of the observed behavior?
Reviewer_3advOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes ways to penalize polynomial regression fits so that the penalty is larger for larger coefficient of higher degrees. The paper proposes two constructions, LiP and ShiP. They both work like a power penalty but introduce an additional term that preserves the ordering of smaller coefficients (a power penalty can penalize coefficients based on degree more but for coefficients smaller than 1 this can have the oppposite effect and reverse that ordering). Then the paper shows that the proposed penalization still leads to a well posed problem and proceeds to present experimental results.

Critical strengths

The paper effectively deals with the main challenge it set out to tackle.

Critical weaknesses

As the paper readily admits, there doesn't seem to be any significant empirical benefit to this. The paper has not identified regimes where this order-preserving nonlinear growth for coefficients could beneficial for the predictive task This also is quite apparent in the beginning of the paper where we are not provided with any motivation why this would be a worthwhile thing to pursue.

It's nice that one could come up with a reasonable penalty to fix this order reversal for small coefficients but the technical contribution is not substantial enough in my opinion to warrant acceptance; some empirical edge needs to also be established. The advantage doesn't have to be universal or even on the majority of instances. It's more about making a clean case for some regime where this kind of penalization becomes relevant. Even some synthetic/contrived example could be a good start though not sufficient in my view.

Questions for authors

see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.