acceptodds
Withdrawn from ICLR 2027

When Interpolation Is Loss-Blind: Quantile Regression at the Interpolation Boundary

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Quantile regression turns a flexible predictor into prediction intervals by fitting several conditional quantiles, and the intervals are informative only when these fits differ across levels. Models that can fit every training response exactly are usually regularized by a ridge penalty, or weight decay, which makes the penalized coefficients unique. In this article, we show that uniqueness does not separate the levels on an explicit range of penalties. For fixed features whose training feature matrix has full row rank, we derive for each quantile level a closed-form penalty boundary, computed with one linear solve, at or below which the fit of that level equals the minimum-norm interpolant, which does not depend on the level. Theoretical results show that two distinct levels share a fit exactly when the common penalty is at or below both boundaries, with or without an unpenalized intercept, and that the same certificate applies to the linear quantile heads of a network at any local minimum of training at which the learned feature matrix has full row rank. With Gaussian features and noise and a feature count growing faster than the sample size, the boundary shrinks at a fixed response scale, and extinct fits have coverage converging to one half. For bounded responses and features fixed before sampling, validation above the boundary over a fixed penalty grid controls the excess pinball risk up to the approximation error of the features. Through simulations, we confirm the predicted coverage transition at the boundary. On four regression data sets and a simulated model, the weight decay used to train a network lies below the boundaries computed from its learned features in 499 of 500 splits, and the five trained quantile heads have mean test coverage between and on every data set, at nominal levels from to . Refitting the heads with a validated weight decay far above the boundary moves the pooled mean coverage at the levels and to and .

Primary area:
learning theory
Keywords:
Quantile regression, loss-blind interpolation, ridge regularization, overparameterized regression, penalty selection, computational efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RH18OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper derives an explicit penalty threshold below which ridge-regularized quantile regression collapses to the same minimum-norm interpolant at every quantile level, studies that threshold under random designs, and applies the resulting certificate to neural-network output heads. The formula itself is clean and useful. My reservations are that the step beyond prior work is modest, the probabilistic model is a long way from the network experiments, and the theory does not, in the end, explain what the experiments show.

Critical strengths

The main result is clean and exact: a closed-form penalty boundary, computable with a single linear solve, below which the ridge-penalized quantile fit collapses to the minimum-norm interpolant (Theorems 1–2), and the certificate extends to network output heads at full-rank local minima (Corollary 2). The fixed-design argument is sound.

Critical weaknesses

The writing seems heavily rely on AI and the Authors do not carefully polish it. So I understand nothing from the abstract. There are also illusions in the reference, for example,riedlander & Tseng, Exact Regularization of Convex Programs, SIAM J. Optim. 18(4):1326–1350 is a paper from 2007 not 2008 of the paper. In summary, the paper introduces too many newly coined labels, giving the prose an artificial, AI-like style—for example, “extinction” and “loss-blind” (pp. 2–3), “median-type predictor” (p. 6), “common-penalty level certificates” (p. 66), and “face atlases” and “stable tiles” (pp. 70–71), requiring the AI-assisted reading for readers which might be suitable in the future (2-3 years) but not now.

Questions for authors
  1. The theory and the network experiments are only loosely connected. The main probabilistic analysis is for independent Gaussian features with a linear response model (pp. 5–6), and the risk guarantees assume the features are fixed before sampling (pp. 6–7). The network experiments, by contrast, use correlated features that are learned from the training responses. Corollary 2 (p. 5) does give a certificate at a local minimum with a full-row-rank feature matrix, where the output heads solve conditional convex problems, so it applies to whatever features the network has learned. But it says nothing about feature learning or training dynamics, and the networks in practice never reach an exact minimizer (p. 8). The authors do acknowledge these gaps and provide supplementary extensions, but the connection stays conditional. I would like to see evidence that the random-design analysis actually explains the observed network behavior, rather than merely that the deterministic certificate applies after training.

  2. Novelty is thin. The paper is upfront about the precedents (pp. 1–2): positive exact-regularization intervals [1], kernel quantile-regression solution paths [2,3], and minimum-norm interpolation with coverage near one half [4]. The explicit threshold and the characterization of cross-level coincidence refine this picture, but the central threshold is a direct consequence of the KKT conditions (p. 3; proof on p. 13), and I am not convinced these refinements change our understanding of quantile regression in a substantial way.

  3. The boundary does not translate into calibration guidance. Coverage can stay near one half well above the boundary, and removing exact extinction does not remove severe miscalibration (p. 8; Tables S.21–S.22, p. 97). Consolidating the extinct candidates preserves the full-grid validation selection, and it avoids 55% of the solver fits — but only 8.4% of the solver time, since the time is spent at penalties above the boundary (p. 8). The supplementary reinvestment experiments tackle this, but the equal-elapsed-time risk improvements come to about 0.41% and 0.19% (Table S.2, p. 80). As it stands, the practical value of locating the exact boundary is unclear.

  4. The abstract needs a rewrite. I found the abstract hard to follow (p. 1): it races through several theoretical regimes, a conditional network result, and a string of coverage numbers, with little intuition for why the question matters. The main insight is buried. The abstract should state the problem and the contribution plainly before getting into qualifications and numerics.

The fixed-design argument itself looks sound. My concerns are the limited advance over prior work, the idealized probabilistic model, and the absence of a convincing explanation of the network experiments.

References

[1] Michael P. Friedlander and Paul Tseng. Exact Regularization of Convex Programs. SIAM Journal on Optimization, 18(4):1326–1350, 2007.

[2] Youjuan Li, Yufeng Liu, and Ji Zhu. Quantile Regression in Reproducing Kernel Hilbert Spaces. Journal of the American Statistical Association, 102(477):255–268, 2007.

[3] Saharon Rosset. Bi-Level Path Following for Cross Validated Solution of Kernel Quantile Regression. Journal of Machine Learning Research, 10(86):2473–2505, 2009.

[4] Yu Bai, Song Mei, Huan Wang, and Caiming Xiong. Understanding the Under-Coverage Bias in Uncertainty Estimation. Advances in Neural Information Processing Systems, 34:18307–18319, 2021.

1 comment

In my previous comment, I say ’ There are also illusions in the reference, for example,riedlander & Tseng, Exact Regularization of Convex Programs, SIAM J. Optim. 18(4):1326–1350 is a paper from 2007 not 2008 of the paper. ‘

After investigation, I think this error (wrong year alone) could be caused by typo or the mistake of citation tools (such as google citation), not `illusions'

Reviewer_f7SwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies ridge-penalized linear quantile regression in overparameterized regime. It shows that sufficiently small positive penalties yield the same minimum-norm interpolating solution for different quantile levels, making the fitted predictor independent of the quantile level. The paper characterizes the exact penalty threshold for this phenomenon. Under Gaussian random designs, it derives asymptotic results for the threshold and establishes conditions under which marginal coverage converges to 1/2 independent of the target quantile level. Under additional assumptions, it also provides bounds on coverage error for properly regularized estimators. Experiments illustrate these theoretical findings and examine how regularization affects calibration, including when quantile regression is applied to learned neural network features.

Critical strengths
  1. The paper provides an explicit characterization of the penalty required to escape exact interpolation. The fact that predicted quantile levels can remain identical for small penalty is a useful observation: adding ridge regularization does not automatically prevent quantile collapse.

  2. The Gaussian analysis quantifies the scale of this threshold and connects the collapsed solution to marginal coverage near 1/2. These results provide more statistical insight than the deterministic characterization alone.

Critical weaknesses
  1. The depth of the central deterministic contribution appears limited. The broad message is that insufficient regularization can cause overparameterized quantile regression to lose its dependence on the quantile level. Although the exact positive-penalty threshold is a useful refinement, it follows from a short KKT argument using the subgradient interval of the pinball loss at zero.

  2. The threshold offers limited guidance for choosing a statistically effective penalty. It identifies when exact collapse ends, but does not determine when the fitted quantiles become accurate or calibrated. Indeed, the experiments show that coverage can remain near 1/2 at penalties many times larger than the threshold. It remains unclear how knowledge of the threshold improves practical penalty selection beyond diagnosing collapse.

Questions for authors
  1. Can the computed threshold guide how far above it the penalty should be chosen to obtain useful calibration? What practical guidance does it offer?

  2. Can the authors more clearly separate the benefits of exact head refitting, stronger head regularization, and using the boundary during validation? In particular, what additional predictive benefit comes from knowing the boundary?

  3. Theorem 6 relies on several assumptions. Could the authors provide a simple, concrete data-generating example satisfying all of them, explicitly verify the assumptions and derive the resulting coverage-error bound? This would help readers understand when the theorem applies and whether its bound is informative at realistic sample sizes.

Reviewer_r6ZaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies ridge-regularized quantile regression and prove that small ridge penalties (weight decay) causes different quantile levels to collapse into a shared, uninformative minimum-norm interpolant. It derives a closed-form penalty boundary for this collapse, below which distinct quantities yield identical fits, results that also apply to neural network linear quantile heads. Empirically, standard small weight decay lies below this boundary and produces biased prediction coverage, while stronger validated regularization restores accurate quantile interval coverage.

Critical strengths
  1. The extinction boundary of Theorem 1 is a major contribution of this manuscript. As it provides an explicit proof for collapse, at each quantile leve. with a converse that rules out the equality of distinct quantile fits outside the shared extinction region.

  2. The establishment of extinction boundary only requires mild assumptions. The extension to neural representation gives a useful connection between linear quantile theory and network learning.

Critical weaknesses
  1. The main asymptotic results require two assumptions: 1) independent Gaussian features, and 2) sample size satisfy n/p→0n/p \to 0. This paper did not examine common deep learning regularization method such as early stopping.

  2. The neural network only uses one structure, i.e., 128 training observations, a single optimizer, and no validation across different experimental setups. More setups, across different structures, sample sizes, and optimizers are required to make this paper more convincing.

Questions for authors
  1. Under high-dimensional scenario, the number of features is greater than that of observations. Combining independent Gaussian design, full-row-rank condition satisfies with probability one whenever p≥np \ge n. Thus, the condition is well justified for Gaussian analysis of this paper. A network with millions of parameters and a 256-dimensional representation cannot have a full-row-rank feature matrix for 10,000 observations. Even when p≥np\ge n, repeated inputs, redundant features, or inactive units can reduce rank. The paper’s width of 256 with 128 training observations makes the condition possible. However, a matrix can have full row rank while XX⊤XX^\top is nearly singular. Since the boundary uses (XX⊤)−1(XX^\top)^{-1}, poor finite-sample performance of straightforwardly taking inverse will make its computation sensitive to perturbations. Requiring the regularity conditions for smallest singular values might be more useful.

  2. How would the extinction boundary change if the ridge penalty were replaced by an ℓ1\ell_1 (lasso) penalty? More generally, consider a neural network that jointly estimates the conditional quantiles at levels 1/4, 1/2, and 3/4. Such a model may incorporate additional penalties to prevent quantile crossing or constraints to enforce monotonicity across quantile levels. Can the proposed framework accommodate these more general forms of regularization, particularly those that couple different quantile heads, and characterize their effects on the extinction boundary?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.