Multiple Descent of Generalization Curve for Optimally Regularized Ridge Regression

It is known that optimally regularized ridge regression can exhibit non-monotone dependence of the generalization risk on the sample size. We show that this behavior can be arbitrarily intricate in finite dimensions: for every $d\ge 2$, there exists a random design linear model in ${\mathbb{R}}^d$ whose generalization risk, viewed as a function of the sample size, exhibits at least $d-1$ local maxima.