Parametric Rates for Nonparametric Conditional-Moment Bilevel Optimization

We study functional bilevel optimization with conditional moments, where plug-in hypergradients can inherit first-order errors from nonparametrically learned conditional expectations. We introduce OBiGrad, an orthogonalized hypergradient estimator whose bias is a product of nuisance errors. Under our regression and complexity conditions, projected gradient descent with the direct plug-in hypergradient yields expected stationarity error $O(T^{-1/2}+n^{-1/2}+n^{-\nu /b})$ for the best iterate, whereas OBiGrad achieves $O(T^{-1/2}+n^{-1/2}+n^{-2\nu /b})$, where $b=\nu (2+p)+\gamma$ combines the approximation and complexity exponents. Consequently, when $\nu (2-p)\ge \gamma$, OBiGrad attains the parametric statistical rate $O(T^{-1/2}+n^{-1/2})$, improving the sufficient sample-size bound relative to the direct plug-in bound for a fixed iteration budget. We illustrate these gains on instrumental-variable and fitted-Q-regression problems, examining stationary-point estimation, returned population stationarity, and regularization sensitivity.