Retrieval Is Density Estimation: Calibrated, Invariant, and Selective Retrieval-Augmented Forecasting

Zero-shot forecasts from time-series foundation models such as Chronos and MOIRAI show severe performance drops when test distributions diverge from the pretraining data. While retrieval augmentation is intended to bridge this out-of-distribution generalization gap by supplying external memory, existing methods often fail. Because existing methods misalign unnormalized series via Euclidean matching, collapse predictive uncertainity through latent pooling, and unconditioned retrieval forces neighbor injection even when candidate trajectories diverge. Across standard benchmarks, unconditioned retrieval degrades baseline accuracy on 41–50% of test windows. We frame retrieval-augmented forecasting as non-parametric conditional density estimation over a scale-translation quotient manifold $\mathcal{X}/\mathcal{G}$. By mapping lookback windows onto invariant shape orbits on $\mathcal{X}/\mathcal{G}$, our approach contracts the covering space ($R^2=0.9845$, scaling exponent $\alpha =0.2199$) and thereby eliminates the need for post-hoc affine alignment. To aggregate forecast distributions, we use weighted 2-Wasserstein barycenters in quantile space. When local candidate scattering indicates negative expected utility, an instance-adaptive controller selectively abstains from retrieval. Under abrupt distribution shifts, a non-exchangeable split-conformal step expands prediction intervals to restore nominal coverage. Across four ETT datasets and three foundation models (Chronos-Bolt, Moirai-2.0, TimesFM-2.5), Calibrated Invariance-Aware Selective Retrieval (CAVIR) lowers CRPS by up to 2.15%, achieves 80.1% coverage under macroeconomic shocks, and stays within 0.85% of baseline error under strict deduplication checks.