Adaptive Conformal Prediction via Learned Residual-Law Retrieval

Conformal prediction provides finite-sample marginal coverage, but constructing efficient prediction sets that adapt to query-dependent uncertainty remains challenging. Current approaches generally obtain local adaptivity either by localizing calibration or by learning a directly parameterized conditional uncertainty model. We introduce Factorized Conformal Prediction (FacCP), which factorizes learned local residual-law retrieval from global calibration. The retrieved law adapts prediction geometry to each query, while conformal calibration remains a single global scalar estimation problem. We show that accurate local geometry induces approximately pivotal scores, making one global quantile approximately appropriate across queries while preserving finite-sample marginal validity. We bound conditional coverage error through geometry approximation and global calibration error, and decompose geometry error into representation, localization, estimation, and geometry-class contributions. FacCP learns an uncertainty-relevant representation with an energy-score objective targeting the full conditional residual law, retrieves this law locally, and instantiates query-specific geometries; the same representation supports multiple set constructors without retraining. Controlled oracle experiments validate the predicted chain from residual-law recovery to geometry, pivotality, and conditional coverage, and show that even oracle local calibration can remain geometrically inefficient. Across high-dimensional robotic reachability tasks, FacCP maintains calibrated coverage and stable set efficiency across wide variations in model capacity and training data, while direct covariance prediction becomes more efficient only in higher-data, higher-capacity regimes.