Beyond Inlier Confidence: Learning Residual Quantiles for Correspondence Pruning

Learned two-view correspondence pruning commonly uses scalar inlier scores to guide geometric estimation. However, correspondences with similar probabilities of satisfying an inlier threshold can have different residual distributions. We propose Residual Quantile Regression (RQR), which augments inlier classification with conditional geometric residual quantiles. Using a pinball objective, RQR learns ordered quantiles of the log epipolar residual for retained correspondences, providing a compact description of residual magnitude and spread. These quantiles guide correspondence learning during training; in the inference-time realization, RQR-F, they condition features through gated updates, while their spreads inform weights for differentiable weighted eight-point estimation. This formulation makes residual statistics explicit in correspondence representations and geometric weighting. Experiments on relative pose and homography estimation demonstrate the effectiveness of residual quantile learning across pruning architectures and geometric tasks.