SatRAP: Adaptive Residual Retrieval for Augmenting Satellite Embeddings

Recent advances in satellite foundation models have produced a growing family of general-purpose embeddings for Earth-observation tasks. Once a downstream predictor is fitted, however, its training residuals no longer contribute to new predictions. We find that these residuals are strongly structured: geographically close observations with similar satellite embeddings tend to exhibit compatible prediction errors. Therefore, we introduce Satellite Residual-Augmented Prediction (SatRAP), a residual retrieval framework that augments embedding-based prediction using a geographically indexed representation–residual memory. SatRAP restricts retrieval by geographic proximity, measures compatibility in the embedding space, and adaptively aggregates the retrieved residuals. Under squared loss, the standard conditional-expectation identity gives the conditional mean residual given location and embedding as the optimal correction. We further derive an upper bound on the error of approximating this correction from finitely many retrieved residuals, in terms of geographic radius, embedding mismatch, and effective retrieval size. Experiments across satellite embeddings, downstream predictors, and global classification and regression tasks show that our approach achieves the best overall performance among the compared methods, improving mean classification accuracy by 4.51 percentage points and mean regression $R^2$ by 0.131 over the base predictors. These results establish structured residual retrieval as an effective complement to embedding-based prediction.