acceptodds
Withdrawn from ICLR 2027

SatRAP: Adaptive Residual Retrieval for Augmenting Satellite Embeddings

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advances in satellite foundation models have produced a growing family of general-purpose embeddings for Earth-observation tasks. Once a downstream predictor is fitted, however, its training residuals no longer contribute to new predictions. We find that these residuals are strongly structured: geographically close observations with similar satellite embeddings tend to exhibit compatible prediction errors. Therefore, we introduce Satellite Residual-Augmented Prediction (SatRAP), a residual retrieval framework that augments embedding-based prediction using a geographically indexed representation–residual memory. SatRAP restricts retrieval by geographic proximity, measures compatibility in the embedding space, and adaptively aggregates the retrieved residuals. Under squared loss, the standard conditional-expectation identity gives the conditional mean residual given location and embedding as the optimal correction. We further derive an upper bound on the error of approximating this correction from finitely many retrieved residuals, in terms of geographic radius, embedding mismatch, and effective retrieval size. Experiments across satellite embeddings, downstream predictors, and global classification and regression tasks show that our approach achieves the best overall performance among the compared methods, improving mean classification accuracy by 4.51 percentage points and mean regression by 0.131 over the base predictors. These results establish structured residual retrieval as an effective complement to embedding-based prediction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Earth observation, remote sensing, satellite embeddings, satellite foundation models, geospatial AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_frgCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SatRAP, a post-hoc correction for predictors trained on frozen satellite embeddings. The core idea is that training residuals are stored with location and embedding. At test time the M=32 geographically nearest entries are reweighted by exp(-dist/h) * exp(beta*cos sim), averaged, shrunk by a learned sigmoid and added to base prediction. The motivation is that, users would tend to discard the residuals after they used RSFMs or their embeddings for donstream tasks, but authors cleverly spotted this issue. Then, there's also SatRAP+, that adds two small MLP affinity terms on top of the analytic score. The authors did a relatively large scale evaluation: the paper has evaluated on 7140 global points, 10 inputs (7 embeddings + 3 raw sensors), 6 tasks (3 LULC, canopy, AGB, DEM), 6 predictors, 5 IID folds, against 16 location encoders. Results also look pretty good at first glance: +4.5pp OA and +0.13 R2 over base predictor, best overall vs baselines.

Critical strengths
  1. The empirical observation itself is useful, i.e., residuals of embedding-based predictors keep clear geospatial semantic structure (Fig 2, 9, 10), Moran’s I of residuals predicts gain almost perfectly and the correction removes it. This is a very clean, reusable finding regardless of the method.

  2. One of the biggest strengths, i think, is that this method is cheap and needs no retraining, which is what practitioners actually want.

Critical weaknesses
  1. The method borrows residuals from nearby labelled points and the targets are spatially autocorrelated, so every test point has labelled neighbours by construction. This measures interpolation inside the labelled region, not generalisation. I would highly recommend authors add spatial block CV tests. For an EO paper this is the central question.
  2. I think it's worth adding more traditional baseline methods, say, regression kriging / residual GP (not cited either, and it is the same idea). for now, I can’t tell what the retrieval adds beyond “use location”.
Questions for authors
  1. What is the residual head for the location-encoder baselines, i assume linear? if so, have the authors considered results with a nonlinear head, and with coordinates / e(s) concatenated into the base predictor’s input?
  2. Which of the 10 inputs saw the target products (or lat/lon) during pretraining? If im not mistaken, AEF has LULC map/AGB/etc. in their training pipeline (similar cases for terramind), so there might be some data leakage issue in your evaluation pipeline. so i wonder how much would this affect your results/conclusions?
  3. Can authors please report training details and cost for SatRAP+ and Full-Param? and was Full-Param tuned with the same effort as SatRAP+?
  4. Does SatRAP+ use the same shrinkage eta as SatRAP at prediction time? Eq 9 doesn’t have it.

Overall, this is a neat and good work. I will consider raising my score if authors address my concerns.

Reviewer_iMQPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes SatRAP, a residual retrieval framework that improves downstream predictions from pretrained satellite embeddings by exploiting geographically and semantically structured prediction errors. The method first trains a downstream predictor on satellite embeddings and constructs a memory of cross-fitted residuals, associated with their geographic coordinates and embeddings. During inference, SatRAP retrieves geographically nearby samples, weights their residuals using geographic distance and embedding similarity. Experiments are done on 6 downstream tasks and SatRAP provides consistent improvements over the SoTA location encoders.

Critical strengths
  1. The motivation of the paper is meaningful. Satellite embeddings capture rich geographic information, but downstream predictors fail to fully exploit it, leaving geographically and semantically structured prediction errors.
  2. The paper proposes SatRAP, which retrieves residuals from geographically nearby and semantically similar observations to adaptively correct predictions, along with a neural variant, SatRAP+.
Critical weaknesses
  1. The method itself is relatively simple (storing the residuals from a predictor and then retrieving them and performing an adaptive pooling at inference) but the presentation especially the method section makes it difficult to follow the contributions of the paper. The method section feels like a word salad with numerous concepts introduced without a clean hierarchy. I think the paper should improve its readability.

  2. The main downside of the method is that it requires a fitted downstream predictor. It is not clear how SatRAP will do when labeled samples are limited or when there is substantial geographic domain gap between the training distribution of the predictor and the testing distribution. I think more experiments are needed on both of these front. For instance, training the predictor with progressively increasing number of samples. The other experiment can be training the predictor on specific geographic regions such as North America and then testing on Europe or Africa. I am sure the nature and distribution of the residuals will be different geographically.

  3. SatRAP explicitly leverages both satellite embeddings and geographic coordinates for residual retrieval, whereas the competing location-encoder baselines, including SatCLIP, RANGE, and TTE, use only geographic representations for their residual correction. This makes it difficult to establish whether SatRAP's advantage comes from its retrieval algorithm or simply from using additional query-specific satellite information.

  4. The paper demonstrates that satellite embeddings already encode substantial geographic information, yet proposes additional location-based residual correction. Why not train the predictor with both satellite embeddings and location embeddings?

  5. The points made above (sample budget, geographic domain gap) combined with limited experimental benchmarks makes it difficult to assess whether the method truly generalizes well in all real life geospatial scenarios.

  6. Label and observation noise can affect the distribution of residuals. A lot of geospatial datasets contain errors in GPS coordinates. How robust is the method to label and observation noise?

  7. Although SatRAP achieves strong overall results, the differences against RANGE+ and other strong location encoders are minimal especially considering SatRAP uses satellite embeddings for residual computation.

Questions for authors
  1. Across tasks, do the residuals change geographically or is there a geographic bias towards the model errors irrespective of the task. For instance, if the predictor did not see data from a particular region across the tasks, will the distribution of the residuals for say landcover prediction and canopy height mapping stay similar?
Reviewer_4GN3OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SatRAP, which corrects an existing predictor by retrieving historical residuals using geographic proximity and satellite-embedding similarity. SatRAP+ further learns refinements to the retrieval weights. The problem is practically relevant, the method is intuitive, and the evaluation covers multiple inputs, predictors, and tasks. The ablations also support the complementary roles of geographic and embedding information.

Critical strengths
  1. SatRAP provides a practical way to improve existing satellite-embedding predictors by reusing historical residuals, without retraining the satellite encoder. Combining geographic proximity with embedding similarity is intuitive and accommodates different downstream predictors. This is consequential because the method could extend the usefulness of existing embedding products through a relatively simple prediction-stage addition.

  2. The evaluation covers 10 satellite inputs, six downstream tasks, and six predictor families, providing evidence that the benefits are not confined to one model or task. The geography-only, embedding-only, and shuffled-similarity ablations further support the complementary value of geographic and semantic matching. This combination of broad evaluation and targeted controls is the strongest empirical support for the proposed design.

Critical weaknesses
  1. The current comparisons focus primarily on location encoders, while the central operation is prediction correction using neighboring residuals. Please include residual kriging, simple residual kernel smoothing, and direct label prediction using the same neighbors and weights. A predictor jointly receiving satellite embeddings and location features would also be informative. These comparisons would help determine whether the gains mainly come from additional geographic information, neighboring labels, or the specific residual-retrieval design.

  2. The theory defines residuals using a single fixed predictor. In the experiments, however, the stored residuals come from different inner cross-fitting models, while test predictions use a model refitted on the full training set. The residual memory therefore does not exactly represent the errors of the final predictor. Please explain how this discrepancy affects the analysis, introduce a model-discrepancy term if necessary, or provide an empirical check showing that its impact is small. This concerns the connection between theory and implementation and does not, by itself, establish an error in the derivation.

  3. Figure 1 shows that removing subspaces identified by linear geolocation probes affects downstream performance similarly to random removal. However, geographic information may remain available through redundant features or nonlinear representations, as also suggested by the random-forest probes in the appendix. Please acknowledge these possible explanations and limit the conclusion to what the intervention directly tests. A brief discussion and more qualified wording would make the interpretation more precise.

  4. The current ablations mainly examine geographic retrieval, embedding similarity, and neural weighting. They do not fully isolate the benefits of adaptive bandwidth and residual shrinkage. Please compare fixed and adaptive bandwidths, together with no shrinkage, constant shrinkage, and adaptive shrinkage, while holding other settings unchanged. This would clarify how much improvement comes from local residual averaging and how much comes from the additional adaptive components.

  5. Please specify the number of inner cross-fitting folds, the hyperparameter-selection procedure, the inputs and architectures of the baseline residual heads, and the network dimensions and optimization settings for SatRAP+. Equation (9) does not include the shrinkage function used in Eqs. (5) and (24), so please clarify whether SatRAP+ retains this module and how the components are trained. Providing code, sampling coordinates, data versions, and splits would support independent reproduction and fair comparison.

Questions for authors

Please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.