- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Evaluating geospatial representations often requires repeated training and held-out evaluation using downstream predictors across many locations. This paper proposes CLOSER, which selects a small subset of the locations for each prediction tasks, aiming to reduce the evaluation time while preserving the ranking agreement while full-location evaluation. CLOSER is applied on four prediction tasks and shows a significant speedups and improved agreement with full-location rankings compared with uniform sampling.
Critical strengths
- The paper introduces an interesting and potentially important question in the light of recent development of large-scale geospatial pretraining models.
- Strong experimental results showing the substantial reduction in training and evaluation time.
- Well-written and clearly presented.
Critical weaknesses
- The current experiments do not fully support the claimed reusability of the selected location subsets. The evaluated pipelines mainly involves concatenations of a set of pretrained geospatial representations and other features, whole relative performance rankings may be easier to preserve on small subset. In practice, newly developed models may employ more complex and different architectures, with their perdictive performance more difficult to infer. It remains unclear whether the method can still reliably preserve rankings across such models.
- It would be useful to also report the Top-k selection accuracy instead of only ranking correlation, as that's often what researchers more interested in during model iteration. The comparison of the predictive performance between different pipelines is also important in understanding the robustness of the method (e.g., whether adding more features consistenlt leads to better predictive performance).
Questions for authors
- Can the authors include more diversed pipelines (e.g., geospatial representations learned using different pretraining architectures, rather than concatenation)?
- It appears that the set of candidate pipelines will affect the automatic budget selection and resulting final selected location subset. If there's a new pipeline, how to ensure that the pre-selected subset still applies?
4 comments
Weakness 2. It would be useful to also report the Top-k selection accuracy instead of only ranking correlation, as that's often what researchers more interested in during model iteration. The comparison of the predictive performance between different pipelines is also important in understanding the robustness of the method (e.g., whether adding more features consistenlt leads to better predictive performance).
A: The original-suite Top-3 winner-retention results are:
| Task | CLOSER Hit@3 (%) | Uniform Hit@3 (%) |
|---|---|---|
| ZHVI | 50.0 | 30.0 |
| Health | 13.3 | 24.0 |
| POI spend | 100.0 | 40.0 |
| COVID-19 | 100.0 | 30.0 |
Hit@3 is the percentage of evaluations retaining the full-reference winner, averaged across regimes and random seeds. POI and COVID also have a 100.0 Hit@2, providing a compact shortlist for full evaluation. Similarly, in the new eight-candidate experiments, COVID has a 100.0 Hit@2 and Health has a 100.0 Hit@4.
Predictive performance of the original pipelines.
The following table reports the full-reference predictive performance of all 11 original pipelines. Adding more features does not consistently improve performance, so the candidate ordering contains meaningful feature-selection trade-offs.
| Original pipeline | ZHVI | Health composite | POI spend | COVID-19 |
|---|---|---|---|---|
| PDFM | 58.83 | 58.46 | 59.87 | 30.01 |
| PDFM + ACS | 63.69 | 63.54 | 69.16 | 73.07 |
| PDFM + ACS + Flow | 62.86 | 63.57 | 69.84 | 74.06 |
| PDFM + ACS + Flow + Node2Vec | 64.78 | 63.42 | 68.39 | 71.00 |
| PDFM + ACS + Node2Vec | 63.96 | 63.43 | 66.61 | 71.86 |
| PDFM + ACS + GraphSAGE | 63.46 | 63.57 | 68.75 | 73.05 |
| PDFM + ACS + GraphSAGE + Flow | 62.55 | 63.72 | 69.24 | 71.55 |
| PDFM + Flow | 64.32 | 58.55 | 66.31 | 52.01 |
| PDFM + Node2Vec | 63.26 | 57.99 | 58.01 | 6.39 |
| GraphSAGE | 45.03 | 34.90 | 46.30 | 14.82 |
| Node2Vec | 23.63 | 20.54 | 21.99 | -87.65 |
All scores are multiplied by 100 and averaged across regimes.
Across all pairs that only differ by one feature set addition, 27 score changes are positive and 21 are negative. This observation shows why explicitly evaluating and ranking the feature-bundle pipelines is useful: feature count alone does not determine which pipeline performs best.
Thank you for your response. The additional experiments demonstrate that the fixed budget remains effective with newly added baselines. I hope this can be discussed, both empirically and theoretically, in the next version of the paper. My other concerns have also been addressed. I have updated my rating.
Thanks for raising the score! We promise to include the additional results with an analysis in the next version of the paper.
Reply to Reviewer gAho
Thank you for the suggestions on new architectures, reuse, and model-selection metrics. We tested frozen subsets with additional candidates and report the results below.
A: We added new candidate pipelines—including an independently pretrained representation—and used none of their scores for calibration: all were evaluated on previously fixed subsets, weights, and budgets.
The extension contains six LightGBM/ExtraTrees pipelines using PDFM, PDFM + ACS, and Node2Vec + ACS, followed by two additional pipelines using SatCLIP (Klemmer et al., AAAI 2025), with and without ACS. SatCLIP provides an independent representation-pretraining approach.
The experiment is performed on COVID and Health, note that Health has the lowest original aggregate rank agreement.
We observe that CLOSER subsets can generalize to new pipelines, including geospatial representations learned using different pretraining architectures.
A: Adding a pipeline does not require retraining the probe, reselecting locations, or recalibrating the budget. Three features of the construction make this possible.
The task representation and fixed-budget selection are candidate-model-independent. The probe learns from task outcomes and regional embeddings; facility location uses the resulting geometry. Selection at a fixed K does not read the added candidate's predictions, parameters, or gradients.
The budget is a task-level operating point. The initial calibration suite identifies a useful coverage/sample-support trade-off for that task. We then freeze K∗ with its selected locations and weights, and evaluate whether this operating point transfers to additional pipelines.
Each candidate's subset score is computed independently. With (PK∗,wK∗) and the 80/20 split-seed sequence fixed, qsub(m;PK∗,wK∗) depends on that candidate's own fits and predictions. Adding a competitor leaves the existing scores unchanged. Only the new pipeline is fitted and evaluated; the candidate list is then reranked.
Q1 directly tests this reuse procedure. Neither the six tree-predictor scores nor the two additional SatCLIP-pipeline scores were used for subset or budget calibration. The observed advantages on COVID and Health therefore come from applying the frozen characteristic sets to the added candidates. This provides empirical evidence that the calibrated task-level configuration can be reused across the tested predictor and representation changes.