acceptodds
Under review as a conference paper at ICLR 2027

CLOSER: Characteristic Locations for Efficient Geospatial Model Evaluation

Abstract

Geospatial prediction tasks estimate outcomes such as house prices or disease incidence for specific locations or regions. Candidate pipelines for such tasks may select different options for representing locations, additional input features, and prediction models. Comparing these pipelines can require repeated training and evaluation across thousands of locations, often for multiple outcomes or time periods. This paper addresses the challenge of efficient candidate pipeline selection: Can we identify a small weighted subset of characteristic locations to approximate the performance of a candidate pipeline over all locations? We introduce CLOSER, which uses knowledge about task-aware location representations and solutions to the facility location problem to select complementary representative locations, where a weight associates the count of locations that are assigned to each representative location. The subset size is determined using predictive scores on the candidate subsets, without requiring full-location pipeline scores. The selected subset and weights is then applied across all candidate pipelines for that task. Based on evaluations over outcomes including house prices, health statistics, COVID-19 incidence, and point-of-interest spending, we demonstrate that the selected subsets achieve higher Spearman rank correlation with all location rankings, in comparison to matched-size uniform sampling, while retaining less than of locations on most tasks. Paired timing measurements show – speedups for the subset, when compared to evaluation over all locations, over the suite of outcomes and candidate pipelines.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Geospatial machine learning, Efficient model evaluation, Representative subset selection, Model ranking, Submodular optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gAhoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Evaluating geospatial representations often requires repeated training and held-out evaluation using downstream predictors across many locations. This paper proposes CLOSER, which selects a small subset of the locations for each prediction tasks, aiming to reduce the evaluation time while preserving the ranking agreement while full-location evaluation. CLOSER is applied on four prediction tasks and shows a significant speedups and improved agreement with full-location rankings compared with uniform sampling.

Critical strengths
  • The paper introduces an interesting and potentially important question in the light of recent development of large-scale geospatial pretraining models.
  • Strong experimental results showing the substantial reduction in training and evaluation time.
  • Well-written and clearly presented.
Critical weaknesses
  • The current experiments do not fully support the claimed reusability of the selected location subsets. The evaluated pipelines mainly involves concatenations of a set of pretrained geospatial representations and other features, whole relative performance rankings may be easier to preserve on small subset. In practice, newly developed models may employ more complex and different architectures, with their perdictive performance more difficult to infer. It remains unclear whether the method can still reliably preserve rankings across such models.
  • It would be useful to also report the Top-k selection accuracy instead of only ranking correlation, as that's often what researchers more interested in during model iteration. The comparison of the predictive performance between different pipelines is also important in understanding the robustness of the method (e.g., whether adding more features consistenlt leads to better predictive performance).
Questions for authors
  • Can the authors include more diversed pipelines (e.g., geospatial representations learned using different pretraining architectures, rather than concatenation)?
  • It appears that the set of candidate pipelines will affect the automatic budget selection and resulting final selected location subset. If there's a new pipeline, how to ensure that the pre-selected subset still applies?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer gAho

Thank you for the suggestions on new architectures, reuse, and model-selection metrics. We tested frozen subsets with additional candidates and report the results below.

Q1. Can the authors include more diversed pipelines (e.g., geospatial representations learned using different pretraining architectures, rather than concatenation)?

A: We added new candidate pipelines—including an independently pretrained representation—and used none of their scores for calibration: all were evaluated on previously fixed subsets, weights, and budgets.

The extension contains six LightGBM/ExtraTrees pipelines using PDFM, PDFM + ACS, and Node2Vec + ACS, followed by two additional pipelines using SatCLIP (Klemmer et al., AAAI 2025), with and without ACS. SatCLIP provides an independent representation-pretraining approach.

The experiment is performed on COVID and Health, note that Health has the lowest original aggregate rank agreement.

TaskAdded candidate suiteCLOSER ρ×100\rho\times100Uniform ρ×100\rho\times100
COVID-19Six tree-predictor pipelines60.042.9
COVID-19Eight pipelines, including SatCLIP51.231.3
HealthSix tree-predictor pipelines65.727.6
HealthEight pipelines, including SatCLIP70.253.2

We observe that CLOSER subsets can generalize to new pipelines, including geospatial representations learned using different pretraining architectures.

Q2. It appears that the set of candidate pipelines will affect the automatic budget selection and resulting final selected location subset. If there's a new pipeline, how to ensure that the pre-selected subset still applies?

Weakness 1. The current experiments do not fully support the claimed reusability of the selected location subsets. The evaluated pipelines mainly involves concatenations of a set of pretrained geospatial representations and other features, whole relative performance rankings may be easier to preserve on small subset. In practice, newly developed models may employ more complex and different architectures, with their perdictive performance more difficult to infer. It remains unclear whether the method can still reliably preserve rankings across such models.

A: Adding a pipeline does not require retraining the probe, reselecting locations, or recalibrating the budget. Three features of the construction make this possible.

The task representation and fixed-budget selection are candidate-model-independent. The probe learns from task outcomes and regional embeddings; facility location uses the resulting geometry. Selection at a fixed KK does not read the added candidate's predictions, parameters, or gradients.

The budget is a task-level operating point. The initial calibration suite identifies a useful coverage/sample-support trade-off for that task. We then freeze K∗K^* with its selected locations and weights, and evaluate whether this operating point transfers to additional pipelines.

Each candidate's subset score is computed independently. With (PK∗,wK∗)(P_{K^*},w_{K^*}) and the 80/20 split-seed sequence fixed, qsub(m;PK∗,wK∗)q^{\mathrm{sub}}(m;P_{K^*},w_{K^*}) depends on that candidate's own fits and predictions. Adding a competitor leaves the existing scores unchanged. Only the new pipeline is fitted and evaluated; the candidate list is then reranked.

Q1 directly tests this reuse procedure. Neither the six tree-predictor scores nor the two additional SatCLIP-pipeline scores were used for subset or budget calibration. The observed advantages on COVID and Health therefore come from applying the frozen characteristic sets to the added candidates. This provides empirical evidence that the calibrated task-level configuration can be reused across the tested predictor and representation changes.

Weakness 2. It would be useful to also report the Top-k selection accuracy instead of only ranking correlation, as that's often what researchers more interested in during model iteration. The comparison of the predictive performance between different pipelines is also important in understanding the robustness of the method (e.g., whether adding more features consistenlt leads to better predictive performance).

A: The original-suite Top-3 winner-retention results are:

TaskCLOSER Hit@3 (%)Uniform Hit@3 (%)
ZHVI50.030.0
Health13.324.0
POI spend100.040.0
COVID-19100.030.0

Hit@3 is the percentage of evaluations retaining the full-reference winner, averaged across regimes and random seeds. POI and COVID also have a 100.0 Hit@2, providing a compact shortlist for full evaluation. Similarly, in the new eight-candidate experiments, COVID has a 100.0 Hit@2 and Health has a 100.0 Hit@4.

Predictive performance of the original pipelines.

The following table reports the full-reference predictive performance of all 11 original pipelines. Adding more features does not consistently improve performance, so the candidate ordering contains meaningful feature-selection trade-offs.

Original pipelineZHVIHealth compositePOI spendCOVID-19
PDFM58.8358.4659.8730.01
PDFM + ACS63.6963.5469.1673.07
PDFM + ACS + Flow62.8663.5769.8474.06
PDFM + ACS + Flow + Node2Vec64.7863.4268.3971.00
PDFM + ACS + Node2Vec63.9663.4366.6171.86
PDFM + ACS + GraphSAGE63.4663.5768.7573.05
PDFM + ACS + GraphSAGE + Flow62.5563.7269.2471.55
PDFM + Flow64.3258.5566.3152.01
PDFM + Node2Vec63.2657.9958.016.39
GraphSAGE45.0334.9046.3014.82
Node2Vec23.6320.5421.99-87.65

All scores are multiplied by 100 and averaged across regimes.

Across all pairs that only differ by one feature set addition, 27 score changes are positive and 21 are negative. This observation shows why explicitly evaluating and ranking the feature-bundle pipelines is useful: feature count alone does not determine which pipeline performs best.

Thank you for your response. The additional experiments demonstrate that the fixed budget remains effective with newly added baselines. I hope this can be discussed, both empirically and theoretically, in the next version of the paper. My other concerns have also been addressed. I have updated my rating.

Thanks for raising the score! We promise to include the additional results with an analysis in the next version of the paper.

Reviewer_WRnQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The article presents an approach to selecting a small subset of a larger geographic prediction dataset to speed up training and evaluation. It proposes a greedy search algorithm based on cosine similarities over task-tuned population dynamics foundation model (PDFM) embeddings. The final size of the subset is selected by a grid search that runs all geographic prediction models to be evaluated and selects the size that yields the highest median predictive performance. Evaluated on four geographic prediction datasets (e.g., Zillow home values), the approach mostly outperforms other sampling approaches, but does not yield usable levels of rank agreement with the real baseline, using the full dataset, with Spearman ρ well below .8 for half of the datasets.

Critical strengths
  • The article focuses on an important task: finding a subset of a geographic prediction dataset that, when used for training and evaluation, yields model rankings similar to those that would have been found if models were trained and evaluated on the full dataset.
  • The article includes a variety of ablations that help the reader understand the importance of different components of the proposed approach.
Critical weaknesses
  • The approach does not seem usable (Spearman ρ well below .8) for half of the tasks it is evaluated on (Zillow home values and Health conditions). Yes, it's better than the ρ over a random subset (and better sometimes than some of the other baselines), but the goal is not that; it's "to find a single (P, w) that achieves high ranking agreement" with the ranking over the full dataset. With such low ranking agreements, it seems like we're stuck with the real baseline, using the full dataset, since the selected subset is too poor of an approximation.
  • The performance-size tradeoff is not correctly evaluated. The proposed approach has significant setup costs that have to be done for every dataset and every set of models to be evaluated (since the models are used to select the subset size, K). Yet those setup costs are excluded when comparing to the full dataset, which does not have any such setup costs. (The performance-size tradeoff is also primarily described in the appendix, yet the main point of the proposed approach is to reduce dataset size while maintaining model ranking, so the main text of the paper should have included this as a main result, ideally as a figure of ρ vs. subset size for the proposed approach and all baselines. Room could be made for this by moving the automatic budget selection analysis to the appendix, since the automatic budget selection isn't very successful, only outperforming a flat 10% budget in 2 of the 4 datasets.)
Questions for authors
  • How do the baselines relate to the prior work referenced on lines 066-069, 074-075, and 084-085? Is geographic k-medoids the same as "spatially balanced sampling"? Please spell out for each prior work whether it was used as a baseline, and if not, why not.
  • What is the task-aware probe trained on? Since no subset has yet been selected, it seems like it must be trained on the full dataset for the task?
  • If the task-aware probe is trained on the task dataset, how much cheaper is training the task-aware probe than training each of the actual models (pipelines)?
  • Why should we expect that selecting K to maximize median predictive performance across all models will select the K that best reflects performance on the entire dataset? Won't an especially easy subset maximize median predictive performance and therefore be selected, even when the full dataset is actually harder?
  • Why does the forecasting extension in Appendix A.3 compare only against a weaker baseline, uniform, rather than against the strongest baseline, embedding-based facility location?
6 comments
AuthorsOpenReview ↗

Reply to Reviewer WRnQ

Thank you for the detailed questions about baselines, budget selection, and practical efficiency. We provide the requested comparisons and cost breakdown below.

Q1. How do the baselines relate to the prior work referenced on lines 066-069, 074-075, and 084-085? Is geographic k-medoids the same as "spatially balanced sampling"? Please spell out for each prior work whether it was used as a baseline, and if not, why not.

A: Geographic k-medoids and spatially balanced probability sampling are different methods; we now include both geographic k-medoids and GRTS. Per request, we provide a table that describes whether we use each cited work as baseline and why.

Cited WorkComparison StatusCore Technical Reason
Spatially Balanced Sampling (Stevens & Olsen, JASA 2004)Evaluated (GRTS)Directly evaluated at matched budget KK. We report results across five equal-inclusion-probability GRTS samples, alongside Geographic kk-medoids as a deterministic spatial control.
Coresets Before Score Sets (Yao et al., arXiv 2026)Partially Evaluated (Embedding-based FL)We evaluate the same greedy facility-location coverage objective on pretrained geographic embeddings. Our baseline uses the geographic representation, similarity construction, and train-and-evaluate protocol specified in our paper. It provides a geographic comparison of the shared selection principle; the original paper’s text-based embedding construction and calibration procedure are not reproduced.
tinyBenchmarks (Maia Polo et al., ICML 2024)Inapplicable (Test-only)Assumes frozen models and a static model–item score matrix. It fits IRT parameters from fixed models' full-set responses to estimate test scores. In our setting, candidate pipelines must be both trained and evaluated on the subset; changing the selected locations alters the training split and the fitted predictor itself, so a fixed response matrix does not exist.
Flash-HELM (Perlitz et al., NAACL 2024)Inapplicable (Test-only / Different output)Gradually evaluates models on an expanding subset. Uses dynamic per-model stopping rather than a shared fixed subset. Flash-HELM relies on the assumption that adding a new data point to the subset does not change the results on the previous data points. Also, it sequentially adjusts the evaluation budget for each pretrained model based on its live leaderboard position. It returns a model-specific stopping decision rather than a common, fixed-KK location pool needed to jointly train and evaluate all candidate pipelines.
Anchor Points (Vivek et al., EACL 2024)Inapplicable (Classification-only)Requires discrete class-confidence vectors. Its selector clusters samples using source classifiers' predicted probabilities on the ground-truth class. Because our tasks predict continuous regression outcomes, no class probabilities exist; clustering in representation space is instead evaluated by our kk-medoids baselines.
SubLIME (Saranathan et al., ACL 2025)Inapplicable (NLP-only)Relies on text-specific linguistic features. Its learned rank-correlation predictor requires Flesch readability, Gunning Fog difficulty, and BERT text dispersion alongside anchor-model responses. Tabular regional features and continuous geospatial outcomes do not provide these text statistics.
BenTo (Zhao et al., ICLR 2025)Inapplicable (NLP-only)Relies on LLM in-context learning (ICL) to compute task transferability. BenTo constructs its similarity matrix by prompting LLMs with in-context examples across language tasks. Standard geospatial predictors (e.g., Ridge, MLP, LightGBM) do not support ICL operations, so its original selector cannot be computed on geographic locations.
SCLCS (Zhan et al., ICLR 2026)Inapplicable (Different input)Requires within-sample functional-connectivity graphs. SCLCS computes structural-perturbation and density-balance scores over internal connectivity matrices learned from multivariate brain time series. Our selectable units are individual geographic regions with feature vectors rather than internal graphs, leaving its selection score undefined.
CRAIG (Mirzasoleiman et al., ICML 2020)Inapplicable (Training data selection)It’s a training data selection method for a specific model, not for benchmark reduction.
GLISTER (Killamsetty et al., AAAI 2021)Inapplicable (Training data selection)It’s a training data selection method for a specific model, not for benchmark reduction.

The result of the added baselines can be found in the next reply.

(Q1 continued) The added GRTS and probe-space k-medoids results for the original experiments are summarized below. Probe-space k-medoids uses the same task-aware probe representations as CLOSER, controlling for the supervision used in location selection.

DatasetCLOSERUniformGRTSProbe-space k-medoids
ZHVI0.6950.5110.4810.659
Health0.4560.4230.3870.429
POI spend0.9140.8240.8020.791
COVID-190.8500.6460.6700.259

Q2. What is the task-aware probe trained on? Since no subset has yet been selected, it seems like it must be trained on the full dataset for the task?

A: It is trained on all locations but not all data points. For example, ZHVI probe is trained only on 4 months in 2025 and evaluate on data from 2015-2024. POI and COVID similarly only use 4 data points per location for probing, compared with 10/9 downstream targets (there was overlap though). Health is a little special in the sense that it does not have dates, so we indeed train on the full dataset.

Per request of reviewer UHgN, we add 2 sets of new experiments that strictly separate the data. First on POI and COVID, we tried only using the held-out 6/5 data points per location for evaluation, and we observe that we still have a strong advantage.

DatasetKCLOSERUniformGRTSProbe-space k-medoids
POI11000.9090.8410.7950.818
COVID510.8270.7050.6530.300

Additionally, we separated the locations following a 30/70 train-test split on COVID-19, where the probe is trained on a 889-county subset, and we only evaluate our method on the held-out set of size 2084. We observe a smaller overall rank agreement advantage under this difficult setting, but we achieve a much higher Hit@3, making our subset more useful in identifying the best method.

MethodMean rank agreement (ρ×100\rho\times100)Hit@3 (%)
CLOSER, source-trained probe79.1100.0
Raw-PDFM facility location78.250.0
Probe-space k-medoids77.30.0
Uniform58.040.0
GRTS46.630.0

Hit@3 measures how many full-reference winners (for each regime) is in the subset's top three. Uniform and GRTS each averages across five random seeds.

Q3. If the task-aware probe is trained on the task dataset, how much cheaper is training the task-aware probe than training each of the actual models (pipelines)?

A: The shared probe takes 0.114–0.307 minutes to train. In the controlled workload, the average full-location evaluation of one pipeline takes 5.74–123.34 minutes. Note that the probe is fitted once and shared across all the pipelines.

TaskProbe training (minutes)Average full-location cost per pipeline (minutes)Percentage (%)
ZHVI0.25280.750.31
Health0.307123.340.25
POI spend0.22820.421.12
COVID-190.1145.741.99

Q4. Why should we expect that selecting K to maximize median predictive performance across all models will select the K that best reflects performance on the entire dataset? Won't an especially easy subset maximize median predictive performance and therefore be selected, even when the full dataset is actually harder?

A: We want to clarify that we are not selecting from all possible subsets, we only select from subsets that are produced by the facility location algorithm, which are constructed to cover diverse parts of the task-aware representation space. In our algorithm, the marginal gain for adding location ii to the selected set PP is

\Delta(i\mid P)=\sum_{j\in B}\max\\!\left\\{C(i,j)-\max_{p\in P}C(p,j),\\,0\right\\}.

A location that duplicates the coverage of existing representatives contributes no gain; a location that covers an underrepresented part of the pool contributes positive gain. Thus an adversarial subset, e.g., one that consists of nearly identical examples, receives little incremental coverage while distinct uncovered regions remain.

We argue that the predictive performance can indicate the quality of the subset. Intuitively, when the training set is very small, the dominant issue will be overfitting. Overfitting will especially limit pipelines that need more samples to realize their full-data advantage. As a result, stronger models are ranked behind weaker models. Better predictive performance can indicate less severe overfitting, so the ranking will more likely reflect pipeline’s true ability, making the median a useful budget-selection signal.

In practice, we observe a positive correlation between the predictive performance and the quality of the subset (measured by the Spearman correlation between the subset and the full set) on 3 out of 4 tasks:

taskcorrelation
ZHVI0.32
Health−0.07
POI spend0.60
COVID-190.60

This could also be a possible explanation of why our result on the health task is not as strong as the other three in our original experiments.

Q5. Why does the forecasting extension in Appendix A.3 compare only against a weaker baseline, uniform, rather than against the strongest baseline, embedding-based facility location?

A: For the forecasting extension, we have added the requested embedding-based facility-location baseline at the same budget, K=50K=50. All three methods use the same 18 forecasting pipelines and 11,511-ZIP evaluation pool.

MethodKKRank agreement (ρ×100\rho\times100)
CLOSER5096.9
Raw-PDFM facility location5075.2
Uniform, mean5088.8

CLOSER improves rank agreement by 0.217 over raw-PDFM FL and 0.081 over uniform.

Weakness 1. The approach does not seem usable (Spearman ρ well below .8) for half of the tasks it is evaluated on… it seems like we're stuck with the real baseline, using the full dataset, since the selected subset is too poor of an approximation.

A: CLOSER's practical use is to screen candidate pipelines at low cost and reserve full-location evaluation for a shortlist. We can adjust the budget according to the accuracy requirement of a particular benchmark.

We also want to argue that useful shortlisting does not require every pair of candidates to be ordered correctly. The original 11-candidate results below report how many percentage of the full-reference winners is retained in the top three:

TaskCLOSER Hit@3 (%)Uniform Hit@3 (%)
ZHVI50.030.0
Health13.324.0
POI spend100.040.0
COVID-19100.030.0

Entries are averaged across regimes and random seeds. Clearly we don’t perform very well on the health task (which we will explain in the next paragraph), but the results on the rest 3 tasks are actually stronger than a single correlation number indicates.

The original Health suite contains closely spaced top candidates, which can make rank agreement sensitive to small score differences. As shown in the full-reference score table (see Reviewer gAho, W2), the top six original pipelines on Health have nearly identical full-set scores (63.42 to 63.72, within a 0.30-point band), where minor perturbations easily swap ranks while preserving linear score association (Pearson r=0.960r = 0.960, Table 4). When we evaluate the exact same frozen Health subset on 8 new, more diverse pipelines (tree predictors and SatCLIP representations, never seen during selection), its Spearman rank agreement reaches 0.702 versus 0.532 for uniform.

Reviewer_UHgNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a simple, greedy-selection based strategy that select subsets of geographical data without hurting model performance. The goal is to keep the subset ranking match full-dataset evaluation. The method:

  • Trains an MLP probe from PDFM regional embeddings to the task outcomes,

  • Runs greedy facility-location selection on cosine similarities between the probe’s hidden states,

  • Weights each selected location by the share of locations assigned to it,

  • Chooses the subset size K* by trading the pipelines’ median predictive score on the subset against a size penalty.

Critical strengths

S1. The problem is practical and under-studied in geospatial ML. Many geographical datasets are extremely large yet not informative. It is worth studying how to reduce the dataset size without hurting model performance.

S2. The method is simple and based on deterministic optimization (e.g. greedy selection. Implementation specifics are well detailed.

Critical weaknesses

W1. The selector uses the evaluation labels, and the main baselines don’t.

In line 83-85, the authors say "We first train a probe to predict observed outcomes from pretrained regional embeddings and use its hidden representations to measure task-specific location similarity." If I understand it correctly, the subsets used to obtain the main results are selected by a probe that have seen ALL the observed labels. This is exactly label-leakage. The uniform and geographical k-medoids baselines obviously do not use the labels. This is unfair comparison.

W2. The paper lacks comparison with traditional spatial survey-sampling literature. In classical geostatistics and survey science, many pure statistical methods are developed to make informed or stratified sampling. For example, spatially balanced sampling and design-based estimators with inclusion probabilities. These are the natural baselines more powerful than the uniform sampling compared in this paper.

Questions for authors

Q1. Have you tried training the probe on a separate held-out labeled data and use the learned probe to select subsets on another training dataset?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer UHgN

Thank you for recognizing the practical importance of efficient geospatial evaluation. We address the supervision setting and the requested comparisons below.

Weakness 1. The selector uses the evaluation labels, and the main baselines don’t… This is exactly label-leakage. The uniform and geographical k-medoids baselines obviously do not use the labels. This is unfair comparison.

A: We fully understand this concern and agree that we could add baselines that use the labels for fair comparison. We have included two new leakage-free settings in answers to Q1. For fair comparison, we also include a new baseline (probe space k-medoid) that uses labels in the same way as we do. The baseline performs clustering on the location embeddings computed by the probe in the first stage of CLOSER.

We also want to clarify that many other papers in this area assume that we have access to the labels, e.g., tinyBenchmarks (Maia Polo et al., ICML 2024), metabench (Kipnis et al., ICLR 2025), and SparseEval (Zhang et al., ICLR 2026). The rationale is that the ground truth for our task is the full-benchmark performance instead of labels for each data point. We will make this assumption clearer in future versions.

Q1. Have you tried training the probe on a separate held-out labeled data and use the learned probe to select subsets on another training dataset?

A: Yes, we have tried that. We respond in two parts: 1. Evaluate on held out data points for seen locations. 2. Evaluate on held out locations.

  1. All of our original experiment on ZHVI evaluates on held-out data points: we used 4 data points per location for training and 40 held-out data points per location for downstream evaluation. We add new experiments on POI/COVID following a similar held-out data-point setting on ZHVI where we use 4 data points per location for training and 6/5 held-out data points per location for downstream evaluation. We observe that we still have a strong advantage over baselines under this no-leakage setting, including the newly added baseline, probe-space k-medoids, that can make use of the label information as we do. (We add generalized random tessellation stratified sampling (GRTS)(JASA 2004) - traditional survey-sampling method. Uniform and GRTS each averages across five random seeds.)
DatasetKCLOSERUniformGRTSProbe-space k-medoids
POI11000.9090.8410.7950.818
COVID510.8270.7050.6530.300
  1. Additionally, we tried to separate the locations into 2 disjoint sets on COVID. We used a 30/70 train-test split. The probe is only trained on 889 counties and the downstream evaluation is only performed on the held-out set of size 2084. Hit@3 measures how many full-reference winners (for each regime) is in the subset's top three. We observe a smaller overall rank agreement advantage under this difficult setting, but we achieve a much higher Hit@3, making our subset more useful in identifying the best method.
MethodMean rank agreement (ρ×100\rho\times100)Hit@3 (%)
CLOSER, source-trained probe79.1100.0
Raw-PDFM facility location78.250.0
Probe-space k-medoids77.30.0
Uniform58.040.0
GRTS46.630.0

Weakness 2. The paper lacks comparison with traditional spatial survey-sampling literature…

A: We added generalized random tessellation stratified sampling (GRTS)(JASA 2004), which is one of the mentioned traditional survey-sampling method. We’ve already included the results of this baseline in Q1 experiments.

For the original experiments, the table below reports GRTS and the newly added probe-space k-medoids baseline. Our method is still much stronger than the newly added baselines.

DatasetCLOSERUniformGRTSProbe-space k-medoids
ZHVI0.6950.5110.4810.659
Health0.4560.4230.3870.429
POI spend0.9140.8240.8020.791
COVID-190.8500.6460.6700.259

Thank you for presenting more experimental results. My concern about the experiment setup (all four datasets are very small, ranging from 10k to 30k) remains. I'd keep my rating.

Thank you for acknowledging the additional experiments. Could you clarify whether these results address your original concerns about baseline fairness, label leakage, and comparisons with traditional spatial survey-sampling methods? We have added label-disjoint and location-disjoint probe experiments, a probe-space k-medoids baseline using the same supervision as CLOSER, and GRTS comparisons. If any of these concerns remain, we would appreciate clarification on the specific unresolved points.

Regarding the newly raised concern about dataset size in your follow-up, we would like to clarify two aspects of the experimental scale.

First, the reported location counts should be interpreted relative to the geographic units studied. ZHVI, Health, and POI use ZIP-code-level regions, while COVID uses counties. The number of distinct locations is therefore naturally bounded by the number of these geographic units in the United States, with data availability determining the usable pool for each task. For context, the 2020 Census lists 33,642 ZIP Code Tabulation Areas and 3,143 counties and county equivalents across the 50 states and DC. Our Health reference contains 31,738 locations, and our COVID dataset contains 2,973 counties. These counts reflect substantial geographic coverage at the resolutions used in our study.

Second, each location contributes multiple prediction targets. The main evaluations use:

DatasetPrediction targets per location
ZHVI40
Health21
POI spend10
COVID-199

Therefore, the effective dataset size is much larger than 10k-30k.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.