acceptodds
Withdrawn from ICLR 2027

When Spatial Blocking Makes Distance Buffering Redundant: Structural Inertness versus Small Effects

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spatial blocking and distance buffering both limit dependence between training and test samples, but their marginal roles are rarely separated. We define the buffer effect L as unbuffered minus buffered AUC on identical test units while matching training size and class counts, then vary only whether those units are scattered or contiguous. Removing near-test training samples costs +.0693 AUC for scattered street-view units at r=100 m (95% CI [+.0357,+.1028]; 7/7 cities). Pooling two disjoint corrected So2Sat LCZ42 city sets gives +.0252 at r=200 m ([+.0136,+.0369]; 29/41 cities; p<10^-4). Compact blocking produces two distinct cases: in 28/48 primary cities the realised buffered and unbuffered training multisets are identical, so L_B=0 by construction; among the remaining active cities, mean L_B ranges from -.0012 to +.0004 and each collection-level 90% interval lies inside a prespecified +/-.005 AUC tolerance, although one city exceeds that tolerance. A coordinate-only three-state rule separates provably inert, certified-bounded, and must-measure cases for spatially local learners, with no violations in 1,368 full-pool checks. Redundancy is therefore a property of the realised geometry and training intervention, not of the label "spatial CV."

Primary area:
datasets and benchmarks
Keywords:
evaluation protocols, data leakage, spatial cross-validation, distance buffering, spatial blocking, remote sensing benchmarks, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_GraVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

As far as I can tell the paper attempts to present insights on when adding distance buffering to spatial blocking can be advantageous in a setting where datapoints have spatial locations. The findings suggest that distance buffering does not always help. The presentation of the paper is such that I cannot discern anything deeper.

Critical strengths

None

Critical weaknesses
  • The presentation; The paper is filled with extreme amounts of jargon, awkward wording, and terms that are introduced without any reasonable context afforded to the reader. See an incomplete list below.
  • The abstract and contribution list give details before context. The reader is given almost no context on the setting or the problem before details, outcomes, and new terms are given. This makes it difficult to interpret correctness or impact of what is being said.
  • The setting introduced in Section 3.1 is unclear and jargon-filled from the very first sentence: "For city c and test unit u, let Tcu be the test samples, Rcu = Dc \ Tcu the unbuffered training pool, and Rcu,r = {i ∈ Rcu : d(i, Tcu) > r} the pool after applying radius r.". What do you mean by "a city", "test unit", "training pool"? What is "Dc"? "radius" of what?
    • This is never clarified and the following sentences just introduce more terms.
    • I find it indecipherable. Since this is the core of the paper, I am unable to interpret the results stemming from it.
  • The introduction states that "Existing comparisons often change the holdout geometry, training size, and effective distance simultaneously" and therefore "cannot tell whether a buffer contributes information beyond a block". This seems to be the gap the paper claims to fill, but no study is cited for it there or anywhere else in the paper.
  • It is unclear where the core experimental methods are defined. It seems to be scattered across 3.1, 3.2, 4, and 5.

Incomplete list of terms used without definition or before reasonable context is given (line of first use): L_B (l. 021), "primary cities" (l. 020), "active cities" (l. 021), "collection-level" interval (l. 022), "corrected" So2Sat LCZ42 (l. 018), "three-state rule/certificate" (l. 024), "certified-bounded" and "must-measure" (l. 024–025), "spatially local learners" (l. 025), "full-pool checks" (l. 026), "distance-matching" and "spatial-plus" (l. 046), "matched decomposition" (l. 051), "geometry-only diagnostics" (l. 052), "three city collections" (l. 053), "probes" (l. 053), "class-matched synthetic system" (l. 053–054), "oracle experiment" (l. 054), "protocol–radius cells" (l. 055), "provably inert" (l. 024), CityRep (l. 057), "action" (l. 073).

Questions for authors

None

Reviewer_9kixOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether distance buffering provides additional benefits when spatial blocking is already used for train-test splitting. The authors compare buffered and unbuffered evaluation under matched training budgets and introduce a coordinate-based diagnostic to identify cases where buffering has no effect or a bounded effect. Experiments on spatial imaging datasets and several public benchmarks suggest that compact spatial blocking often makes additional short-distance buffering redundant.

Critical strengths

The paper focuses on a relevant issue in spatial machine learning evaluation, where dependencies between training and test samples may lead to misleading generalization estimates.

The authors conduct extensive empirical analyses across different spatial datasets, splitting strategies, and model configurations, and provide additional theoretical results concerning the effect of buffering on local learners.

Critical weaknesses

My main concern is the limited methodological novelty. The observation that distance buffering becomes redundant when spatial blocking already separates training and test samples is relatively intuitive. Although the paper introduces a three-state diagnostic, its contribution beyond existing spatial cross-validation methods remains unclear.

The theoretical contribution is also limited. The structural inertness result follows directly from identical training sets, while the proposed effect bounds apply only to local learners and do not directly support the main experiments using global models.

Another concern is the limited generalizability of the findings. The results depend heavily on spatial geometry, datasets, and buffering radii, and some zero effects simply arise from unchanged training sets. The conclusions are also not consistent across datasets.

Finally, the experiments mainly rely on frozen representations and simple prediction models, without sufficient validation in end-to-end learning settings. Overall, the technical contribution appears relatively weak despite the extensive experimental analysis.

Questions for authors

See weaknesses above.

Reviewer_dMxwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether distance buffering provides additional benefits when spatial blocking is already used for model evaluation. The authors distinguish between structural inertness, where buffering leaves the training set unchanged, and small empirical effects, where buffering changes the training data but has little impact on performance. They propose a matched evaluation framework and a coordinate-based diagnostic to assess the effect of buffering without necessarily training models. Experiments on street-view and satellite imagery datasets show that buffering can substantially affect evaluation under scattered splits, while its impact is often negligible under compact spatial blocking. The paper further validates these findings through synthetic experiments and audits of existing spatial benchmarks.

Critical strengths
  1. The paper addresses a practical question in spatial machine learning: whether distance buffering is still necessary after spatial blocking. The distinction between structural inertness and small empirical effects is useful for understanding when additional buffering changes the evaluation protocol. The proposed coordinate-based diagnostics also provide a practical way to identify unnecessary buffering without additional model training in certain settings.

  2. The authors use matched training sizes and class counts to isolate the effect of buffering, reducing potential confounding factors. The evaluation includes multiple datasets, spatial splitting strategies, buffer radii, feature representations, and learning algorithms.

Critical weaknesses
  1. The central observation that buffering has no effect when the realized training sets are identical is relatively straightforward. Similarly, the proposed bounds for local learners follow from identifying which test predictions can change and bounding their contributions to the evaluation metric. The paper would benefit from a clearer explanation of what is fundamentally new beyond existing spatial validation methods and basic properties of local prediction algorithms.

  2. The main experiments focus on two imaging data sources and primarily use frozen visual representations with relatively simple downstream classifiers. Although additional learners are considered in sensitivity analyses, the paper does not directly evaluate end-to-end trained neural networks. Furthermore, the coordinate-based performance bounds apply only to learners with explicitly defined local dependencies and cannot generally certify small effects for global models. These limitations make it difficult to assess how broadly the practical conclusions extend to modern deep learning settings.

Questions for authors
  1. The structural inertness condition follows directly from the equality of the realized training sets, while the metric bounds are derived by considering the test predictions or sample pairs that may change. Could the authors clarify the main theoretical novelty of the proposed certificate relative to existing analyses of local learning algorithms and spatial validation?

  2. The primary experiments rely on frozen representations and downstream probes. Would similar conclusions hold when the feature extractor is fine-tuned end-to-end? In particular, could removing a small number of spatially adjacent training samples have a larger effect on learned representations and downstream performance?

  3. For global learning algorithms, the proposed decision rule can only certify structural inertness or indicate that measurement is required. Given this limitation, could the authors clarify how often the framework provides actionable conclusions beyond directly checking whether buffering changes the training set?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.