acceptodds
Under review as a conference paper at ICLR 2027

Learning to Localize by Retrieval: Spectral-Spatial Contrastive Representation Learning for Multispectral Satellite Imagery

Abstract

In most cases, supervised object localization in multispectral images is based on dense bounding-box annotations which is not always possible when these annotations are not available or are costly. We formulate localization as similarity search in a learned embedding space and avoid learning to predict a bounding box, thereby eliminating the need for bounding-box supervision for detector-free localization. We train a four channel multispectral encoder with self-supervised momentum contrastive learning to learn the discriminative patch representations. A partition of the target imagery is constructed into overlapping patches, which are incorporated into a normalized representation space, and then indexed so that it is easy to retrieve the patch corresponding to the nearest imagery. Similar patches are retrieved and translated into localized regions by similarity filtering and spatial consolidation, given a query region. We test the learned representation from four complementary viewpoints: embedding separability, retrieval without labels, localization consistency despite query perturbations and retrieval scalability with the growth of the indexed corpus. Based on our results, self-supervised contrastive learning yields structured multispectral representations, which enables repeatable query retrieval based localization at larger corpses and exhibit a low query latency. The results obtained in this study show the viability of an alternative approach to localization in data-constrained multispectral remote-sensing environments based on detector-free retrieval, instead of using annotation intensive detection.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-Supervised Learning, Contrastive Learning, Representation Learning, Multispectral Remote Sensing, Object Localization, Visual Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_6a7eOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper formulates candidate localization in multispectral satellite imagery as similarity retrieval using self-supervised patch representations. Given a query exemplar, the method retrieves similar patches and combines their spatial metadata with similarity filtering and spatial aggregation to generate candidate regions. The experiments examine retrieval behavior, stability, sensitivity to inference parameters, and computational cost.

Critical strengths

The overall design is conceptually reasonable: retrieving similar patches and using their spatial metadata to generate candidate regions provides a natural connection between visual similarity and localization. This offers a plausible approach to exploring localization when bounding-box annotations are scarce.

Critical weaknesses
  1. The experimental evidence is insufficient to substantiate the localization and repeatability claims. The evaluation primarily reports embedding similarities, candidate counts, retrieval overlaps, and search timings. Without independent target annotations or another validated assessment of semantic correctness, these measurements do not establish whether the retrieved regions contain the intended objects. The paper also provides no matched RGB-only baseline, alternative representation-learning baseline, or evaluation on an annotated benchmark, leaving the contribution of the proposed representation unestablished. Moreover, the abstract's claim of repeatable retrieval-based localization is difficult to reconcile with the reported results: horizontal flipping yields an agreement of only 0.1297 and a zero-match rate of 43.44%, while the median Jaccard@100 between same-class queries is zero. These results require a substantially more qualified interpretation. Overall, the experiments characterize the behavior of the pipeline but do not provide sufficient evidence for its effectiveness as a localization method.

  2. The writing, figures, and treatment of prior work make the method difficult to follow and its contribution difficult to assess. The manuscript contains imprecise and internally inconsistent descriptions. For example, Section 4.2 specifies a projection head of 2048 → 2048 → 128, whereas Table 1 lists 2048–128–128. Section 8 states that “the number of candidates retrieved influences the number of candidates retrieved,” which does not communicate an interpretable finding. Figure 1 is crowded with small text, while the experimental figures and their brief captions provide limited guidance for interpreting the results and connecting them to the paper's claims. Several experimental figures and tables, including Figures 3–5 and Table 2, are not explicitly referenced in the main-text discussion, making it difficult to connect the narrative to the corresponding results and follow the experimental analysis. The related-work section largely summarizes a small set of general representation-learning and search components, without adequately comparing the proposed pipeline with prior exemplar-based retrieval and localization methods. Citation accuracy also needs attention: Section 2.1 attributes the MoCo-v2 improvements to Chen et al. (2020a), although the corresponding reference is Chen et al. (2020b). Together, these issues go beyond surface-level editing: they impede understanding of the implemented method and assessment of what is contributed beyond established components.

Questions for authors
  1. What evidence supports the semantic correctness of the localized candidates, beyond high embedding similarity and retrieval overlap? If any independently checked examples or quantitative validation were collected, please identify them and explain the assessment protocol.

  2. What configuration was actually used for the reported experiments? In particular, please resolve the projection-head discrepancy between Section 4.2 and Table 1, and provide the class-specific similarity thresholds and the procedure used to select them. How were these thresholds established without target annotations, and why do all tested threshold offsets produce exactly the same candidate count and mean score?

  3. Which specific methodological elements distinguish this pipeline from existing exemplar-based retrieval and localization approaches? Please clarify the closest relevant methods and whether any comparisons with them were performed but omitted from the submission.

4 comments
AuthorsOpenReview ↗

Response to Weakness 2: Presentation, Consistency, and Related Work

We thank the reviewer for identifying these weaknesses. We have carefully revised the manuscript to improve the technical clarity, positioning with respect to prior work, and overall presentation.

1. Improved positioning and comparison with related work. We agree that the initial version of Section 2 did not sufficiently distinguish our approach from existing representation-learning and localization methods. We have substantially revised this section. Section 2.1 now explains the distinction between conventional supervised classification pipelines, which use classification heads to produce categorical predictions, and our approach, which retains intermediate feature projections to construct an embedding space for exemplar-based retrieval. Section 2.3 further clarifies the distinction between dense prediction approaches and our retrieval-based framework. In particular, we explain how associating gallery embeddings with their source-image identifiers and spatial coordinates, together with FAISS-based similarity retrieval, enables retrieved patches to be mapped into spatial candidate regions without training a bounding-box prediction head. These revisions provide a clearer explanation of the methodological motivation and the specific role of spatial metadata in our framework.

2. Correction of the implementation configuration in Table 1. We identified a discrepancy in the projection-head dimensions reported in the original Table 1. The previously reported configuration, �0��0�, was an inadvertent formatting/transcription error introduced during the final preparation of the submission. The actual implemented architecture uses a two-layer projection head with dimensions �1��1�, mapping the 2048-dimensional backbone representation through a 2048-dimensional hidden layer to the final 128-dimensional embedding. We have corrected Table 1 to reflect the implemented architecture and checked the configuration for consistency with the method description.

3. Expanded Discussion and Limitations. We have revised Section 8 to provide a clearer interpretation of the experimental findings and a more explicit discussion of the study's limitations. The revised section distinguishes retrieval behaviour and repeatability from verified localization accuracy, acknowledges the absence of independent target-image annotations, and clarifies the limitations of evaluating a single four-channel ResNet-50 momentum-contrastive representation. We also identify the need for matched RGB-only comparisons, alternative self-supervised objectives and architectures, and evaluation on publicly annotated multispectral benchmarks.

4. Manuscript presentation and cross-referencing. We have also reviewed the manuscript for presentation and consistency. Figure and table captions, numbering, and in-text references have been checked and corrected where necessary. We have additionally reviewed the in-text citations against the reference list and corrected citation and formatting inconsistencies. The supplementary appendices have also been organized to provide additional mathematical, implementation, and experimental details.

We appreciate the reviewer’s constructive feedback. These revisions have helped us improve the clarity of the methodological contribution, ensure that the reported implementation configuration accurately reflects the actual model, and present the scope and limitations of the study more transparently.

AuthorsOpenReview ↗

Response to Question 1: Semantic correctness of retrieved candidates

We thank the reviewer for identifying this important distinction. We agree that high embedding similarity and retrieval consistency alone do not establish the semantic correctness of the generated candidate regions.

In our evaluation setting, the available annotations are used to construct class-specific query exemplars, whereas independent ground-truth annotations for the 40 target images are unavailable. Consequently, we cannot independently verify whether each retrieved candidate contains the intended object. We have not conducted a separate, systematic human verification study or quantitative target-object localization evaluation. The visual inspection performed during heuristic threshold selection was intended to guide parameter selection and should not be interpreted as an independent correctness assessment.

The quantitative experiments in the manuscript measure different aspects of retrieval behaviour. The similarity-distribution analysis characterizes embedding relationships; the spatial-concentration analysis examines the distribution of retrieved patches across target images; and the perturbation and multi-query consistency experiments measure retrieval repeatability. In particular, matching candidate boxes between original and perturbed queries measures consistency between two retrieval outputs, not agreement with target-object ground truth.

We have therefore clarified the scope of the claims. Each generated box represents the spatial footprint of a retrieved gallery patch, mapped using its stored image identity and coordinates, rather than an independently predicted or verified object boundary. The method is consequently more accurately described as retrieval-based spatial candidate generation (or candidate localization), rather than a ground-truth-validated object detector.

We acknowledge that independently annotated target subsets and a predefined human-assessment protocol would be necessary to establish semantic correctness. We identify these as important directions for further evaluation.

AuthorsOpenReview ↗

Response to Question 2: Implementation configuration and similarity thresholds

We thank the reviewer for identifying the configuration discrepancy and the need to clarify the threshold-selection procedure.

Projection-head configuration. The implemented projection head maps the 2048-dimensional ResNet-50 backbone representation through a 2048-dimensional hidden layer to a 128-dimensional embedding, with Batch Normalization and ReLU between the projection layers. Thus, the correct configuration is 2048→2048→1282048\rightarrow2048\rightarrow128. The previously reported 2048→128→1282048\rightarrow128\rightarrow128 entry in Table 1 was an inadvertent formatting/transcription error during final manuscript preparation. We have corrected Table 1 and clarified the architecture description in Section 4.2 to ensure consistency with the implementation.

Threshold selection. The class-specific thresholds were selected heuristically through iterative trial-and-error and visual inspection of retrieved candidates. The query-chip annotations were used to define the eight query classes and construct their exemplars. However, the thresholds were not optimized or calibrated against independent target-image annotations, which were unavailable. The original thresholds and the complete class-wise score distributions are now reported in Appendix D.

Explanation of unchanged threshold offsets. We audited the Top-500 similarity scores for all 261 query chips, producing 130,500 query-gallery matches. The minimum observed similarity across all matches was 0.995068, whereas the maximum threshold tested under the +0.10+0.10 offset was 0.60. Therefore, every retrieved match passed all the original thresholds and tested offsets. The threshold comparison was executed, but the selected cutoffs were non-binding over the observed score distribution. Consequently, the unchanged candidate count and mean score do not indicate threshold robustness; they result from thresholds that were overly permissive relative to the observed similarities.

To investigate sensitivity at the actual score scale, we added a class-wise percentile-based threshold analysis. At the 90th-percentile setting, the final candidate count decreases to 5,316, compared with 13,288 under the original thresholds. At the 95th percentile, 3,359 candidates remain. The complete threshold values, score distributions, candidate counts at successive inference stages, and spatial-coverage visualizations are provided in Appendix D.

We emphasize that the percentile-based settings characterize candidate-generation sensitivity and are not interpreted as calibrated confidence thresholds or optimal localization thresholds. Without independent target annotations, their effect on localization correctness cannot be established.

AuthorsOpenReview ↗

Response to Question 3: Methodological distinction from existing approaches

We thank the reviewer for requesting a clearer explanation of the methodological distinction and its relationship to existing exemplar-based retrieval and localization approaches.

We agree that the individual components of our pipeline, including contrastive representation learning, similarity search, FAISS indexing and non-maximum suppression, are established techniques. Our contribution is not to introduce a new similarity-search algorithm or a new detection architecture, but to investigate their integration for retrieval-driven spatial candidate generation in four-channel multispectral satellite imagery.

The distinction lies in the way the learned representation is used. Rather than generating region proposals through a separate detection network and subsequently classifying them, our framework constructs a gallery of overlapping multispectral patches. Each gallery embedding retains its source-image identifier and spatial coordinates. A query exemplar is embedded in the same representation space, and exact FAISS inner-product search retrieves the Top-KK most similar gallery patches. Their stored metadata maps them back to spatial candidate regions, which are subsequently processed using class-specific filtering and non-maximum suppression.

We have further clarified the relationship to representative few-shot detection approaches. For example, Li et al. (2022) employ a YOLOv3-based meta-feature and feature-reweighting framework with a bounding-box prediction module. Bou et al. (2024) use a two-stage detection architecture in which pretrained visual features and class-reference prototypes classify region proposals. In contrast, our approach performs direct patch-level nearest-neighbour retrieval and spatial recovery from gallery metadata, without a separate region proposal network, prototype classifier or bounding-box regression head.

We also clarify that no direct empirical comparisons against these methods, a matched pretrained RGB/NIR retrieval baseline, or a weakly supervised detector were performed and subsequently omitted. The current study is therefore not intended to establish comparative superiority over existing detection approaches. We have strengthened the Related Work and Limitations sections to make this distinction and the absence of comparative evaluation explicit.

Accordingly, we position the contribution as a retrieval-based framework for spatial candidate generation in multispectral satellite imagery, and not as a replacement for or a quantitatively superior alternative to established object detectors.

Reviewer_B8UHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper trains a four-channel ResNet-50 with MoCo-style self-supervision and uses its 128-dimensional patch embeddings for exemplar retrieval in multispectral satellite imagery. Overlapping target patches are indexed with exact FAISS search; retrieved patches become fixed-size candidate boxes after class-specific thresholding and NMS. Evaluation on 221 queries and a 32,542-patch gallery studies embedding similarity, retrieval depth, latency, perturbation stability, parameter sensitivity, and agreement among same-class queries. No independent target annotations are available, so the work does not measure localization precision, recall, AP, or box accuracy.

Critical strengths
  1. The paper is transparent about its limitations and includes useful stress tests showing that high embedding cosine does not imply stable retrieval neighborhoods.
  2. The detector-free pipeline is simple and may be useful for exploratory candidate generation when labels are genuinely unavailable.
Critical weaknesses
  1. The central localization claim is not validated. Without independent target ground truth, the paper cannot establish whether retrieved candidate boxes correspond to target objects; similarity, self-consistency, and latency are not substitutes for precision, recall, AP, or IoU. This is a fundamental evidentiary gap rather than a minor missing ablation.
  2. There are no matched retrieval or detection baselines, no RGB-only control, and no alternative self-supervised encoder. Moreover, retrieval is highly unstable under flips and small noise, same-class exemplars often retrieve disjoint neighborhoods, and threshold changes have no effect, suggesting that the proposed scores are poorly calibrated for localization.
Questions for authors
  1. Can the authors evaluate on any annotated target subset or public multispectral localization benchmark and compare against simple baselines such as pretrained RGB/NIR retrieval and a weakly supervised detector? Without this, what evidence supports the term localization rather than candidate retrieval?
  2. How were class-specific thresholds selected without target labels, and why do offsets of +/-0.10 leave every output unchanged despite similarities being concentrated near 0.999? This may reveal saturation or an implementation/calibration issue.
3 comments
Reviewer_B8UHOpenReview ↗

Response by Reviewer

Hi authors, I'm looking forward to discussing the paper. If my concerns are addressed, I will reconsider my score.

AuthorsOpenReview ↗

Response to Question 2: Threshold Selection and Sensitivity

Response to Question 2: Selection and sensitivity of class-specific similarity thresholds

We thank the reviewer for this question. We address each part below and have added the supporting details and analyses to Appendix D for verification.

How the thresholds were selected. The class-specific thresholds (τc\tau_c, ranging from 0.35 to 0.50 across classes) were selected heuristically through iterative trial-and-error and visual inspection of the retrieved candidates for each class. The annotated query chips were used to define the query categories and construct the class-wise exemplars. Independent target-image annotations and target-object bounding boxes were unavailable, so the thresholds were not optimized or calibrated against target-object ground truth. In hindsight, visual inspection could not guide these values, because they lay far below the range of the retrieved similarity scores. The selection procedure and the threshold of each class are now documented in Appendix D.

Why the offsets of ±0.10\pm 0.10 left the outputs unchanged. We audited the Top-500 similarity scores of all 261 query chips (130,500 query-gallery matches). The minimum class-wise similarity ranged from 0.995068 to 0.999046, and the overall range was 0.995068 to 0.999909, whereas the largest threshold tested under the +0.10+0.10 offset was only 0.60. All retrieved matches therefore passed the original thresholds and every tested offset. The thresholding step was executed, but the cutoffs were non-binding over the observed score distributions. This is a calibration issue rather than an implementation fault: as a check, the lowest-percentile setting in the new analysis (P0P_0) reproduces the original output exactly, while higher percentiles change the candidate counts. The result also reflects the compressed similarity range discussed in Section 6.1, in which meaningful thresholds must lie within a very narrow band. The unchanged counts should therefore not be read as evidence of threshold robustness; they show that the original thresholds were overly permissive.

Additional percentile-based sensitivity analysis. To examine thresholds within the observed score range, we pooled the Top-500 scores of all queries of each class into Sc\mathcal{S}_c and set

τc(p)=Qp(Sc),\tau_c^{(p)} = Q_p(\mathcal{S}_c),

where QpQ_p is the empirical pp-th percentile, evaluated from P0P_0 to P99.5P_{99.5}. Candidate generation is sensitive to thresholds placed in this range. At P90P_{90}, 13,060 matches remain after thresholding, 7,924 after NMS, and 5,316 after the per-image detection cap. At P95P_{95}, the corresponding counts are 6,539, 4,288, and 3,359. At P99P_{99}, 900 final candidates remain.

Additional material in Appendix D. For the reviewer's verification, Appendix D reports: (i) the original thresholds and per-class Top-500 similarity statistics, including the pass rate of each class; (ii) the empirical percentile thresholds for all eight classes; (iii) aggregate and class-wise candidate counts after thresholding, NMS, and the detection cap; and (iv) figures showing the sensitivity of candidate counts and the spatial coverage across target images.

Scope. These percentile thresholds characterize how candidate generation depends on similarity filtering. They remove a fixed fraction of matches by construction and do not establish optimal calibration or improved localization accuracy. Without independent target-image annotations, we cannot determine whether stricter thresholds retain more correct object locations, and we state this limitation in Appendix D. We frame the approach as retrieval-based spatial candidate generation, not a ground-truth-validated object detector.

AuthorsOpenReview ↗

Response to Question 1: Localization Claim, Annotations, and Baselines

We thank the reviewer for this question, which concerns what our evidence can and cannot support. We address its three parts in turn.

1. Annotated target subset or public benchmark. The 40 target images in our challenge setting have no independent object annotations, so we cannot measure whether the retrieved boxes coincide with true objects. For this reason we do not report target-box IoU, AP, mAP, precision, or recall (Section 5.1). The class-wise annotations available to us are used only to construct the query exemplars; they are not target-object annotations. Our experiments therefore characterize the behavior of the retrieval-to-region pipeline: similarity structure (Section 6.1), query stability (Sections 7.1-7.2), sensitivity to inference parameters (Section 7.3), multi-query consistency (Section 7.4) and computational cost (Section 6.2). We agree that evaluation on an independently annotated target subset or a public multispectral benchmark is necessary to establish localization accuracy, and we state this in Sections 8 and 9.

2. Baselines. We agree that the contribution of the four-channel representation is not isolated in the current study. As stated in Section 5.1, no matched RGB-only retraining, alternative self-supervised objective, alternative architecture, or public benchmark is evaluated, and we therefore make no claim that our encoder outperforms an ImageNet-pretrained RGB/NIR encoder or any other representation. The random-pair comparison in Section 6.1 and Figure 2 is used only to describe the similarity distribution; it is not a competing baseline. A weakly supervised detector, as surveyed in Fasana et al. (2022), requires at least image-level labels for the target data, which are not available in our setting, so a fair comparison would need a separately designed protocol. We will state this explicitly in the revised limitations.

3. Evidence for the term "localization". Without annotated evaluation, the evidence in the paper supports retrieval-based spatial candidate generation, not verified localization. Each gallery embedding stores its source-image identifier and patch coordinates (Section 4.4); retrieved patches are mapped back to image coordinates as candidate boxes, which are then filtered by similarity and consolidated by class-wise NMS (Sections 3 and 4.5). This links visual similarity to spatial position without a bounding-box prediction head, but it establishes where retrieved patches are, not whether they contain the intended object. The manuscript already says this: the output is "a set of candidate spatial regions, not direct object-box predictions" (Section 3), it is not a calibrated detector (Section 4.5), the boxes are "retrieval-derived spatial candidates rather than ground-truth-verified object detections" (Appendix A.5), and the framework "should be taken as a similarity-based candidate localization system" (Section 7.4). Our conclusion likewise describes the contribution as spatial candidate generation (Section 9).

Reviewer_jFNWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a detector-free target localization framework for multispectral remote sensing images, aiming to address the reliance of traditional supervised localization methods on dense bounding box annotations. This method models target localization as a similarity retrieval task in the learned embedding space. It trains a four-channel multispectral encoder through self-supervised momentum contrastive learning to obtain discriminative image block representations. Specifically, the system divides the target image into overlapping image blocks, extracts normalized features and establishes an index; after a query region is given, the localization result is generated through similarity image block retrieval, similarity filtering, and spatial merging. The authors evaluate the method from four aspects: embedding separability, unlabeled retrieval, localization consistency under query perturbation, and scalability of index size. Experimental results show that this method can learn structured multispectral representations and maintain stable localization performance and low query latency in a large-scale retrieval database.

Critical strengths

Aiming to address the reliance of traditional supervised localization methods on dense bounding box annotations.

Critical weaknesses

The paper is very difficult to understand, the writing is particularly poor and unprofessional, and it seems to have an excessive AI component.

such as, The model used for all experiments is the checkpoint saved asencoder moco resnet50 4ch ep360.pt.

Why are there so many specific numerical values in the method section?

Questions for authors

None

1 comment

We sincerely thank the reviewer for the valuable feedback and constructive observations.

We have carefully considered the concerns raised and have made the necessary revisions to improve the overall clarity, readability, and technical presentation of the manuscript. The methodological descriptions have been refined, the rationale behind the numerical parameters has been clarified, and additional attention has been given to explaining the model configuration and experimental implementation.

We have also worked on improving the organization and academic language of the manuscript to ensure that the proposed approach, its underlying decisions, and experimental procedures are communicated more clearly and transparently.

We appreciate the reviewer's critical assessment and believe that these revisions have significantly improved the quality, clarity, and comprehensibility of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.