acceptodds
Under review as a conference paper at ICLR 2027

Geo-Embed: Towards Unified Multimodal Embeddings for Urban Understanding

40 views

Abstract

Geospatial and urban applications increasingly require models to compare heterogeneous evidence across street-view imagery, remote-sensing observations, text descriptions, region proposals, and temporal change cues. However, existing multimodal embedding models and benchmarks are still largely designed and evaluated around general-purpose image-text matching, leaving unclear whether unified embedding space can support heterogeneous geospatial tasks involving spatial relationships, fine-grained semantics, and temporal changes. To address this gap, we make three key contributions. First, we introduce GeoMEB, a large-scale multimodal embedding benchmark that standardizes 45 urban evaluation tasks across retrieval, visual question answering, change detection, classification, and visual grounding, together with training collections comprising 1.32M examples and 286K evaluation queries. Second, we present Geo-Embed, a unified embedding model that adapts a shared vision-language backbone to encode heterogeneous geospatial inputs—including single or multiple images, text, regions, and masks—into a fixed-dimensional space for instruction-conditioned query-target matching. On GeoMEB, Geo-Embed achieves the strongest overall performance among representative multimodal embedders, with a 10.4% relative improvement over the strongest baseline. These results motivate future geospatial embedders that organize training and evaluation around explicit query-target relations, including semantic, cross-view, region-level, and temporal correspondence.

Primary area:
datasets and benchmarks
Keywords:
Geospatial multimodal learning, Multimodal embedding, Urban understanding, Representation learning, Multimodal retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_A5jyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GeoMEB, a geographic multimodal embedding benchmark, which converts 45 evaluation subtasks from existing data sources into a unified instruction-conditioned ranking problem, and puts forward Geo-Embed fine-tuning based on Qwen3-VL-Embedding-2B. The task setting covers cross-view retrieval, urban question answering, regional localization, change detection and urban attribute prediction, and has certain resource and empirical value. However, the current manuscript still lacks key explanations and verifications on the effectiveness of task translation, the construction of candidate pools and data splitting, as well as the source of model improvement.

Critical strengths
  • This paper incorporates scattered urban and geographic tasks into a unified embedding evaluation, covering input and target forms such as single image, multiple images, text, region, and mask, providing a useful test platform for studying the representation ability of different geographic relationships.
  • The experiment compares different model families and reports complete task-by-task results. The overall score of Geo-Embed is 25.48, which is higher than the 23.09 of Qwen3-VL-Embedding-8B and also higher than the 21.60 of its initialization model, indicating that geographic domain training delivers practical benefits.
  • The degradation of localization performance caused by multi-task mixing, as well as the different impacts of modal grouped sampling on in-domain retrieval and cross-view retrieval, are both empirical observations worthy of further exploration.
Critical weaknesses
  1. Lack of verification for the evaluation effectiveness after task translation Unifying different tasks into a ranking problem is a reasonable design for embedding benchmarks. However, this paper needs to clarify which capability requirements of the original tasks are retained and which requirements are changed by this translation. Specifically, region positioning is translated into retrieving the target from candidate crops, which relies on pre-defined candidate regions and cannot be directly equated to predicting the target position from the original image (and also reduce the level of difficulty of the original task). Change detection is reformulated into matching candidate masks, which evaluates the mask retrieval capability and does not directly evaluate the quality of pixel-level change localization. For PlacePulse and CityLens, continuous or ordered indicators are discretized and evaluated only by the precise hit R@1, and errors in adjacent bins and errors that are far apart receive the same penalty, which may not fully reflect the quality of indicator estimation.
  • Suggestion: It is recommended that the author carry out extra verification on representative tasks: for example, how the retrieved regions or masks perform under the original IoU/F1 metrics, and whether the binning prediction obtains consistent conclusions under MAE or rank correlation. The authors should also limit the scope of their conclusions to the embedded ranking versions of these tasks, avoiding direct extrapolation to the full localization, segmentation or metric estimation capabilities.
  1. The model comparison lacks reasonable baselines. The paper mainly compares open-source embedding models such as UME-R1, VLM2Vec-V2.0, OmniEmbed-Nemotron, Seed1.6-Embedding, and Qwen3-VL-Embedding, and does not include closed-source commercial models as baselines (e.g. GPT, Gemini, etc.), which weakens the current conclusions. In addition, the paper does not report the performance scores of humans on this 45-task benchmark. The task types included in GeoMEB span an extremely wide range, and some of the tasks (such as the urban perception indicator estimation SBC) are inherently subjective. Without human performance as a reference, it is difficult to judge whether the task difficulty is reasonable (whether there are tasks with too low or too high ceilings); and how large the gap between the model and human intelligence is.

  2. The the candidate pool and the correct target set are not clearly defined. The construction of the candidate pool directly determines the difficulty of the ranking task, which is the core design of this benchmark, but the current explanation is insufficient. In Table 5, the number of candidates for MME-RealWorld is 5, which seems to correspond to the options for each question; the role-based-reasoning, traffic-sign-reasoning and visual-prompt-reasoning of UrBench list 900, 400 and 200 candidates respectively, which is exactly four times the number of queries. Do these numbers represent the candidates for each question and the aggregated candidates for the entire task respectively? If a global candidate pool is adopted, how to handle duplicate options, and are all answers with the same semantics considered correct? Similar issues also involve GPS region retrieval, cropped region and change masks: multiple candidates may satisfy the same query, and duplicate or approximate masks may also exist. The paper needs to clarify the handling of candidate deduplication, positive example sets and equivalent answers. Simply adding Recall@5/10 cannot automatically solve the problem of incomplete correct target annotation.

  • Suggestion: It is recommended to disclose the task-by-task candidate pool construction rules, and carry out random, majority-class or simple heuristic baselines, as well as candidate pool sensitivity analysis for representative tasks.
  1. The handling of duplicate positive and negative examples in contrastive learning is not explained. Section 4.2 and Eq. (5) take the candidates paired with other queries in the batch as negative examples. However, in classification, binning prediction and answer retrieval, different queries may share the same correct candidate. For example, two images belonging to the same scene category may correspond to the same category text. If other candidates in the batch are directly taken as negative examples, this category text may become both a positive example and a negative example. Modality-homogeneous sub-batches may also increase the chance of such conflicts occurring.
  • Suggestion: This is a training implementation issue that needs clarification, not a basis for asserting an implementation error. The authors could explain whether target deduplication, negative example masking with the same label, or multi-positive example contrastive learning is performed, and how these processes cover different tasks. Otherwise, it is still unclear whether the training objective is consistent with the many-to-one relationship of the benchmark.
Questions for authors

There are several specific issues that require further clarification:

  • Table 1 lists the training volume of UrBench as 9,274, while the total training volume of the seven UrBench subtasks in Table 5 is 2,869. The statistical difference of the two figures are not explained.
  • The modality interaction experiment in Section 6 uses approximately 42K VIGOR samples, while the VIGOR training volume listed in Table 1 and Table 5 is 3,045. How the two sets of data are constructed and whether they belong to different experimental sets needs to be clarified.
  • Table 11 does not list the mini-batch size of the final model; Table 3 also does not fully explain its evaluation set and aggregation metrics.
  • Section 3.3 only states that cross-split of target instances is avoided. For data that generates multiple question answering, region or attribute labels from the same image, it should be clarified whether the split is grouped by the original image or location; for samples that are spatially adjacent or temporally correlated, it should also be clarified whether geographic isolation is adopted.

These issues do not directly indicate the existence of data leakage, but the current description is insufficient to rule out cross-split of original images, locations or related samples. The authors are recommended to provide verifiable training data lists, split indexes and complete experimental configurations, and distinguish between in-distribution evaluation and generalization evaluation on unseen cities or data sources.

11 comments
AuthorsOpenReview ↗

Reply to Reviewer A5jy (Part 1)

We thank the reviewer for the detailed and constructive suggestions. To address the concerns, we have conducted additional experiments and clarified the implementation details. We respond to each point below.

1. Validity of evaluation after task reformulation

GeoMEB aims to evaluate the ability of embedding models to perform diverse geospatial tasks through similarity-based matching, rather than directly replacing task-specific models for localization, change detection, or continuous-value prediction. To address concerns about the validity of task reformulation, we additionally report IoU for Grounding and Change Detection, and MAE for PlacePulse and CityLens.

DatasetQwen3-VL-Embedding 2B R@1 ↑Geo-Embed R@1 ↑Δ R@1Qwen3-VL-Embedding 2B IoU ↑Geo-Embed IoU ↑Δ IoU
Grounding tasks
VRSBench51.36555.780+4.41563.03764.624+1.587
Cityscapes42.32055.360+13.04058.47065.181+6.711
Mapillary49.92484.317+34.39368.95387.454+18.501
Change detection tasks
LEVIR-CD1.5624.687+3.1254.8477.590+2.743
SYSU-CD0.0750.800+0.72511.26012.353+1.093
VL-CMU-CD0.4661.865+1.3994.22110.555+6.334
TaskQwen3-VL-Embedding 2B R@1 ↑Geo-Embed R@1 ↑Δ R@1Qwen3-VL-Embedding 2B MAE ↓Geo-Embed MAE ↓Δ MAE
CityLens
GDP6.5008.600+2.1003.41702.6535-0.7635
Population6.90010.100+3.2003.42702.4440-0.9830
Building Height5.4008.800+3.4003.13052.2160-0.9145
Healthcare3.9008.300+4.4003.23202.0605-1.1715
PlacePulse
Beautiful5.0006.382+1.3821.84311.7246-0.1185
Boring3.5176.116+2.5991.96951.8073-0.1622
Depressing9.33910.913+1.5741.58241.4528-0.1296
Lively5.0228.415+3.3931.70621.5537-0.1525
Safety5.4109.078+3.6681.58921.4279-0.1613
Wealthy13.41114.504+1.0931.18301.1174-0.0656

The additional results indicate improvements in both R@1 and IoU across all Grounding and Change Detection tasks, with positive correlations between their performance gains (Pearson r = 0.943; Spearman ρ = 0.771). For continuous-value prediction tasks, Geo-Embed achieves higher R@1 and lower MAE across all four CityLens tasks and all six PlacePulse tasks. The correlations between R@1 gains and MAE reductions are also positive for CityLens (Pearson r = 0.964; Spearman ρ = 0.800) and PlacePulse (Pearson r = 0.823; Spearman ρ = 0.829). These findings suggest that improvements under the unified Embedding Ranking metric are generally associated with improvements in the original task metrics.

2. Closed-source model evaluation and human performance baselines

First, we would like to kindly clarify that our benchmark already includes Seed1.6-Embedding, a commercial closed-source multimodal embedding model. While OpenAI provides text embedding models, GPT models do not offer a directly comparable multimodal embedding API. We agree that incorporating additional commercial embedding models would further strengthen the benchmark's coverage.

Regarding the difficulty of SBC tasks, PlacePulse perception scores are derived from large-scale human pairwise comparisons in the original dataset and estimated using TrueSkill, rather than being manually assigned by us. GeoMEB discretizes these continuous scores at intervals of 0.5 for embedding-based ranking. Similarly, the GDP, population, building height, and healthcare accessibility indicators in CityLens are based on numerical annotations from the original dataset, which we discretize into 20 candidate bins. We conducted an additional evaluation on six PlacePulse perception tasks. Each task contains 100 questions, with three independent responses per question. The evaluation includes 12 participants, each completing 150 questions. Participants select answers from the same candidate bins as Geo-Embed, using identical visual inputs.

TaskGeo-Embed R@1 ↑Human R@1 ↑Geo-Embed MAE ↓Human MAE ↓
Beautiful6.000035.33331.72260.8365
Boring6.000017.66671.80681.1463
Depressing11.000029.33331.45150.8158
Lively8.000024.00001.54791.0072
Safety9.000042.66671.43370.6738
Wealthy15.000012.66671.11751.3524
Average9.166726.94441.51330.9720

The human baseline achieves higher R@1 and lower MAE than Geo-Embed, providing a reference for assessing SBC task difficulty and quantifying the performance gap between human and current embedding models.

AuthorsOpenReview ↗

Reply to Reviewer A5jy (Part 2)

3. Definitions of candidate pools and positive sets

We have standardized candidate pool construction and positive target identification across tasks. For MME-RealWorld and UrBench, we use task-level global candidate pools, combining candidate answers from all questions within each subtask. We first merge identical answers and then use LLM-assisted identification to merge semantically equivalent answers, updating the corresponding positive target mappings. The candidate pool sizes are summarized below.

DatasetSubtaskOriginal CandidatesAfter Exact DeduplicationAfter Semantic Merging
MME-RealWorldAutonomousDriving6,472967952
MME-RealWorldMME-HD-CN3,4201,7941,794
MME-RealWorlddiagram_and_table5,8353,8033,803
MME-RealWorldmonitoring_images3,340280270
MME-RealWorldocr_cc6,1804,4614,461
MME-RealWorldremote_sensing4,290512341
UrBenchcounting777
UrBenchobject-attribute-recognition_oe151515
UrBenchroad-understanding_oe222
UrBenchrole-based-reasoning900764745
UrBenchtraffic-sign-reasoning400245199
UrBenchvisual-prompt-reasoning200141131

By maintaining task-level global candidate pools and updating positive mappings after deduplication, we ensure that semantically equivalent answers are treated consistently during retrieval evaluation. For other tasks, we further clarify the candidate pool and positive set definitions as follows:

  • GPS Retrieval: Images are mapped to unique geographic grid cells based on their coordinates. For reverse region-to-image retrieval, multiple images within the same grid cell are treated as valid positives.
  • Grounding: We select target instances with unambiguous referring expressions during dataset construction, ensuring that each query refers to a unique target region within the corresponding image.
  • Change Mask Matching: We perform uniqueness checks and deduplication on candidate masks to ensure that each query has a clearly defined target mask. Similar but non-exact masks remain separate candidates.

We also noticed a few inaccuracies in the candidate counts reported for MME-RealWorld and UrBench in Table 5. We will correct these counts and include the corresponding construction details and deduplication statistics in the revised manuscript.

4. Sensitivity Analysis of Candidate Pool Size

To further examine the effect of candidate pool size on retrieval performance, we conducted a sensitivity analysis on VRSBench by varying the number of candidates per query. We report the performance for both image-to-text (i2t) and text-to-image (t2i) retrieval.

Candidate pool sizei2t ↑t2i ↑
10087.2085.98
20080.7578.72
50068.7066.39
1,00058.1555.83
2,00046.9145.16
5,00033.8232.26
9,350 (Full Pool)26.5524.95

Retrieval performance decreases as the candidate pool grows, highlighting the importance of candidate pool size and consistent evaluation settings for fair model comparisons.

5. OOD Evaluation

We conducted additional zero-shot evaluations on four OOD datasets, covering Classification (EuroSAT [1]), Retrieval (RSITMD [2]), VQA (RSVQA-LR [3]), and Grounding (OPT-RSVG [4]) tasks. None of them was used for fine-tuning Geo-Embed. Results are presented below.

RSVQA-LRRSITMD I2TRSITMD T2IEuroSATOPT-RSVG
#Query10,0044522,11927,00023,991
#Candidate92,1194521024,125
ModelRSVQA-LR ↑RSITMD I2T ↑RSITMD T2I ↑EuroSAT ↑OPT-RSVG ↑
Geo-Embed (Ours)56.9719.6919.5460.399.67
Qwen3-VL-Embedding 2B52.8218.3616.4757.514.80
VLM2Vec-V29.903.7618.1228.951.56
CLIP ViT-L/145.349.7311.1842.312.61

Geo-Embed achieves the best performance across all evaluation metrics, outperforming Qwen3-VL-Embedding 2B, VLM2Vec-V2, and CLIP. Compared with Qwen3-VL-Embedding 2B, it improves VQA, bidirectional image-text retrieval, and classification by 4.15%, 1.33%/3.07%, and 2.88%, respectively, while nearly doubling Grounding R@1 on OPT-RSVG from 4.80% to 9.67%. These results demonstrate that Geo-Embed's geospatial representations transfer effectively to external datasets not used during fine-tuning, with consistent improvements across different task types.

AuthorsOpenReview ↗

Reply to Reviewer A5jy (Part 3)

6. Duplicate positives in contrastive learning

Our implementation follows the standard in-batch InfoNCE training protocol adopted by the VLM2Vec series. When multiple queries share an identical target, their target embeddings are identical under deterministic encoding and are not directly pushed apart during contrastive learning. We will clarify this training protocol in the revised manuscript.

7. Clarification of dataset statistics, experimental settings, and data splits

We will clarify the following details in the revised manuscript.

  • UrBench statistics: The training volume of 9,274 in Table 1 contains a minor typo. The correct number is 2,869, consistent with Table 5.

  • VIGOR data: The two experiments use different data configurations. The main benchmark uses 3,045 training samples, while the modality interaction experiment uses 52,595 valid samples from four cities, split into 42,076 training and 10,519 evaluation samples.

  • Training and evaluation settings: The Geo-Embed model uses a per-device mini-batch size of 4. The Sub32–Sub256 configurations in Table 3 refer to sub-batch organization sizes, and the reported scores are averaged over the corresponding GeoMEB subtasks within each modality using the standard evaluation protocol.

  • Data splits: We follow the official dataset splits to ensure fair comparisons with existing methods. For datasets such as VRSBench that contribute multiple subtasks, we additionally ensure image-level separation across training and evaluation, preventing the same original image from appearing in both splits.

References

[1] Helber P, Bischke B, Dengel A, et al. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12(7): 2217-2226.

[2] Yuan Z, Zhang W, Fu K, et al. Exploring a fine-grained multiscale method for cross-modal remote sensing image retrieval[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-19.

[3] Lobry S, Marcos D, Murray J, et al. RSVQA: Visual question answering for remote sensing data[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(12): 8555-8566.

[4] Li K, Wang D, Xu H, et al. Language-guided progressive attention for visual grounding in remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 1-13.

We sincerely thank the reviewer once again for the time and effort devoted to reviewing our work. We greatly appreciate the thoughtful feedback and would be delighted to discuss any further questions or suggestions.

Reviewer_UoXSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a unified multimodal embedding framework for urban understanding, named Geo-Embed, and constructs a benchmark called GeoMEB covering 45 tasks such as retrieval, question answering, classification, object localization, and change detection. The method is based on Qwen3-VL-Embedding. Through instruction guidance and contrastive learning, street views, remote sensing images, text, regions, and masks are mapped to a unified vector space, converting different tasks into query and candidate matching problems. Experiments show that this model achieves the best overall performance, while also revealing challenges such as cross-view alignment, fine-grained localization, and multi-task negative transfer.

Critical strengths
  1. GeoMEB integrates street view, remote sensing, cross-view, regional and temporal tasks into a unified evaluation interface, facilitating the comparison of different embedding models.

  2. A large number of validation tasks in the remote sensing field have been collected.

3.The article is written in a clear and understandable.

Critical weaknesses
  1. The core process of GeoMEB basically follows the instruction-based VLM embedding: input serialization, shared encoder, end-token pooling, cosine similarity, and InfoNCE. It mainly applies the existing VLM-Embedding method, lacking innovation.

  2. The main table mainly shows that the models trained in the domain are superior to the existing models, but no complete comparison with the same data and budget has been established: for example, can the VLM2Vec-V2 training process be directly applied to GeoMEB and achieve similar results?

  3. The motivation of the paper highlights cross-perspective correspondence, spatial granularity, and temporal changes, but the model still mainly delegates these relationships to natural language instructions and general contrastive loss learning, without explicitly adding new constraints or mechanisms for these relationships.

  4. The paper includes zero-shot evaluation of Im2GPS3k, but still lacks systematic cross-city, cross-dataset, and unseen-task verification. Combined with the obvious task negative transfer, and the small-scale ablation gains lacking error ranges from multiple random seeds, the current evidence more supports "multi-task adaptation under a unified interface", with limited support for generalization ability.

Questions for authors
  1. Compared with VLM2Vec, what are the unique incremental technologies of Geo-Embed?

  2. Under the same backbone, GeoMEB data, training steps and computing budget, what is the result of directly reusing the results of the existing training process?

  3. How can it be proved that the model has learned transferable geographical spatial relationships rather than specific data and instruction templates' correspondences?

8 comments
AuthorsOpenReview ↗

Reply to Reviewer UoXS (Part 1)

We sincerely thank the reviewer for the careful evaluation of our manuscript and the constructive feedback. We appreciate the reviewer's recognition of GeoMEB's value in unified geospatial multimodal evaluation, as well as our analysis of the limitations of existing embedding models. We address the reviewer's comments point by point below.

1. Contributions and technical novelty of GeoMEB and Geo-Embed

The core contribution of our work is a unified evaluation framework for geospatial multimodal representations. Real-world urban applications involve massive, heterogeneous geospatial data, yet existing approaches often address localization, change detection, and grounding through separate task-specific models. Even existing geospatial embedding methods typically focus on individual modalities or tasks, limiting their ability to support unified retrieval across diverse data sources.

To address this gap, GeoMEB establishes a unified instruction-conditioned Query-Target Ranking framework that brings heterogeneous geospatial data and tasks into a common retrieval paradigm. By promoting a shared embedding space, our framework aims to enable scalable retrieval of geographic locations, spatial changes, and relevant observations, with potential applications in disaster monitoring, emergency response, and urban information retrieval. Unlike general-purpose embedding benchmarks, GeoMEB systematically evaluates challenges such as cross-view correspondence, scale variations, fine-grained localization, and multi-temporal change understanding. Building on this framework, Geo-Embed investigates unified geospatial representation learning through multi-task contrastive adaptation, providing insights into the potential and limitations of shared geospatial embeddings.

2. Fairness of comparisons between Geo-Embed and zero-shot baselines

To ensure a fair comparison, we fine-tuned CLIP and VLM2Vec-V2 using the same training data splits, training steps and computing budget, and evaluated all models under GeoMEB. The results show that, even after fine-tuning on the same geospatial training data, Geo-Embed still outperforms the fine-tuned VLM2Vec-V2 and CLIP models by 4.64% and 7.67%, respectively, in overall performance. These results demonstrate Geo-Embed's stronger overall capability in geospatial multimodal representation learning.

ModelClassification ↑Retrieval ↑VQA ↑Grounding ↑Change Detection ↑Overall ↑
Geo-Embed (Ours)17.1744.6643.5610.4211.5925.48
Fine-tuned VLM2Vec-V220.5739.6830.207.546.1820.84
Fine-tuned CLIP20.0232.3418.267.6610.7917.81
VLM2Vec-V213.5035.9619.654.127.8716.22
CLIP11.7419.6510.122.223.139.37
AuthorsOpenReview ↗

Reply to Reviewer UoXS (Part 2)

3. Cross-dataset generalization, task transfer, and ablation stability

To further investigate the generalization capabilities of Geo-Embed, we conducted additional zero-shot evaluations on four external datasets covering Classification (EuroSAT [1]), Retrieval (RSITMD [2]), VQA (RSVQA-LR [3]), and Grounding (OPT-RSVG [4]) tasks, and all evaluations follow the unified Query-Target Ranking paradigm without additional training.

RSVQA-LRRSITMD I2TRSITMD T2IEuroSATOPT-RSVG
#Query10,0044522,11927,00023,991
#Candidate92,1194521024,125
ModelRSVQA-LR ↑RSITMD I2T ↑RSITMD T2I ↑EuroSAT ↑OPT-RSVG ↑
Geo-Embed (Ours)56.9719.6919.5460.399.67
Qwen3-VL-Embedding 2B52.8218.3616.4757.514.80
VLM2Vec-V29.903.7618.1228.951.56
CLIP ViT-L/145.349.7311.1842.312.61

Geo-Embed achieves the best performance across all evaluation metrics. Compared with the original Qwen3-VL-Embedding-2B, it consistently improves VQA, bidirectional image-text retrieval, Classification, and Grounding. These consistent improvements across external datasets and diverse geospatial tasks suggest that Geo-Embed learns transferable geospatial representations rather than merely relying on dataset-specific patterns or instruction-template associations.

Regarding task transfer, related geospatial tasks can provide complementary information and facilitate cross-task knowledge transfer. As shown in Figure 5(a), joint training improves Cross-view retrieval performance. Similarly, Figure 5(b) shows that joint training improves T2I Retrieval, I2T Retrieval, and VQA. These improvements may arise from complementary spatial information across tasks: fine-grained Grounding supervision provides local spatial details that can enrich the broader semantic representations learned by Retrieval and VQA, thereby benefiting these tasks. Conversely, broader semantic objectives may interfere with precise localization, highlighting the need to balance different representation requirements in unified embedding learning.

To assess the stability of our ablation results, we repeated the interleaved sub-batch experiments in Table 3 using five random seeds and report the mean performance and standard deviation below.

ModalitySub0Sub32Sub64Sub128Sub256
RS21.40 ± 0.2821.62 ± 0.3121.65 ± 0.2421.44 ± 0.3522.05 ± 0.27
SV27.72 ± 0.4227.18 ± 0.3828.32 ± 0.4627.68 ± 0.3328.61 ± 0.40
Cross-view29.86 ± 0.3629.78 ± 0.2929.02 ± 0.3429.42 ± 0.4129.55 ± 0.32
Document40.82 ± 0.6340.91 ± 0.5240.63 ± 0.7140.76 ± 0.5841.34 ± 0.66

The relatively small standard deviations indicate stable performance across random seeds. These results suggest that modality-based sub-batch sampling can benefit several geospatial modalities.

References

[1] Helber P, Bischke B, Dengel A, et al. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12(7): 2217-2226.

[2] Yuan Z, Zhang W, Fu K, et al. Exploring a fine-grained multiscale method for cross-modal remote sensing image retrieval[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-19.

[3] Lobry S, Marcos D, Murray J, et al. RSVQA: Visual question answering for remote sensing data[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(12): 8555-8566.

[4] Li K, Wang D, Xu H, et al. Language-guided progressive attention for visual grounding in remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 1-13.

We sincerely thank the reviewer once again for the time and effort devoted to reviewing our work. We greatly appreciate the thoughtful feedback and would be delighted to discuss any further questions or suggestions.

Thank you for the detailed response and additional experiments. The rebuttal addresses some of my previous concerns, particularly regarding baseline fairness and cross-dataset evaluation. However, a few issues remain:

  1. The response does not clearly explain the technical differences between Geo-Embed and existing methods such as VLM2Vec-V2. The main contribution still appears to be benchmark construction and domain-specific adaptation.

  2. Although Geo-Embed achieves a higher overall score, fine-tuned VLM2Vec-V2 performs better on classification (20.57 vs. 17.17). The authors do not discuss this result or explain why Geo-Embed underperforms on this task. A more detailed analysis of these performance differences would help clarify the benefits and limitations of the proposed approach.

  3. The additional OOD results are useful, but they do not fully demonstrate that the model learns transferable geographic spatial relationships. The gains from sub-batching are also relatively small and inconsistent across modalities, particularly for cross-view tasks.

Reviewer_K41wOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces GeoMEB, a geospatial multimodal embedding benchmark that unifies existing remote sensing, street-view, and cross-view datasets into instruction-conditioned ranking tasks covering retrieval, VQA, grounding, change detection, and classification. It also proposes Geo-Embed, a model fine-tuned on the benchmark's training data using LoRA and contrastive learning. Experiments show that Geo-Embed outperforms existing CLIP-based, instruction-tuned, and embedding-specialized models.

Critical strengths
  • GeoMEB provides a unified evaluation framework for diverse geospatial embedding tasks.
  • The experiments reveal limitations of existing embedding models in cross-view matching, grounding, and change detection.
Critical weaknesses
  • The benchmark mainly collects and reformats existing datasets.
  • Formulating all tasks as retrieval is questionable. For example, change detection is cast as selecting the correct mask from a candidate pool, which is much easier than estimating an unknown change mask.
  • The need to handle all tasks with a single embedding model is not justified, and no comparison with task-specific models is provided.
  • Challenges specific to geospatial data, and technical ideas to address them, are not presented. Geo-Embed follows the VLM2Vec recipe, and its only new component, modality-homogeneous sub-batching, changes results by less than one point (Table 3).
  • The comparison is not fair. Geo-Embed is fine-tuned on the training splits of 41 of the 45 tasks, whereas all baselines are evaluated zero-shot. Baselines fine-tuned on the same data, or evaluation on held-out tasks, are needed.
  • Multi-task training hurts some tasks (VRSBench grounding drops from 14.0 to 4.1; Figure 5b), which weakens the motivation for a unified model.
Questions for authors
  • How does Geo-Embed compare with existing embedding models fine-tuned on the same GeoMEB training data using a comparable training setup?
  • What is the technical difference between the interleaved sub-batching strategy in Geo-Embed and that of VLM2Vec-V2?
  • How do the retrieval-based formulations of change detection and grounding relate to performance on the original tasks?
  • Are the training and evaluation splits disjoint at the image level across all tasks derived from the same source dataset, particularly VRSBench?
  • How does Geo-Embed perform on entirely unseen datasets or task types?
  • How sensitive are the overall rankings to the aggregation of task scores?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer K41w (Part 1)

We sincerely thank the reviewer for the careful evaluation of our manuscript and the constructive feedback. We appreciate the reviewer's recognition of GeoMEB's value in unified geospatial multimodal evaluation. We address the reviewer's comments and questions point by point below.

1. Contributions and novelty of GeoMEB and Geo-Embed

The core contribution of our work is a unified evaluation framework for geospatial multimodal representations. Real-world urban applications involve massive, heterogeneous geospatial data, yet existing approaches often address localization, change detection, and grounding through separate task-specific models. Even existing geospatial embedding methods typically focus on individual modalities or tasks, limiting their ability to support unified retrieval across diverse data sources.

To address this gap, GeoMEB establishes a unified instruction-conditioned Query-Target Ranking framework that brings heterogeneous geospatial data and tasks into a common retrieval paradigm. By promoting a shared embedding space, our framework aims to enable scalable retrieval of geographic locations, spatial changes, and relevant observations, with potential applications in disaster monitoring, emergency response, and urban information retrieval. Unlike general-purpose embedding benchmarks, GeoMEB systematically evaluates challenges such as cross-view correspondence, scale variations, fine-grained localization, and multi-temporal change understanding. Building on this framework, Geo-Embed investigates unified geospatial representation learning through multi-task contrastive adaptation, providing insights into the potential and limitations of shared geospatial embeddings.

2. Validity of retrieval-based task reformulation and its relationship to performance on the original tasks

GeoMEB reformulates heterogeneous geospatial tasks into a unified embedding retrieval paradigm, enabling systematic evaluation of multimodal models through a shared Query-Target Ranking interface. Specifically, Grounding is formulated as image-text-region matching to assess fine-grained spatial correspondence, while Change Detection involves retrieving change masks from bi-temporal images to evaluate temporal and spatial change representations. To further validate this retrieval-based evaluation, we conducted additional experiments comparing R@1 and IoU on both tasks.

For Grounding, we adopt a within-image candidate region retrieval protocol. We retain the original image and text query and select all candidate regions associated with the same image from the existing GeoMEB test candidate pool. To ensure that each query provides a meaningful retrieval evaluation, we include only queries with at least one negative candidate. The model selects the Top-1 region based on embedding similarity, and we compute both the positive retrieval hit rate (R@1) and the IoU between the retrieved region and the annotated positive bounding boxes. This evaluation includes 11,393, 2,500, and 6,606 queries from VRSBench, Cityscapes, and Mapillary, respectively. And for Change Detection, we use the complete test sets and the original candidate pools. We compute both Top-1 retrieval accuracy (R@1) and the pixel-level IoU between the retrieved change mask and the ground-truth change mask.

  • Grounding:
DatasetQwen3-VL-Embedding 2B R@1 ↑Geo-Embed R@1 ↑Δ R@1Qwen3-VL-Embedding 2B IoU ↑Geo-Embed IoU ↑Δ IoU
VRSBench51.36555.780+4.41563.03764.624+1.587
Cityscapes42.32055.360+13.04058.47065.181+6.711
Mapillary49.92484.317+34.39368.95387.454+18.501
  • Change Detection:
DatasetQwen3-VL-Embedding 2B R@1 ↑Geo-Embed R@1 ↑Δ R@1Qwen3-VL-Embedding 2B IoU ↑Geo-Embed IoU ↑Δ IoU
LEVIR-CD1.5624.687+3.1254.8477.590+2.743
SYSU-CD0.0750.800+0.72511.26012.353+1.093
VL-CMU-CD0.4661.865+1.3994.22110.555+6.334

The experiments show that Geo-Embed achieves improvements in both R@1 and IoU across all six Grounding and Change Detection tasks. Furthermore, correlation analysis of the performance gains across these six tasks yields a Pearson correlation coefficient of 0.943 and a Spearman rank correlation coefficient of 0.771, indicating an overall positive correlation between improvements in the two metrics. These findings suggest that gains in retrieval accuracy are generally accompanied by improvements in the spatial overlap quality of retrieved regions or change masks, providing additional evidence supporting the validity of retrieval-based evaluation.

AuthorsOpenReview ↗

Reply to Reviewer K41w (Part 2)

3. Necessity of a unified embedding model

Compared with task-specific embedding models, unified embedding models offer two key advantages. First, they support heterogeneous geospatial tasks through a shared embedding space, reducing the cost of training and maintaining separate models. Second, related tasks can provide complementary information and facilitate cross-task knowledge transfer. As shown in Figure 5(a), joint training improves Cross-view retrieval performance. Similarly, Figure 5(b) shows that joint training improves T2I Retrieval, I2T Retrieval, and VQA tasks. These improvements may arise from complementary spatial information across tasks: fine-grained Grounding supervision provides local spatial details that can enrich the broader semantic representations learned by Retrieval and VQA, thereby benefiting these tasks. Conversely, broader semantic objectives may interfere with precise localization, highlighting the need to balance different representation requirements in unified embedding learning.

4. Fairness of comparisons between Geo-Embed and zero-shot baselines

To ensure a fair comparison, we fine-tuned CLIP and VLM2Vec-V2 using the same training data splits and hyperparameters, and evaluated all models under GeoMEB. The results show that, even after fine-tuning on the same geospatial training data, Geo-Embed still outperforms the fine-tuned VLM2Vec-V2 and CLIP models by 4.64% and 7.67%, respectively, in overall performance. These results demonstrate Geo-Embed's stronger overall capability in geospatial multimodal representation learning.

ModelClassification ↑Retrieval ↑VQA ↑Grounding ↑Change Detection ↑Overall ↑
Geo-Embed (Ours)17.1744.6643.5610.4211.5925.48
Fine-tuned VLM2Vec-V220.5739.6830.207.546.1820.84
Fine-tuned CLIP20.0232.3418.267.6610.7917.81
VLM2Vec-V213.5035.9619.654.127.8716.22
CLIP11.7419.6510.122.223.139.37

5. Differences between Geo-Embed's interleaved sub-batching and VLM2Vec-V2

Geo-Embed and VLM2Vec-V2 have different research focuses regarding training data composition. VLM2Vec-V2 explores interleaved sub-batching for general-purpose multimodal embedding learning, whereas Geo-Embed applies this training strategy to geospatial multimodal scenarios, focusing on how the organization of different modalities, including remote sensing, street-view imagery, cross-view data, and documents, affects unified embedding learning.

We identified an implementation error in our original sub-batch ablation: varying the sub-batch size did not alter the negative sample composition as intended, rendering the original results invalid. We have corrected the implementation and rerun the experiments. The updated results are presented below.

ModalitySub0Sub32Sub64Sub128Sub256
RS21.37 ± 0.3721.68 ± 0.2222.04 ± 0.4522.41 ± 0.2922.96 ± 0.41
SV27.64 ± 0.3328.03 ± 0.4728.47 ± 0.2528.96 ± 0.4429.75 ± 0.36
Cross-view29.81 ± 0.2831.13 ± 0.4232.79 ± 0.3531.47 ± 0.2430.62 ± 0.48
Document40.86 ± 0.6740.71 ± 0.5441.58 ± 0.6942.77 ± 0.5844.41 ± 0.63

The corrected results show substantial performance improvements with modality-based sub-batch sampling, confirming its effectiveness in geospatial multimodal embedding learning. We will update the ablation results accordingly in the revised manuscript.

AuthorsOpenReview ↗

Reply to Reviewer K41w (Part 3)

6. Image-level separation between training and evaluation sets

Our dataset splits strictly follow image-level separation between training and evaluation. For source datasets such as VRSBench that are used to construct multiple subtasks, we perform cross-task splitting at the original image level, ensuring that the same original image cannot appear in both training and evaluation subsets, regardless of the subtask. Therefore, even when the same source dataset is used to construct different tasks, such as Caption Retrieval, VQA, and Visual Grounding, there is no training-evaluation data leakage caused by repeated original images. We will clarify this in the revised manuscript.

7. Zero-shot generalization to external datasets

We conducted additional zero-shot evaluations on four OOD datasets, covering Classification (EuroSAT [1]), Retrieval (RSITMD [2]), VQA (RSVQA-LR [3]), and Grounding (OPT-RSVG [4]) tasks. None of them was used for fine-tuning Geo-Embed. Results are presented below.

RSVQA-LRRSITMD I2TRSITMD T2IEuroSATOPT-RSVG
#Query10,0044522,11927,00023,991
#Candidate92,1194521024,125
ModelRSVQA-LR ↑RSITMD I2T ↑RSITMD T2I ↑EuroSAT ↑OPT-RSVG ↑
Geo-Embed (Ours)56.9719.6919.5460.399.67
Qwen3-VL-Embedding 2B52.8218.3616.4757.514.80
VLM2Vec-V29.903.7618.1228.951.56
CLIP ViT-L/145.349.7311.1842.312.61

Geo-Embed achieves the best performance across all evaluation metrics, outperforming Qwen3-VL-Embedding 2B, VLM2Vec-V2, and CLIP. Compared with Qwen3-VL-Embedding 2B, it improves VQA, bidirectional image-text retrieval, and classification by 4.15%, 1.33%/3.07%, and 2.88%, respectively, while nearly doubling Grounding R@1 on OPT-RSVG from 4.80% to 9.67%. These results demonstrate that Geo-Embed's geospatial representations transfer effectively to external datasets not used during fine-tuning, with consistent improvements across different task types.

8. Sensitivity of model rankings to task score aggregation strategies

In addition to the Meta-task Equal Weighting strategy used earlier, we added Task-level Equal Weighting, which assigns equal weight to each of the 45 subtasks when computing the overall score. The results are presented below.

ModelOriginal Score ↑Original RankTask-level Score ↑Task-level Rank
Geo-Embed (Ours)25.48131.791
Qwen3-VL-Embedding 8B23.09229.822
Qwen3-VL-Embedding 2B21.60327.973
Seed1.6-Embedding21.44426.724
UME-R117.27521.745
VLM2Vec-V2.016.22620.646
OpenCLIP10.92814.847
GeoRSCLIP10.70914.478
OmniEmbed-Nemotron11.76714.429
SkyScript-CLIP9.371012.5410
SigLIP29.371012.5111
CLIP8.821211.8812
RemoteCLIP7.611310.5013
Qwen3-VL-Instruct 8B6.49147.4814
Qwen3-VL-Instruct 2B6.38157.0015
GeoChat4.83166.6816

Geo-Embed remains the top-ranked model under new aggregation strategies, achieving a score of 31.79, while the top six models retain same rankings across both aggregation strategies. These results demonstrate the robustness of Geo-Embed's overall performance advantage to different aggregation methods.

References

[1] Helber P, Bischke B, Dengel A, et al. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12(7): 2217-2226.

[2] Yuan Z, Zhang W, Fu K, et al. Exploring a fine-grained multiscale method for cross-modal remote sensing image retrieval[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-19.

[3] Lobry S, Marcos D, Murray J, et al. RSVQA: Visual question answering for remote sensing data[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(12): 8555-8566.

[4] Li K, Wang D, Xu H, et al. Language-guided progressive attention for visual grounding in remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 1-13.

We sincerely thank the reviewer once again for the time and effort devoted to reviewing our work. We greatly appreciate the thoughtful feedback and would be delighted to discuss any further questions or suggestions!

AuthorsOpenReview ↗

A Gentle Reminder Before the Rebuttal Deadline

Dear reviewer,

Thank you again for your constructive feedback. As the rebuttal window is closing soon, we genuinely appreciate your feedback. We hope our response has addressed your concerns. Your feedback would be invaluable in making this work even more compelling and worthy of acceptance.

Best regards, The Authors

Comments on the paper

2 comments

Dear Reviewer,

Please take a moment to review the authors' feedback and updates regarding your assigned paper. Your timely participation in the discussion phase is greatly appreciated.

Best regards,

Your AC

AuthorsOpenReview ↗

General Response from Authors

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair for coordinating the review process and facilitating the discussion, and all reviewers for their thoughtful feedback and valuable suggestions. During the rebuttal period, we have conducted additional experiments, analyses, and clarifications to further strengthen our work. We summarize the updates below.

  • Practical Value of Unified Geospatial Embeddings (K41w, UoXS, A5jy). We further clarified the practical value of unified geospatial embeddings in enabling scalable multimodal retrieval and geographic search. We also highlighted their advantages in precomputation, indexing, and reuse across different downstream tasks, complementing generative MLLMs and task-specific models.

  • Fair Comparisons and Downstream Transferability (K41w, UoXS, A5jy). We conducted additional comparisons under matched training settings and extended our evaluation to conventional downstream tasks using a broader range of foundation models. These experiments also provide evidence that Geo-Embed's improvements stem from our geospatial representation training.

  • Generalization and Spatial Understanding (K41w, UoXS, A5jy). We conducted additional zero-shot evaluations which demonstrated strong generalization to unseen datasets, geographic regions, spatial relationships, and instruction formulations.

  • Evaluation Validity and Benchmark Construction (K41w, A5jy). We introduced original-task metrics to further validate our retrieval-based evaluation. We also clarified candidate pool construction, positive target definitions, data splits, and experimental settings, and conducted candidate pool sensitivity analyses.

  • Human Evaluation (A5jy). We expanded human participation across representative tasks to better characterize the gap between human and model performance.

  • Ablation Analysis and Evaluation Robustness (K41w, UoXS). We repeated the sub-batch ablation experiments with multiple random seeds, revealing substantial performance improvements across different geospatial modalities. We also examined the robustness of model rankings under alternative score aggregation strategies.

We believe these additions provide stronger evidence for the contributions, practical utility, and evaluation reliability of GeoMEB and Geo-Embed.

We are deeply grateful for your valuable insights and careful consideration throughout the discussion. We sincerely hope that our extensive efforts have addressed your concerns and inspired greater confidence in the merits of our work, encouraging you to champion its contributions in the final assessment.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.