- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes GeoMEB, a geographic multimodal embedding benchmark, which converts 45 evaluation subtasks from existing data sources into a unified instruction-conditioned ranking problem, and puts forward Geo-Embed fine-tuning based on Qwen3-VL-Embedding-2B. The task setting covers cross-view retrieval, urban question answering, regional localization, change detection and urban attribute prediction, and has certain resource and empirical value. However, the current manuscript still lacks key explanations and verifications on the effectiveness of task translation, the construction of candidate pools and data splitting, as well as the source of model improvement.
Critical strengths
- This paper incorporates scattered urban and geographic tasks into a unified embedding evaluation, covering input and target forms such as single image, multiple images, text, region, and mask, providing a useful test platform for studying the representation ability of different geographic relationships.
- The experiment compares different model families and reports complete task-by-task results. The overall score of Geo-Embed is 25.48, which is higher than the 23.09 of Qwen3-VL-Embedding-8B and also higher than the 21.60 of its initialization model, indicating that geographic domain training delivers practical benefits.
- The degradation of localization performance caused by multi-task mixing, as well as the different impacts of modal grouped sampling on in-domain retrieval and cross-view retrieval, are both empirical observations worthy of further exploration.
Critical weaknesses
- Lack of verification for the evaluation effectiveness after task translation Unifying different tasks into a ranking problem is a reasonable design for embedding benchmarks. However, this paper needs to clarify which capability requirements of the original tasks are retained and which requirements are changed by this translation. Specifically, region positioning is translated into retrieving the target from candidate crops, which relies on pre-defined candidate regions and cannot be directly equated to predicting the target position from the original image (and also reduce the level of difficulty of the original task). Change detection is reformulated into matching candidate masks, which evaluates the mask retrieval capability and does not directly evaluate the quality of pixel-level change localization. For PlacePulse and CityLens, continuous or ordered indicators are discretized and evaluated only by the precise hit R@1, and errors in adjacent bins and errors that are far apart receive the same penalty, which may not fully reflect the quality of indicator estimation.
- Suggestion: It is recommended that the author carry out extra verification on representative tasks: for example, how the retrieved regions or masks perform under the original IoU/F1 metrics, and whether the binning prediction obtains consistent conclusions under MAE or rank correlation. The authors should also limit the scope of their conclusions to the embedded ranking versions of these tasks, avoiding direct extrapolation to the full localization, segmentation or metric estimation capabilities.
-
The model comparison lacks reasonable baselines. The paper mainly compares open-source embedding models such as UME-R1, VLM2Vec-V2.0, OmniEmbed-Nemotron, Seed1.6-Embedding, and Qwen3-VL-Embedding, and does not include closed-source commercial models as baselines (e.g. GPT, Gemini, etc.), which weakens the current conclusions. In addition, the paper does not report the performance scores of humans on this 45-task benchmark. The task types included in GeoMEB span an extremely wide range, and some of the tasks (such as the urban perception indicator estimation SBC) are inherently subjective. Without human performance as a reference, it is difficult to judge whether the task difficulty is reasonable (whether there are tasks with too low or too high ceilings); and how large the gap between the model and human intelligence is.
-
The the candidate pool and the correct target set are not clearly defined. The construction of the candidate pool directly determines the difficulty of the ranking task, which is the core design of this benchmark, but the current explanation is insufficient. In Table 5, the number of candidates for MME-RealWorld is 5, which seems to correspond to the options for each question; the role-based-reasoning, traffic-sign-reasoning and visual-prompt-reasoning of UrBench list 900, 400 and 200 candidates respectively, which is exactly four times the number of queries. Do these numbers represent the candidates for each question and the aggregated candidates for the entire task respectively? If a global candidate pool is adopted, how to handle duplicate options, and are all answers with the same semantics considered correct? Similar issues also involve GPS region retrieval, cropped region and change masks: multiple candidates may satisfy the same query, and duplicate or approximate masks may also exist. The paper needs to clarify the handling of candidate deduplication, positive example sets and equivalent answers. Simply adding Recall@5/10 cannot automatically solve the problem of incomplete correct target annotation.
- Suggestion: It is recommended to disclose the task-by-task candidate pool construction rules, and carry out random, majority-class or simple heuristic baselines, as well as candidate pool sensitivity analysis for representative tasks.
- The handling of duplicate positive and negative examples in contrastive learning is not explained. Section 4.2 and Eq. (5) take the candidates paired with other queries in the batch as negative examples. However, in classification, binning prediction and answer retrieval, different queries may share the same correct candidate. For example, two images belonging to the same scene category may correspond to the same category text. If other candidates in the batch are directly taken as negative examples, this category text may become both a positive example and a negative example. Modality-homogeneous sub-batches may also increase the chance of such conflicts occurring.
- Suggestion: This is a training implementation issue that needs clarification, not a basis for asserting an implementation error. The authors could explain whether target deduplication, negative example masking with the same label, or multi-positive example contrastive learning is performed, and how these processes cover different tasks. Otherwise, it is still unclear whether the training objective is consistent with the many-to-one relationship of the benchmark.
Questions for authors
There are several specific issues that require further clarification:
- Table 1 lists the training volume of UrBench as 9,274, while the total training volume of the seven UrBench subtasks in Table 5 is 2,869. The statistical difference of the two figures are not explained.
- The modality interaction experiment in Section 6 uses approximately 42K VIGOR samples, while the VIGOR training volume listed in Table 1 and Table 5 is 3,045. How the two sets of data are constructed and whether they belong to different experimental sets needs to be clarified.
- Table 11 does not list the mini-batch size of the final model; Table 3 also does not fully explain its evaluation set and aggregation metrics.
- Section 3.3 only states that cross-split of target instances is avoided. For data that generates multiple question answering, region or attribute labels from the same image, it should be clarified whether the split is grouped by the original image or location; for samples that are spatially adjacent or temporally correlated, it should also be clarified whether geographic isolation is adopted.
These issues do not directly indicate the existence of data leakage, but the current description is insufficient to rule out cross-split of original images, locations or related samples. The authors are recommended to provide verifiable training data lists, split indexes and complete experimental configurations, and distinguish between in-distribution evaluation and generalization evaluation on unseen cities or data sources.
11 comments
Reply to Reviewer A5jy (Part 2)
3. Definitions of candidate pools and positive sets
We have standardized candidate pool construction and positive target identification across tasks. For MME-RealWorld and UrBench, we use task-level global candidate pools, combining candidate answers from all questions within each subtask. We first merge identical answers and then use LLM-assisted identification to merge semantically equivalent answers, updating the corresponding positive target mappings. The candidate pool sizes are summarized below.
| Dataset | Subtask | Original Candidates | After Exact Deduplication | After Semantic Merging |
|---|---|---|---|---|
| MME-RealWorld | AutonomousDriving | 6,472 | 967 | 952 |
| MME-RealWorld | MME-HD-CN | 3,420 | 1,794 | 1,794 |
| MME-RealWorld | diagram_and_table | 5,835 | 3,803 | 3,803 |
| MME-RealWorld | monitoring_images | 3,340 | 280 | 270 |
| MME-RealWorld | ocr_cc | 6,180 | 4,461 | 4,461 |
| MME-RealWorld | remote_sensing | 4,290 | 512 | 341 |
| UrBench | counting | 7 | 7 | 7 |
| UrBench | object-attribute-recognition_oe | 15 | 15 | 15 |
| UrBench | road-understanding_oe | 2 | 2 | 2 |
| UrBench | role-based-reasoning | 900 | 764 | 745 |
| UrBench | traffic-sign-reasoning | 400 | 245 | 199 |
| UrBench | visual-prompt-reasoning | 200 | 141 | 131 |
By maintaining task-level global candidate pools and updating positive mappings after deduplication, we ensure that semantically equivalent answers are treated consistently during retrieval evaluation. For other tasks, we further clarify the candidate pool and positive set definitions as follows:
- GPS Retrieval: Images are mapped to unique geographic grid cells based on their coordinates. For reverse region-to-image retrieval, multiple images within the same grid cell are treated as valid positives.
- Grounding: We select target instances with unambiguous referring expressions during dataset construction, ensuring that each query refers to a unique target region within the corresponding image.
- Change Mask Matching: We perform uniqueness checks and deduplication on candidate masks to ensure that each query has a clearly defined target mask. Similar but non-exact masks remain separate candidates.
We also noticed a few inaccuracies in the candidate counts reported for MME-RealWorld and UrBench in Table 5. We will correct these counts and include the corresponding construction details and deduplication statistics in the revised manuscript.
4. Sensitivity Analysis of Candidate Pool Size
To further examine the effect of candidate pool size on retrieval performance, we conducted a sensitivity analysis on VRSBench by varying the number of candidates per query. We report the performance for both image-to-text (i2t) and text-to-image (t2i) retrieval.
| Candidate pool size | i2t ↑ | t2i ↑ |
|---|---|---|
| 100 | 87.20 | 85.98 |
| 200 | 80.75 | 78.72 |
| 500 | 68.70 | 66.39 |
| 1,000 | 58.15 | 55.83 |
| 2,000 | 46.91 | 45.16 |
| 5,000 | 33.82 | 32.26 |
| 9,350 (Full Pool) | 26.55 | 24.95 |
Retrieval performance decreases as the candidate pool grows, highlighting the importance of candidate pool size and consistent evaluation settings for fair model comparisons.
5. OOD Evaluation
We conducted additional zero-shot evaluations on four OOD datasets, covering Classification (EuroSAT [1]), Retrieval (RSITMD [2]), VQA (RSVQA-LR [3]), and Grounding (OPT-RSVG [4]) tasks. None of them was used for fine-tuning Geo-Embed. Results are presented below.
| RSVQA-LR | RSITMD I2T | RSITMD T2I | EuroSAT | OPT-RSVG | |
|---|---|---|---|---|---|
| #Query | 10,004 | 452 | 2,119 | 27,000 | 23,991 |
| #Candidate | 9 | 2,119 | 452 | 10 | 24,125 |
| Model | RSVQA-LR ↑ | RSITMD I2T ↑ | RSITMD T2I ↑ | EuroSAT ↑ | OPT-RSVG ↑ |
|---|---|---|---|---|---|
| Geo-Embed (Ours) | 56.97 | 19.69 | 19.54 | 60.39 | 9.67 |
| Qwen3-VL-Embedding 2B | 52.82 | 18.36 | 16.47 | 57.51 | 4.80 |
| VLM2Vec-V2 | 9.90 | 3.76 | 18.12 | 28.95 | 1.56 |
| CLIP ViT-L/14 | 5.34 | 9.73 | 11.18 | 42.31 | 2.61 |
Geo-Embed achieves the best performance across all evaluation metrics, outperforming Qwen3-VL-Embedding 2B, VLM2Vec-V2, and CLIP. Compared with Qwen3-VL-Embedding 2B, it improves VQA, bidirectional image-text retrieval, and classification by 4.15%, 1.33%/3.07%, and 2.88%, respectively, while nearly doubling Grounding R@1 on OPT-RSVG from 4.80% to 9.67%. These results demonstrate that Geo-Embed's geospatial representations transfer effectively to external datasets not used during fine-tuning, with consistent improvements across different task types.
Reply to Reviewer A5jy (Part 3)
6. Duplicate positives in contrastive learning
Our implementation follows the standard in-batch InfoNCE training protocol adopted by the VLM2Vec series. When multiple queries share an identical target, their target embeddings are identical under deterministic encoding and are not directly pushed apart during contrastive learning. We will clarify this training protocol in the revised manuscript.
7. Clarification of dataset statistics, experimental settings, and data splits
We will clarify the following details in the revised manuscript.
-
UrBench statistics: The training volume of 9,274 in Table 1 contains a minor typo. The correct number is 2,869, consistent with Table 5.
-
VIGOR data: The two experiments use different data configurations. The main benchmark uses 3,045 training samples, while the modality interaction experiment uses 52,595 valid samples from four cities, split into 42,076 training and 10,519 evaluation samples.
-
Training and evaluation settings: The Geo-Embed model uses a per-device mini-batch size of 4. The Sub32–Sub256 configurations in Table 3 refer to sub-batch organization sizes, and the reported scores are averaged over the corresponding GeoMEB subtasks within each modality using the standard evaluation protocol.
-
Data splits: We follow the official dataset splits to ensure fair comparisons with existing methods. For datasets such as VRSBench that contribute multiple subtasks, we additionally ensure image-level separation across training and evaluation, preventing the same original image from appearing in both splits.
References
[1] Helber P, Bischke B, Dengel A, et al. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12(7): 2217-2226.
[2] Yuan Z, Zhang W, Fu K, et al. Exploring a fine-grained multiscale method for cross-modal remote sensing image retrieval[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-19.
[3] Lobry S, Marcos D, Murray J, et al. RSVQA: Visual question answering for remote sensing data[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(12): 8555-8566.
[4] Li K, Wang D, Xu H, et al. Language-guided progressive attention for visual grounding in remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 1-13.
We sincerely thank the reviewer once again for the time and effort devoted to reviewing our work. We greatly appreciate the thoughtful feedback and would be delighted to discuss any further questions or suggestions.
Reply to Reviewer A5jy (Part 1)
We thank the reviewer for the detailed and constructive suggestions. To address the concerns, we have conducted additional experiments and clarified the implementation details. We respond to each point below.
1. Validity of evaluation after task reformulation
GeoMEB aims to evaluate the ability of embedding models to perform diverse geospatial tasks through similarity-based matching, rather than directly replacing task-specific models for localization, change detection, or continuous-value prediction. To address concerns about the validity of task reformulation, we additionally report IoU for Grounding and Change Detection, and MAE for PlacePulse and CityLens.
The additional results indicate improvements in both R@1 and IoU across all Grounding and Change Detection tasks, with positive correlations between their performance gains (Pearson r = 0.943; Spearman ρ = 0.771). For continuous-value prediction tasks, Geo-Embed achieves higher R@1 and lower MAE across all four CityLens tasks and all six PlacePulse tasks. The correlations between R@1 gains and MAE reductions are also positive for CityLens (Pearson r = 0.964; Spearman ρ = 0.800) and PlacePulse (Pearson r = 0.823; Spearman ρ = 0.829). These findings suggest that improvements under the unified Embedding Ranking metric are generally associated with improvements in the original task metrics.
2. Closed-source model evaluation and human performance baselines
First, we would like to kindly clarify that our benchmark already includes Seed1.6-Embedding, a commercial closed-source multimodal embedding model. While OpenAI provides text embedding models, GPT models do not offer a directly comparable multimodal embedding API. We agree that incorporating additional commercial embedding models would further strengthen the benchmark's coverage.
Regarding the difficulty of SBC tasks, PlacePulse perception scores are derived from large-scale human pairwise comparisons in the original dataset and estimated using TrueSkill, rather than being manually assigned by us. GeoMEB discretizes these continuous scores at intervals of 0.5 for embedding-based ranking. Similarly, the GDP, population, building height, and healthcare accessibility indicators in CityLens are based on numerical annotations from the original dataset, which we discretize into 20 candidate bins. We conducted an additional evaluation on six PlacePulse perception tasks. Each task contains 100 questions, with three independent responses per question. The evaluation includes 12 participants, each completing 150 questions. Participants select answers from the same candidate bins as Geo-Embed, using identical visual inputs.
The human baseline achieves higher R@1 and lower MAE than Geo-Embed, providing a reference for assessing SBC task difficulty and quantifying the performance gap between human and current embedding models.