- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes the SURB-Bench benchmark to test the potential capabilities of UMMs in understanding urban dynamics. The authors design a series of benchmark tasks and constructs datasets from four stages: Sensing, Understanding, Reasoning, and Decision. However, this paper is confront with problems such as the rationality of task stage design, insufficient coverage (being too narrow), the relevance between tasks and reality (over-simplifying actual task workflows), and model evaluation.
Critical strengths
- The research objective proposed is of great value, emphasizing the importance of understanding and predicting the urban dynamics. This paper attempts to achieve the four major task objectives through understanding and generation from the perspective of time-series remote sensing images and UMMs. However, understanding urban dynamics only from the perspective of time-series remote sensing images is an excessive simplification of this problem.
Critical weaknesses
-
Mismatch between evaluation indicators and task objectives. SURD-Bench claims to evaluate the urban spatio-temporal cognition ability of UMMs, but its evaluation protocol has fundamental flaws. For image generation tasks (CR, TSR, FSF), only general image quality indicators such as PSNR, SSIM, LPIPS and CLIP-Score are used, which cannot measure whether the generation results conform to the physical laws and geographic semantics in the field of remote sensing. For example, a model may generate a visually realistic but geographically implausible urban landscape, yet still obtain a high LPIPS score. For language tasks (DTC, RA, PR), the LLM-as-Judge paradigm is adopted, but GPT-5-mini as the judge itself may lack professional knowledge of remote sensing, leading to systematic deviations between the scoring standards and the judgments of domain experts.
-
Data leakage risk in benchmark construction. SURD-Bench integrates multiple public datasets (LoveDA, CUHK-CR, DESTRUCTION, xBD, UV-SAM, DVL-Bench, etc.), but does not specify whether these datasets are already included in the training data of the evaluated models. For example, general models such as GPT-4o and Qwen3-VL have most likely seen samples from these public datasets during training, leading to inflated evaluation results. The paper does not provide any data leakage detection or deduplication mechanisms, which severely calls into question the fairness and credibility of the benchmark. In addition, when integrating public datasets, the paper does not expand on the discussion of how to set principles and evaluation methods to quantify that these datasets appropriately cover the task requirements, and are representative and diverse.
-
Task design lacks real-world relevance. The task settings of SURD-Bench are overly simplified, which has a significant gap with real urban remote sensing application. For example, the PR task requires the model to propose planning suggestions based on multi-temporal images, but real urban planning needs to integrate multi-source information such as socio-economic data, population distribution, and policy constraints, so reasonable decisions cannot be made solely relying on remote sensing images. Similarly, the RA task only requires identifying risk types, but actual risk assessment needs to quantify dimensions such as risk level, impact scope, and potential loss. Such simplification may lead to the model performing well on the benchmark but failing in practical applications. In addition, the scope of the spatio-temporal cognition targeted by SURD-Bench should be relatively broad, as the paper does not have a targeted application domain. However, the task setting seems to be extremely lopsided. It is very strange why less relevant and relatively narrow tasks are chosen, such as CR and TSR in the sensing category. There should be more low-level tasks required in remote sensing, even if they are only targeted at spatio-temporal cognition; there are only PR and RA in Decision, which seems to imply a narrower application field rather than broad spatio-temporal cognition.
-
Bias in model selection. This paper evaluated 18 models, but there is an obvious bias in model selection. First, remote sensing-specific models (TEOChat, VHM, LHRS-Bot-nova, etc.) only cover language understanding tasks, while generative models (FLUX.2, Qwen-Image-2.0-Pro, etc.) only cover image generation tasks, leading to a lack of fairness in cross-category comparisons. Second, recently released SOTA models (such as RS-WorldModel, Uni-RS, etc.) are not included, which limits the timeliness and value of the benchmark. In addition, all models use zero-shot prompts, and whether the models have been fine-tuned in the remote sensing field is not explicitly considered, which is not conducive to fairly evaluating the domain adaptation ability of the models.
-
Insufficient statistical significance and reproducibility. The paper does not report the standard deviation or confidence interval from multiple runs, and only presents the results of a single evaluation. Given the stochastic nature of generative models (e.g., Gemini-3.1-image), the results of a single evaluation may not reflect the true performance of the model. In addition, the paper does not provide public links to the evaluation code and the complete prompt template (it only mentions "Appendix"), nor does it specify key settings such as the temperature parameter and maximum token count of the GPT-5-mini evaluator, which makes it difficult for other researchers to reproduce the experimental results. For the LLM-as-Judge paradigm, the consistency coefficient between the evaluator and human assessors (e.g., Kappa value) is not reported, so the reliability of the evaluation cannot be verified.
Questions for authors
- The authors can make their contribution more realistic and focused, by narrowing down their benchmark targets to a certain application, e.g., understanding the spatio-temporal dynamics for what purposes, and hence their choice of certain tasks. This could include a renaming of their benchmark to give an impression of a more focus target. Currently the mismatch between SIRD-Bench targets and the task setting is quite large.
- Further, more appropriate indicators need to be developed for image generation tasks (e.g., CR, TSR, FSF), e.g. those that can verify the plausibility and physical realism of the generated content. After all, remote sensing images is different from natural images, in that unrealistic images do not matter that much especially in artistic domains.
- The oversimplification of the tasks is a vital flaw, but seems not to be easily corrected. Many tasks cannot simply be reduced to visual understanding or time series prediction problem solely based on remote sensing images. Such tasks can only be treated as a playground/toy examples to show the ability of current UMM models, rather than tasks that have real-world implications. So their results should be carefully interpreted and confined.
14 comments
Response to Reviewer 9UVk (2/4)
W1 & Q2. Image Generation Metrics and LLM-as-Judge
“Mismatch between evaluation indicators and task objectives.”
A2:
We fully understand your concern. We added copy baselines for TSR and FSF and tested scoring reliability in two ways. First, we compared automated scores with domain experts’ judgments. Second, we examined how different LLM judges scored the same responses and ranked the models. Collectively, SURD-Bench scores provide a reasonably reliable basis for comparing model performance.
(1) Image generation metrics and copy baselines. We compare each generated image with an observation of the same location. The reference is a cloud-free image for CR, the held-out frame for TSR, and the observed future frame for FSF. For example, in CR we use PSNR, SSIM, and LPIPS to assess pixel-level, structural, and perceptual fidelity, respectively. These metrics are commonly used in remote sensing cloud-removal studies [1–3].
For the two temporal tasks, we added copy baselines (Appendix D.3). We copied the frame immediately before the target for TSR and the last input frame for FSF. Copying outscored the best generative model, Gemini-3.1-image, by 0.37 points on TSR and 2.25 points on FSF. In other words, current models attempt to predict and generate local changes rather than simply copying historical imagery. Generative models face two challenges: preserving the observed textures and spatial layout as faithfully as direct copying does, and generating plausible changes in the correct locations. The copy baseline’s higher scores indicate that the gains from local updates did not offset the errors in appearance and structure introduced during generation. This aligns with the low CD scores (IoU below 10% for all evaluated models). Together, current models still struggle to use temporal information to generate plausible changes in the correct locations.
| Task | LPIPS ↓ | CLIP-I ↑ | Score ↑ |
|---|---|---|---|
| TSR copy baseline | 0.5512 | 0.9115 | 68.02 |
| FSF copy baseline | 0.4309 | 0.9545 | 76.18 |
(2) Agreement with domain experts. We sampled 30 scenes from each of five tasks (TSR, FSF, DTC, RA, and PR) and evaluated three model outputs per scene, for a total of 450 outputs. For TSR and FSF, we compared the composite image score based on LPIPS and CLIP-I with expert scores. For DTC, RA, and PR, we compared GPT-5-mini scores with expert scores. The Pearson correlation coefficients (r) for each task are reported below. Automated and expert scores were strongly correlated on average (mean r = 0.722).
| Metric | TSR | FSF | DTC | RA | PR | Mean |
|---|---|---|---|---|---|---|
| Pearson’s r | 0.856 | 0.911 | 0.587 | 0.613 | 0.642 | 0.722 |
(3) Role of the LLM judge and comparison across judges. GPT-5-mini scores each response against a text reference using task-specific criteria (Appendices B.3–B.4). For PR, it prepares the reference by rephrasing an existing description of the held-out final image as a planning proposal. The evaluated models see only the historical images. GPT-5-mini itself is not evaluated on PR.
We used Qwen3.8-27B and Qwen3.5-122B-A10B to rescore the same 90 responses per language task. Although the scores differed across judges, all three agreed on the top model for each task. They ranked GPT-4o first on DTC and RA and Gemini-3.1-image first on PR.
| Task | Evaluated model | GPT-5-mini | Qwen3.8-27B | Qwen3.5-122B-A10B |
|---|---|---|---|---|
| DTC | Gemini-3.1-image | 28.89 | 31.11 | 24.22 |
| DTC | BAGEL | 26.00 | 28.67 | 26.67 |
| DTC | GPT-4o | 40.67 | 48.00 | 40.00 |
| RA | Gemini-3.1-image | 43.33 | 38.67 | 56.67 |
| RA | BAGEL | 57.33 | 37.33 | 44.00 |
| RA | GPT-4o | 80.00 | 64.00 | 76.67 |
| PR | Gemini-3.1-image | 48.00 | 38.22 | 38.89 |
| PR | BAGEL | 35.78 | 25.11 | 24.67 |
| PR | GPT-4o | 44.44 | 36.00 | 35.33 |
References
[1] J. Sui et al. “Diffusion Enhancement for Cloud Removal in Ultra-Resolution Remote Sensing Imagery.” IEEE Transactions on Geoscience and Remote Sensing, 2024.
[2] X. Zou et al. “DiffCR: A Fast Conditional Diffusion Framework for Cloud Removal From Optical Satellite Images.” IEEE Transactions on Geoscience and Remote Sensing, 2024.
[3] Y. Liu et al. “Effective Cloud Removal for Remote Sensing Images by an Improved Mean-Reverting Denoising Model with Elucidated Design Space.” CVPR 2025.
Response to Reviewer 9UVk (3/4)
W2. Data Leakage and Dataset Coverage
“Data leakage risk in benchmark construction.”
A3:
We would like to clarify three points: potential overlap between our test images and model pretraining data, how we split the data for training and testing, and how we selected data sources for each task.
(1) Potential overlap with pretraining data. SURD-Bench combines six public datasets with multitemporal samples we constructed from SuperView-1 and historical Google Earth imagery. For models such as GPT-4o and Qwen3-VL, we lack the image-level records needed to verify whether their pretraining data included any SURD-Bench test images. We will acknowledge this uncertainty in the limitations.
(2) Train–test splits. For public data, we retain each source dataset’s original split. For our in-house multitemporal data, we split by spatial tile. All observations of the same tile, across all years, belong to the same split.
(3) Task coverage and diversity. We selected each source for the observations and annotations its task requires. CUHK-CR provides high-resolution cloud-free references for cloud removal. LoveDA provides pixel-level land-cover labels for semantic segmentation. DVL-Bench provides multitemporal descriptions of where changes occur and in what order. For risk assessment, DESTRUCTION, xBD, and UV-SAM together cover building damage and urban village identification. Our in-house data span 16 cities, with 4–6 observations per sequence. Appendix B.1 reports the sample distribution by task.
W4. Model Selection and Fair Comparison
“This paper evaluated 18 models, but there is an obvious bias in model selection.”
A4: We fully understand your concern. We clarify our model selection and evaluation below.
(1) Cross-category comparisons. We evaluate each model on the tasks it supports. The original eight-task average reflects both task coverage and performance. We therefore added Image-Avg and Text-Avg, which average scores over supported tasks within each modality. We also report the task counts so readers can see which comparisons cover the same tasks (Appendix D.2).
(2) Remote sensing adaptation. We evaluated four models fine-tuned for remote sensing (TEOChat, VHM, LHRS-Bot-nova, and ScoreRS) as a separate group (Section 4.1 and Table 2). To directly assess the effect of task-specific fine-tuning, we fine-tuned BAGEL on semantic segmentation (SS). With its original color-mask output, LoRA fine-tuning increased BAGEL’s mIoU from 8.97% to 13.16%. With a segmentation head that predicts pixel classes directly, BAGEL achieved 43.00% mIoU (Appendix D.6).
(3) Recent SOTA models. To our knowledge, neither RS-WorldModel nor Uni-RS has publicly released official model weights for inference.
Response to Reviewer 9UVk (1/4)
Dear Reviewer 9UVk,
We appreciate your careful evaluation of our work and the valuable feedback. Our responses to each point are presented in detail.
W3, Q1 & Q3. Benchmark Scope and Relevance to Urban Remote Sensing
A1:
We agree that urban planning and risk assessment require population data, socioeconomic information, and policy context alongside imagery. SURD-Bench is a first step toward this broader goal. It evaluates the part of urban spatiotemporal cognition grounded in remote sensing imagery: what unified multimodal models (UMMs) can reconstruct, identify, track, and predict from repeated observations of the same area. These are the observations available to the models and the evidence they can contribute to downstream decisions. Even within this scope, the best eight-task average was 41.85, and all models evaluated on change detection (CD) scored below 10% IoU.
Appendix B.4). The score combines risk identification accuracy with assessment quality (Equation 10). Estimating the full impact or potential losses requires additional data.We will clarify in the revised manuscript that RA and PR evaluate image-based risk identification and planning support, respectively. Real-world decisions also require socioeconomic data and policy context. We retain the name SURD-Bench because it reflects the four stages that organize the tasks: sensing, understanding, reasoning, and decision.