acceptodds
Under review as a conference paper at ICLR 2027

SURD-Bench: Benchmarking Unified Multimodal Models for Urban Spatiotemporal Cognition

12 views

Abstract

Understanding and modeling urban dynamics from Earth observation data is critical for urban remote sensing. Achieving this goal requires models to perceive urban objects in their spatial context, understand how they evolve over time, and predict their future states. Recent advances in Unified Multimodal Models (UMMs) have provided new opportunities to advance urban remote sensing from task-specific recognition toward comprehensive spatiotemporal cognition. However, existing benchmarks are mainly designed for general visual understanding or specific remote sensing tasks, and are not suitable for evaluating the unified understanding and generation capabilities of UMMs in urban remote sensing. The capability boundaries of UMMs in urban spatiotemporal cognition remain insufficiently characterized. Inspired by human cognitive processes, we propose SURD-Bench around four cognitive stages: Sensing, Understanding, Reasoning, and Decision, for evaluating UMMs in urban spatiotemporal cognition. In constructing SURD-Bench, we integrate in-house and public datasets, develop eight comprehensive tasks through an AI-assisted curation workflow with human verification, and design a unified evaluation framework for diverse outputs. Using SURD-Bench, we evaluate 18 representative commercial and open-source multimodal models. The results show that even the best-performing commercial model, Gemini-3.1-image, achieves an average score of only 41.85%, indicating that current UMMs still face substantial challenges in urban spatiotemporal cognition. Further analysis identifies limitations in fine-grained pixel-level semantic understanding, spatiotemporal relationship modeling, and the integration of understanding and generation capabilities. These findings offer insights into the future development of remote sensing UMMs.

Primary area:
datasets and benchmarks
Keywords:
Unified Multimodal Models, Benchmark, Earth Observation, Spatiotemporal Cognition

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9UVkOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes the SURB-Bench benchmark to test the potential capabilities of UMMs in understanding urban dynamics. The authors design a series of benchmark tasks and constructs datasets from four stages: Sensing, Understanding, Reasoning, and Decision. However, this paper is confront with problems such as the rationality of task stage design, insufficient coverage (being too narrow), the relevance between tasks and reality (over-simplifying actual task workflows), and model evaluation.

Critical strengths
  • The research objective proposed is of great value, emphasizing the importance of understanding and predicting the urban dynamics. This paper attempts to achieve the four major task objectives through understanding and generation from the perspective of time-series remote sensing images and UMMs. However, understanding urban dynamics only from the perspective of time-series remote sensing images is an excessive simplification of this problem.
Critical weaknesses
  1. Mismatch between evaluation indicators and task objectives. SURD-Bench claims to evaluate the urban spatio-temporal cognition ability of UMMs, but its evaluation protocol has fundamental flaws. For image generation tasks (CR, TSR, FSF), only general image quality indicators such as PSNR, SSIM, LPIPS and CLIP-Score are used, which cannot measure whether the generation results conform to the physical laws and geographic semantics in the field of remote sensing. For example, a model may generate a visually realistic but geographically implausible urban landscape, yet still obtain a high LPIPS score. For language tasks (DTC, RA, PR), the LLM-as-Judge paradigm is adopted, but GPT-5-mini as the judge itself may lack professional knowledge of remote sensing, leading to systematic deviations between the scoring standards and the judgments of domain experts.

  2. Data leakage risk in benchmark construction. SURD-Bench integrates multiple public datasets (LoveDA, CUHK-CR, DESTRUCTION, xBD, UV-SAM, DVL-Bench, etc.), but does not specify whether these datasets are already included in the training data of the evaluated models. For example, general models such as GPT-4o and Qwen3-VL have most likely seen samples from these public datasets during training, leading to inflated evaluation results. The paper does not provide any data leakage detection or deduplication mechanisms, which severely calls into question the fairness and credibility of the benchmark. In addition, when integrating public datasets, the paper does not expand on the discussion of how to set principles and evaluation methods to quantify that these datasets appropriately cover the task requirements, and are representative and diverse.

  3. Task design lacks real-world relevance. The task settings of SURD-Bench are overly simplified, which has a significant gap with real urban remote sensing application. For example, the PR task requires the model to propose planning suggestions based on multi-temporal images, but real urban planning needs to integrate multi-source information such as socio-economic data, population distribution, and policy constraints, so reasonable decisions cannot be made solely relying on remote sensing images. Similarly, the RA task only requires identifying risk types, but actual risk assessment needs to quantify dimensions such as risk level, impact scope, and potential loss. Such simplification may lead to the model performing well on the benchmark but failing in practical applications. In addition, the scope of the spatio-temporal cognition targeted by SURD-Bench should be relatively broad, as the paper does not have a targeted application domain. However, the task setting seems to be extremely lopsided. It is very strange why less relevant and relatively narrow tasks are chosen, such as CR and TSR in the sensing category. There should be more low-level tasks required in remote sensing, even if they are only targeted at spatio-temporal cognition; there are only PR and RA in Decision, which seems to imply a narrower application field rather than broad spatio-temporal cognition.

  4. Bias in model selection. This paper evaluated 18 models, but there is an obvious bias in model selection. First, remote sensing-specific models (TEOChat, VHM, LHRS-Bot-nova, etc.) only cover language understanding tasks, while generative models (FLUX.2, Qwen-Image-2.0-Pro, etc.) only cover image generation tasks, leading to a lack of fairness in cross-category comparisons. Second, recently released SOTA models (such as RS-WorldModel, Uni-RS, etc.) are not included, which limits the timeliness and value of the benchmark. In addition, all models use zero-shot prompts, and whether the models have been fine-tuned in the remote sensing field is not explicitly considered, which is not conducive to fairly evaluating the domain adaptation ability of the models.

  5. Insufficient statistical significance and reproducibility. The paper does not report the standard deviation or confidence interval from multiple runs, and only presents the results of a single evaluation. Given the stochastic nature of generative models (e.g., Gemini-3.1-image), the results of a single evaluation may not reflect the true performance of the model. In addition, the paper does not provide public links to the evaluation code and the complete prompt template (it only mentions "Appendix"), nor does it specify key settings such as the temperature parameter and maximum token count of the GPT-5-mini evaluator, which makes it difficult for other researchers to reproduce the experimental results. For the LLM-as-Judge paradigm, the consistency coefficient between the evaluator and human assessors (e.g., Kappa value) is not reported, so the reliability of the evaluation cannot be verified.

Questions for authors
  • The authors can make their contribution more realistic and focused, by narrowing down their benchmark targets to a certain application, e.g., understanding the spatio-temporal dynamics for what purposes, and hence their choice of certain tasks. This could include a renaming of their benchmark to give an impression of a more focus target. Currently the mismatch between SIRD-Bench targets and the task setting is quite large.
  • Further, more appropriate indicators need to be developed for image generation tasks (e.g., CR, TSR, FSF), e.g. those that can verify the plausibility and physical realism of the generated content. After all, remote sensing images is different from natural images, in that unrealistic images do not matter that much especially in artistic domains.
  • The oversimplification of the tasks is a vital flaw, but seems not to be easily corrected. Many tasks cannot simply be reduced to visual understanding or time series prediction problem solely based on remote sensing images. Such tasks can only be treated as a playground/toy examples to show the ability of current UMM models, rather than tasks that have real-world implications. So their results should be carefully interpreted and confined.
14 comments
AuthorsOpenReview ↗

Response to Reviewer 9UVk (1/4)

Dear Reviewer 9UVk,

We appreciate your careful evaluation of our work and the valuable feedback. Our responses to each point are presented in detail.


W3, Q1 & Q3. Benchmark Scope and Relevance to Urban Remote Sensing

“The task settings of SURD-Bench are overly simplified, which has a significant gap with real urban remote sensing application.”

A1:

We agree that urban planning and risk assessment require population data, socioeconomic information, and policy context alongside imagery. SURD-Bench is a first step toward this broader goal. It evaluates the part of urban spatiotemporal cognition grounded in remote sensing imagery: what unified multimodal models (UMMs) can reconstruct, identify, track, and predict from repeated observations of the same area. These are the observations available to the models and the evidence they can contribute to downstream decisions. Even within this scope, the best eight-task average was 41.85, and all models evaluated on change detection (CD) scored below 10% IoU.

  • Risk assessment (RA). Models must identify the risk type, describe relevant scene features, cite visual evidence, and assess severity (Appendix B.4). The score combines risk identification accuracy with assessment quality (Equation 10). Estimating the full impact or potential losses requires additional data.
  • Cloud removal (CR) and time series reconstruction (TSR). CR addresses cloud-obscured areas in optical imagery. TSR addresses missing observations in an image time series. Both tasks assess whether models can recover information needed for subsequent spatiotemporal analysis.
  • Decision support through RA and PR. RA supports responses to existing risks, for example by identifying building damage or urban villages in imagery. PR asks models to use observed changes to propose options for future urban development.

We will clarify in the revised manuscript that RA and PR evaluate image-based risk identification and planning support, respectively. Real-world decisions also require socioeconomic data and policy context. We retain the name SURD-Bench because it reflects the four stages that organize the tasks: sensing, understanding, reasoning, and decision.


AuthorsOpenReview ↗

Response to Reviewer 9UVk (2/4)

W1 & Q2. Image Generation Metrics and LLM-as-Judge

“Mismatch between evaluation indicators and task objectives.”

A2:

We fully understand your concern. We added copy baselines for TSR and FSF and tested scoring reliability in two ways. First, we compared automated scores with domain experts’ judgments. Second, we examined how different LLM judges scored the same responses and ranked the models. Collectively, SURD-Bench scores provide a reasonably reliable basis for comparing model performance.

(1) Image generation metrics and copy baselines. We compare each generated image with an observation of the same location. The reference is a cloud-free image for CR, the held-out frame for TSR, and the observed future frame for FSF. For example, in CR we use PSNR, SSIM, and LPIPS to assess pixel-level, structural, and perceptual fidelity, respectively. These metrics are commonly used in remote sensing cloud-removal studies [1–3].

For the two temporal tasks, we added copy baselines (Appendix D.3). We copied the frame immediately before the target for TSR and the last input frame for FSF. Copying outscored the best generative model, Gemini-3.1-image, by 0.37 points on TSR and 2.25 points on FSF. In other words, current models attempt to predict and generate local changes rather than simply copying historical imagery. Generative models face two challenges: preserving the observed textures and spatial layout as faithfully as direct copying does, and generating plausible changes in the correct locations. The copy baseline’s higher scores indicate that the gains from local updates did not offset the errors in appearance and structure introduced during generation. This aligns with the low CD scores (IoU below 10% for all evaluated models). Together, current models still struggle to use temporal information to generate plausible changes in the correct locations.

TaskLPIPS ↓CLIP-I ↑Score ↑
TSR copy baseline0.55120.911568.02
FSF copy baseline0.43090.954576.18

(2) Agreement with domain experts. We sampled 30 scenes from each of five tasks (TSR, FSF, DTC, RA, and PR) and evaluated three model outputs per scene, for a total of 450 outputs. For TSR and FSF, we compared the composite image score based on LPIPS and CLIP-I with expert scores. For DTC, RA, and PR, we compared GPT-5-mini scores with expert scores. The Pearson correlation coefficients (r) for each task are reported below. Automated and expert scores were strongly correlated on average (mean r = 0.722).

MetricTSRFSFDTCRAPRMean
Pearson’s r0.8560.9110.5870.6130.6420.722

(3) Role of the LLM judge and comparison across judges. GPT-5-mini scores each response against a text reference using task-specific criteria (Appendices B.3–B.4). For PR, it prepares the reference by rephrasing an existing description of the held-out final image as a planning proposal. The evaluated models see only the historical images. GPT-5-mini itself is not evaluated on PR.

We used Qwen3.8-27B and Qwen3.5-122B-A10B to rescore the same 90 responses per language task. Although the scores differed across judges, all three agreed on the top model for each task. They ranked GPT-4o first on DTC and RA and Gemini-3.1-image first on PR.

TaskEvaluated modelGPT-5-miniQwen3.8-27BQwen3.5-122B-A10B
DTCGemini-3.1-image28.8931.1124.22
DTCBAGEL26.0028.6726.67
DTCGPT-4o40.6748.0040.00
RAGemini-3.1-image43.3338.6756.67
RABAGEL57.3337.3344.00
RAGPT-4o80.0064.0076.67
PRGemini-3.1-image48.0038.2238.89
PRBAGEL35.7825.1124.67
PRGPT-4o44.4436.0035.33

References

[1] J. Sui et al. “Diffusion Enhancement for Cloud Removal in Ultra-Resolution Remote Sensing Imagery.” IEEE Transactions on Geoscience and Remote Sensing, 2024.

[2] X. Zou et al. “DiffCR: A Fast Conditional Diffusion Framework for Cloud Removal From Optical Satellite Images.” IEEE Transactions on Geoscience and Remote Sensing, 2024.

[3] Y. Liu et al. “Effective Cloud Removal for Remote Sensing Images by an Improved Mean-Reverting Denoising Model with Elucidated Design Space.” CVPR 2025.


AuthorsOpenReview ↗

Response to Reviewer 9UVk (3/4)

W2. Data Leakage and Dataset Coverage

“Data leakage risk in benchmark construction.”

A3:

We would like to clarify three points: potential overlap between our test images and model pretraining data, how we split the data for training and testing, and how we selected data sources for each task.

(1) Potential overlap with pretraining data. SURD-Bench combines six public datasets with multitemporal samples we constructed from SuperView-1 and historical Google Earth imagery. For models such as GPT-4o and Qwen3-VL, we lack the image-level records needed to verify whether their pretraining data included any SURD-Bench test images. We will acknowledge this uncertainty in the limitations.

(2) Train–test splits. For public data, we retain each source dataset’s original split. For our in-house multitemporal data, we split by spatial tile. All observations of the same tile, across all years, belong to the same split.

(3) Task coverage and diversity. We selected each source for the observations and annotations its task requires. CUHK-CR provides high-resolution cloud-free references for cloud removal. LoveDA provides pixel-level land-cover labels for semantic segmentation. DVL-Bench provides multitemporal descriptions of where changes occur and in what order. For risk assessment, DESTRUCTION, xBD, and UV-SAM together cover building damage and urban village identification. Our in-house data span 16 cities, with 4–6 observations per sequence. Appendix B.1 reports the sample distribution by task.


W4. Model Selection and Fair Comparison

“This paper evaluated 18 models, but there is an obvious bias in model selection.”

A4: We fully understand your concern. We clarify our model selection and evaluation below.

(1) Cross-category comparisons. We evaluate each model on the tasks it supports. The original eight-task average reflects both task coverage and performance. We therefore added Image-Avg and Text-Avg, which average scores over supported tasks within each modality. We also report the task counts so readers can see which comparisons cover the same tasks (Appendix D.2).

(2) Remote sensing adaptation. We evaluated four models fine-tuned for remote sensing (TEOChat, VHM, LHRS-Bot-nova, and ScoreRS) as a separate group (Section 4.1 and Table 2). To directly assess the effect of task-specific fine-tuning, we fine-tuned BAGEL on semantic segmentation (SS). With its original color-mask output, LoRA fine-tuning increased BAGEL’s mIoU from 8.97% to 13.16%. With a segmentation head that predicts pixel classes directly, BAGEL achieved 43.00% mIoU (Appendix D.6).

(3) Recent SOTA models. To our knowledge, neither RS-WorldModel nor Uni-RS has publicly released official model weights for inference.


Reviewer_LbzhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SURD-Bench evaluates urban spatiotemporal cognition in unified multimodal models. It examines urban change understanding, future scene generation, and decision support beyond object recognition. The benchmark covers four stages and eight tasks, with 15,452 samples: 11,767 for training and 3,685 for testing. Experiments evaluate 18 multimodal models and characterize their capabilities and limitations across tasks.

Critical strengths
  1. Comprehensive evaluation framework. The benchmark covers sensing, understanding, reasoning, and decision-making, offering a broad perspective on urban spatiotemporal cognition.

  2. Systematic multi-model evaluation. The evaluation of 18 models reveals differences in urban change understanding and future scene generation, providing useful empirical references.

Critical weaknesses
  1. Overlap with existing benchmarks. XLRS-Bench (Wang et al., 2025) covers perception and reasoning. VLRS-Bench (Luo et al., 2026) includes cognition, decision, and prediction. OmniEarth (Fu et al., 2026) includes perception, reasoning, grounding, and segmentation. The distinct contribution of integrating these tasks into a unified framework needs clearer justification.

  2. Data and evaluation reliability need further validation. The independence of data sources and geographic separation between training and test sets need clarification. LLM-based text scoring and the aggregation of heterogeneous task scores also require stronger validation.

References:

· Wang, F., et al. (2025). XLRS-Bench: Could Your Multimodal LLMs Understand Extremely Large Ultra-High-Resolution Remote Sensing Imagery? CVPR 2025.

· Luo, Z., et al. (2026). VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensing. arXiv:2602.07045.

· Fu, R., et al. (2026). OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks. arXiv:2603.09471.

Questions for authors
  1. Could the authors clarify how geographically or temporally related samples are assigned to training and test sets across tasks and data sources?

  2. What evidence supports agreement between automated scores and domain-expert judgments? This is particularly relevant to future-scene generation under planning conditions and to planning recommendations. Comparisons with simple baselines and scoring-agreement analyses would help validate these evaluations.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Lbzh (1/2)

Dear Reviewer Lbzh,

We appreciate your careful evaluation of our work and the valuable feedback. Our responses to each point are presented in detail.


W1. Overlap with Existing Benchmarks

“The distinct contribution of integrating these tasks into a unified framework needs clearer justification.”

A1: We would like to clarify how SURD-Bench differs from existing benchmarks. Although individual tasks inevitably overlap with those in existing benchmarks, SURD-Bench makes three distinct contributions.

  1. Models evaluated. SURD-Bench targets unified multimodal models (UMMs) and evaluates their understanding and generation capabilities in urban spatiotemporal settings using remote sensing imagery. XLRS-Bench, VLRS-Bench, and OmniEarth primarily assess how well vision language models (VLMs) understand and reason about remote sensing imagery.

  2. Task design. We organize eight tasks into four stages of urban spatiotemporal cognition: sensing, understanding, reasoning, and decision. This design brings low-level image restoration, change understanding, and future scene generation into the evaluation of the same model. For example, VLRS-Bench evaluates prediction through text responses. Our future scene forecasting (FSF) task requires the model to generate a future image, allowing us to assess the spatial layout of its prediction.

  3. Empirical value. Joint evaluation reveals capability differences both within and across models. A model can generate a plausible scene yet still struggle to identify which features changed and where (Section 4.2). Across models, UMMs show no consistent advantage over specialized models on image or text tasks (Figure 5(b)).


W2 & Q1. Data Sources and Train–Test Splits

“how geographically or temporally related samples are assigned to training and test sets...”

A2:

We retain the original train–test splits of the public datasets. For our in-house multitemporal data, we split by spatial tile so that all observations of the same tile remain in the same split across years. We have clarified this in the revised manuscript.


AuthorsOpenReview ↗

Response to Reviewer Lbzh (2/2)

W2 & Q2. Expert Agreement, Copy Baselines, and Score Aggregation

“What evidence supports agreement between automated scores and domain-expert judgments?”

A3:

We examined agreement with experts, consistency across LLM judges, copy baselines, and the stability of the overall ranking. Collectively, SURD-Bench scores provide a reasonably reliable basis for comparing model performance.

(1) Agreement with domain experts. We sampled 30 scenes from each of five tasks (TSR, FSF, DTC, RA, and PR) and evaluated three model outputs per scene, for a total of 450 outputs. For TSR and FSF, we compared the composite image score based on LPIPS and CLIP-I with expert scores. For DTC, RA, and PR, we compared GPT-5-mini scores with expert scores. We report Pearson’s r for each task below. Automated and expert scores were strongly correlated on average (mean r = 0.722).

MetricTSRFSFDTCRAPRMean
Pearson’s r0.8560.9110.5870.6130.6420.722

(2) Comparison across judges. We used Qwen3.8-27B and Qwen3.5-122B-A10B to rescore the same 90 responses per language task. Although the scores differed across judges, all three agreed on the top model for each task. They ranked GPT-4o first on DTC and RA and Gemini-3.1-image first on PR.

TaskEvaluated modelGPT-5-miniQwen3.8-27BQwen3.5-122B-A10B
DTCGemini-3.1-image28.8931.1124.22
DTCBAGEL26.0028.6726.67
DTCGPT-4o40.6748.0040.00
RAGemini-3.1-image43.3338.6756.67
RABAGEL57.3337.3344.00
RAGPT-4o80.0064.0076.67
PRGemini-3.1-image48.0038.2238.89
PRBAGEL35.7825.1124.67
PRGPT-4o44.4436.0035.33

(3) Copy baselines for TSR and FSF. We copied the frame immediately before the target for TSR and the last input frame for FSF. Copying outscored the best generative model, Gemini-3.1-image, by 0.37 points on TSR and 2.25 points on FSF (Appendix D.3). In other words, current models attempt to predict and generate local changes rather than simply copying historical imagery. Generative models face two challenges: preserving the observed textures and spatial layout as faithfully as direct copying does, and generating plausible changes in the correct locations. The copy baseline’s higher scores indicate that the gains from local updates did not offset the errors in appearance and structure introduced during generation. This aligns with the low CD scores (IoU below 10% for all evaluated models). Together, current models still struggle to use temporal information to generate plausible changes in the correct locations.

TaskLPIPS ↓CLIP-I ↑Score ↑
TSR Copy Baseline0.55120.911568.02
FSF Copy Baseline0.43090.954576.18

(4) Score aggregation and ranking stability. We added Image-Avg and Text-Avg over each model’s supported tasks (Appendix D.2). We also conducted a split-half analysis within each task(Appendix D.7). The overall ranking of all 18 models was identical across the two halves.

AuthorsOpenReview ↗

Follow-up to Reviewer Lbzh

Dear Reviewer Lbzh,

Thank you again for your positive assessment and helpful questions. As the discussion period ends soon, we would like to briefly recap our responses:

  • Distinction from existing benchmarks (W1). We clarified how SURD-Bench differs from existing benchmarks by jointly evaluating the understanding and generation capabilities of unified multimodal models (UMMs) on urban remote sensing imagery.
  • Train–test splits (W2 & Q1). Public datasets keep their original splits. Our in-house multitemporal data are split by spatial tile, so all observations of the same tile stay in the same split across years.
  • Scoring reliability (W2 & Q2). We examined agreement with experts, consistency across LLM judges, copy baselines, and the stability of the overall ranking. Collectively, SURD-Bench scores provide a reasonably reliable basis for comparing model performance.

We hope these responses address your questions, and we would be glad to clarify anything further before the discussion closes. If they do, we would greatly appreciate your continued support of the paper.

Best regards,

Authors of Submission 30236

Reviewer_YcvPOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SURD-Bench, a benchmark designed to evaluate Unified Multimodal Models (UMMs) on urban spatiotemporal cognition using remote sensing (RS) imagery. Inspired by human cognitive processes, the authors organize urban perception and modeling into a progressive four-stage taxonomy, encompassing eight specific tasks that require diverse output modalities. The benchmark is curated by combining multi-temporal in-house satellite imagery across 16 cities with six public RS datasets via an AI-assisted filtering and human-verified pipeline, yielding 3,685 test samples and 11,767 training samples. To standardize evaluation across heterogeneous outputs, the authors unify pixel-level metrics (via nearest-color matching for generated masks), perceptual/semantic image generation metrics, and LLM-as-a-Judge scoring into a normalized 0–100 scale. Empirical validation across 18 commercial and open-source models (spanning understanding-only MLLMs, generative models, and UMMs) reveals that even the top-performing model (Gemini-3.1-image) achieves an average score of only 41.85%, highlighting severe bottlenecks in pixel-level semantic grounding, spatiotemporal change localization, and the synergy between understanding and generation

Critical strengths
  1. Existing remote sensing benchmarks predominantly evaluate single-image perception or isolated temporal events using discriminative formats, which are vulnerable to language priors and cannot evaluate generative capabilities. By formulating urban spatiotemporal cognition as a four-stage progressive chain that requires both open-ended textual reasoning and conditioned visual generation (inpainting, segmentation mask generation, and future scene forecasting), SURD-Bench fills a critical gap. This is highly consequential because it shifts the evaluation paradigm of Earth observation foundation models from static visual recognition toward holistic spatiotemporal world modeling.

  2. Beyond merely reporting leaderboard numbers across 18 models, the paper provides sharp empirical insights into the capability boundaries of current UMMs. Most notably, it uncovers a striking discrepancy between temporal scene generation (where models score up to ~70% on Time Series Reconstruction and Future Scene Forecasting) and explicit temporal change understanding (where all models score below 10% on Change Detection), as well as demonstrating that current UMMs fail to outperform specialized generative or understanding models on their respective subtasks. This analysis is consequential because it provides concrete evidence that current unified architectures capture superficial appearance statistics without achieving pixel-level semantic grounding or true cross-modal synergy.

Critical weaknesses
  1. Lack of a Copy-Baseline in Temporal Image Generation Tasks. In multi-temporal urban satellite imagery, the vast majority of the spatial background remains unchanged across consecutive timestamps. Consequently, a trivial “Identity/Copy Baseline” (i.e., simply outputting the adjacent historical frame without modeling any temporal dynamics) will naturally achieve an extremely high CLIP score and a favorable LPIPS score.
  2. PR uses the same GPT-5-mini model to generate ground-truth references and judge model outputs, risking self-preference bias while inadequately accounting for the multiple valid solutions inherent in urban planning. Moreover, no baselines were fine-tuned on the 11,767-sample training set, leaving both the learnability of its annotations and the potential of domain-specific fine-tuning to improve the extremely low SS and CD scores (<10%) unverified.
Questions for authors
  1. Please report the LPIPS and CLIP-I scores of a simple copy baseline: the last input frame for FSF and the frame immediately preceding the target for TSR. Computing PSNR, SSIM, and LPIPS only within regions where changes actually occurred, identified using change masks or bounding boxes, would also help determine whether models such as Gemini-3.1-image and FLUX.2 capture temporal changes.

  2. The vast majority of models score below 9% under nearest-neighbor color matching. How much of this poor performance comes from spatial misalignment or structural distortion in the generated masks, and how much comes from incorrect semantic labels at correctly aligned locations? Please quantify the contributions of these two types of error.

  3. Since GPT-5-mini both generates the reference proposals and serves as the judge, how do the authors check for potential self-preference bias? Also, how does the evaluation avoid penalizing reasonable urban renewal proposals grounded in historical evolution simply because they differ from the changes observed in the final year?

  4. The current eight-task Avg treats unsupported tasks as zero, so it reflects both task coverage and model performance. Please also report separate averages for image generation tasks (Image-Avg) and text generation tasks (Text-Avg) to support clearer and fairer comparisons between specialized models and unified multimodal models (UMMs).

6 comments
AuthorsOpenReview ↗

Response to Reviewer YcvP (1/3)

Dear Reviewer YcvP,

We appreciate your careful evaluation of our work and the valuable feedback. Our responses to each point are presented in detail.


W1 & Q1. Copy Baselines and Evaluation in Regions of Change

“Lack of a Copy-Baseline in Temporal Image Generation Tasks..." "...only within regions where changes actually occurred...”

A1:

We thank the reviewer for highlighting the importance of scene persistence when evaluating temporal image generation. We added a copy baseline and evaluated FSF in the main regions of change to further examine how well the models generate local changes.

(1) Copy baseline. For TSR, we copied the frame immediately preceding the target. For FSF, we copied the last input frame. The copy baseline outperformed all generative models, including the best-performing Gemini-3.1-image, by 0.37 points on TSR and 2.25 points on FSF.

TaskLPIPS ↓CLIP-I ↑Score ↑
TSR Copy Baseline0.55120.911568.02
FSF Copy Baseline0.43090.954576.18

(2) Evaluation in manually annotated regions of change. We manually annotated bounding boxes around changes in all 492 FSF test samples and computed the FSF score within these boxes.

MethodFull-image score ↑Change-region score ↑
Harmon-RecA-plus44.4544.28
BAGEL50.3144.72
Bagel-RecA52.8947.69
SenseNova-U1-8B-MoT57.4951.55
OmniGen264.6859.59
EMU3.569.6665.05
FLUX.271.7266.70
GPT-Image-271.1168.32
Gemini-3.1-image73.9369.91
Copy Baseline76.1872.84

The models attempted to predict local changes instead of simply copying historical imagery. They faced two challenges in FSF: preserving the observed textures and spatial layout as faithfully as copying does, and generating plausible changes in the correct locations. The copy baseline scored higher on both full images and manually annotated regions of change. This suggests that the gains from local updates did not offset the errors in appearance and structure introduced during generation.

This aligns with the low CD scores (IoU below 10% for all evaluated models). Together, these results suggest that current models still struggle to use temporal information to generate plausible changes in the correct locations. We added the copy baseline results to Appendix D.3 and revised Section 4.2 to discuss these two challenges.


AuthorsOpenReview ↗

Response to Reviewer YcvP (2/3)

W2 & Q3. Judge Bias

“Since GPT-5-mini both generates the reference proposals and serves as the judge, how do the authors check for potential self-preference bias?”

A2:

We would like to clarify that GPT-5-mini is not among the models evaluated on PR. The content of each reference proposal comes from an existing description of the held-out final image. GPT-5-mini’s role is limited to rewriting that description as a planning proposal. The evaluated models see only the historical images.

To assess the reliability of automated scoring in SURD-Bench, we conducted two analyses. First, we examined agreement between automated scores and domain experts’ judgments. Second, we compared the scores that different LLM judges assigned to the same outputs to assess how judge selection affected scores and model rankings.

(1) Agreement with domain experts. We sampled 30 scenes from each of five tasks (TSR, FSF, DTC, RA, and PR) and evaluated three model outputs per scene, for a total of 450 outputs. For TSR and FSF, we compared the composite image score based on LPIPS and CLIP-I with expert scores. For DTC, RA, and PR, we compared GPT-5-mini scores with expert scores. We report Pearson’s r for each task below. Automated and expert scores were strongly correlated on average (mean r = 0.722).

MetricTSRFSFDTCRAPRMean
Pearson’s r0.8560.9110.5870.6130.6420.722

(2) Comparison across judges. We used Qwen3.8-27B and Qwen3.5-122B-A10B to rescore the same 90 responses per language task. Although the scores differed across judges, all three agreed on the top model for each task. They ranked GPT-4o first on DTC and RA and Gemini-3.1-image first on PR.

TaskEvaluated modelGPT-5-miniQwen3.8-27BQwen3.5-122B-A10B
DTCGemini-3.1-image28.8931.1124.22
DTCBAGEL26.0028.6726.67
DTCGPT-4o40.6748.0040.00
RAGemini-3.1-image43.3338.6756.67
RABAGEL57.3337.3344.00
RAGPT-4o80.0064.0076.67
PRGemini-3.1-image48.0038.2238.89
PRBAGEL35.7825.1124.67
PRGPT-4o44.4436.0035.33

(3) Proposal evaluation. PR evaluates how well models capture patterns of urban development from historical imagery. The observed development serves as one verifiable reference. As specified in the evaluation prompt (Appendix B.4), each proposal is scored separately on spatial grounding accuracy, planning action fidelity, and development outcome relevance. A proposal that recommends a different intervention can therefore still earn credit for identifying the relevant area or anticipating a similar benefit.

We also compared the reference-based scores with expert judgments on 90 proposals. The scores were positively correlated (Pearson’s r = 0.642). When comparing two proposals for the same scene, the automatic judge and the experts preferred the same proposal in 76.2% of non-tied comparisons. These results provide some support for using reference-based scores to evaluate planning proposals.


W2. Fine-Tuning on the SURD-Bench Training Set

“no baselines were fine-tuned on the 11,767-sample training set...”

A3:

To test whether models can learn from the semantic segmentation (SS) annotations, we conducted three supervised training experiments:

  1. SegFormer-B2: We fine-tuned all model parameters on the SS training set.
  2. BAGEL with LoRA: We fine-tuned BAGEL using LoRA while retaining its original color-mask output.
  3. BAGEL with a segmentation head: We added a segmentation head to BAGEL to predict the class of each pixel directly.
ModelOutputmIoU (%) ↑
BAGEL (zero-shot)Color mask8.97
BAGEL-LoRAColor mask13.16
SegFormer-B2Pixel labels38.92
BAGEL + segmentation headPixel labels43.00

With the same color-mask output, LoRA fine-tuning improved BAGEL’s mIoU from 8.97% to 13.16%, showing that it can learn from the SS annotations. With a segmentation head, BAGEL reached 43.00% mIoU, outperforming SegFormer-B2 (38.92%). This suggests that BAGEL’s pretrained representations can support pixel-level semantic prediction. We report these experiments in Appendix D.6.


AuthorsOpenReview ↗

Response to Reviewer YcvP (3/3)

Q2. Structural and Label Errors in Semantic Segmentation

“How much of this poor performance comes from spatial misalignment or structural distortion in the generated masks, and how much comes from incorrect semantic labels at correctly aligned locations?”

A4:

We added two quantitative analyses to distinguish spatial structure errors from semantic label errors. Across all models, errors affecting entire regions contributed the most to segmentation error. Even when models located a region correctly, they sometimes assigned it the wrong land-cover class.

(1) Mostly whole-region errors. Following Bernhard et al. (2024), we decomposed segmentation errors into boundary, extent, and segment errors. For every model, whole-region errors contributed the most, followed by errors in region extent. Whole-region errors were especially pronounced for Harmon-RecA-plus and EMU3.5. They also dominated for GPT-Image-2 and Gemini-3.1-image, although errors in region extent accounted for a larger share of total error for this pair than for the first two models. Boundary errors accounted for only 0.07%–9.16% of total EoU across models.

(2) Right location, wrong label. We matched predicted regions to ground-truth regions based solely on their overlap (IoU > 0.5), regardless of class labels. On average across models, 12.91% of all misclassified pixels fell within these overlaps. Even when BAGEL and OmniGen2 found the corresponding region, they assigned the wrong land-cover class to most overlapping pixels. GPT-Image-2 and Gemini-3.1-image achieved higher match coverage and correctly labeled most overlapping pixels.

ModelmIoUBoundary EoUExtent EoUSegment EoU
Harmon-RecA-plus1.971.0520.3676.62
SenseNova-U12.484.9530.7661.82
EMU3.51.300.071.1597.49
FLUX.24.173.3225.0767.44
OmniGen25.294.3930.3759.94
Bagel-RecA8.145.2928.1558.42
BAGEL8.974.2627.5259.25
Qwen-Image-2.0-Pro14.794.3426.3854.49
GPT-Image-224.086.9623.0845.88
Gemini-3.1-image31.895.1219.8043.19
Mean10.313.9723.2662.45
ModelMatch coverageLabel error rate within overlapsShare of all errors in overlaps
Harmon-RecA-plus9.1980.018.03
SenseNova-U113.6497.1514.54
EMU3.52.540.250.01
FLUX.215.8677.2313.87
OmniGen215.2282.9915.22
Bagel-RecA17.5775.4016.76
BAGEL22.6168.5620.19
Qwen-Image-2.0-Pro21.5752.0116.02
GPT-Image-229.1720.4311.69
Gemini-3.1-image42.4112.2012.75
Mean18.9856.6212.91

We have added both analyses, the metric definitions, and the results to Appendix D.5.

Reference

M. Bernhard et al. “What's Outside the Intersection? Fine-Grained Error Analysis for Semantic Segmentation Beyond IoU.” WACV 2024.


Q4. Image-Avg and Text-Avg

“Please also report separate averages for image generation tasks (Image-Avg) and text generation tasks (Text-Avg) ...”

A5:

We added Image-Avg and Text-Avg alongside the original eight-task average. Each metric gives equal weight to the tasks a model supports within that modality. Gemini-3.1-image achieved the highest Image-Avg (42.16) and Overall Avg (41.85), while GPT-4o achieved the highest Text-Avg (54.12). We have added these results and the metric definitions to Appendix D.2.

ModelImage-AvgText-AvgOverall Avg
TEOChat—13.49 (3/3)5.06
VHM—22.89 (3/3)8.59
LHRS-Bot-nova—24.61 (3/3)9.23
ScoreRS w/ RL—27.68 (3/3)10.38
InternVL3.5-8B—39.59 (3/3)14.85
Qwen3-VL-8B-Instruct—45.97 (3/3)17.24
GPT-4o—54.12 (3/3)20.29
Qwen-Image-2.0-Pro23.21 (2/5)—5.80
FLUX.236.09 (5/5)—22.56
GPT-Image-239.15 (5/5)—24.47
Janus-Pro—35.22 (3/3)13.21
Harmon-RecA-plus25.85 (5/5)17.48 (3/3)22.71
BAGEL27.96 (5/5)37.60 (3/3)31.57
EMU3.533.52 (5/5)28.74 (3/3)31.73
Bagel-RecA29.66 (5/5)38.14 (3/3)32.84
OmniGen234.53 (5/5)31.16 (3/3)33.27
SenseNova-U1-8B-MoT31.92 (5/5)38.54 (3/3)34.40
Gemini-3.1-image42.16 (5/5)41.32 (3/3)41.85

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response to Reviewers

We appreciate the reviewers’ thoughtful comments on SURD-Bench. We are encouraged that reviewers recognized its contribution to filling a critical gap in existing remote sensing benchmarks (Reviewer YcvP), its comprehensive evaluation framework and systematic multi-model evaluation (Reviewer Lbzh), and its empirical insights into the capability boundaries of current UMMs (Reviewers YcvP and Lbzh), while also acknowledging the importance of understanding and predicting urban dynamics (Reviewer 9UVk).

Additions to the revised manuscript are shown in blue.

Summary of Updates

  1. Benchmark scope (Reviewers Lbzh and 9UVk). We clarified how SURD-Bench differs from existing benchmarks by jointly evaluating the understanding and generation capabilities of unified multimodal models (UMMs) on urban remote sensing imagery.
  2. Data sources and splits (Reviewers Lbzh and 9UVk). We explained how we selected data sources for each task and clarified the train–test split rules.
  3. Temporal image generation (Reviewers YcvP, Lbzh, and 9UVk). We added copy baselines for time series reconstruction and future scene forecasting. We also evaluated forecasts in manually annotated change regions to assess how well models capture local changes.
  4. Reliability of automated scoring (Reviewers YcvP, Lbzh, and 9UVk). We compared automated scores with expert judgments and used different LLM judges to score the same responses.
  5. Segmentation errors and fine-tuning (Reviewers YcvP and 9UVk). We quantified boundary, extent, and whole-segment errors and examined label errors in spatially matched regions. We also fine-tuned SegFormer-B2 and evaluated two ways to adapt BAGEL for segmentation: generating color masks and predicting pixel classes directly.
  6. Model selection and score aggregation (Reviewers YcvP, Lbzh, and 9UVk). We clarified the selection of specialized and unified models. Alongside the overall average, we now report Image-Avg, Text-Avg, and the number of tasks contributing to each score.
  7. Statistical stability and reproducibility (Reviewers Lbzh and 9UVk). We added bootstrap confidence intervals and a split-half analysis of model rankings. We also identified where to find the evaluation code, prompts, and judge settings in the supplementary materials.

Our individual responses address each reviewer’s comments in detail. We thank all reviewers again for their time and constructive suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.