acceptodds
Withdrawn from ICLR 2027

GeoR-Bench: Evaluating Geoscience Visual Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Geoscience intelligence is expected to understand, reason about, and predict earth system changes to support human decision-making in critical domains such as disaster response, climate adaptation and environmental protection. Although current research has shown promising progress on specific geoscience tasks, such as remote sensing interpretation, geographic question-answering, existing benchmarks remain largely task-specific which failing to capture the open-ended real world geoscience problems. As a result, it remains unclear how far current AI systems are from achieving genuine geoscience intelligence. To address this gap, we present , a for evaluating visual through reasoning informed visual editing tasks. GeoR-Bench contains 440 curated samples spanning 6 geoscience categories and 24 task types, covering earth observation imagery and structured scientific representations such as maps and diagrams. We evaluate outputs along three dimensions, including reasoning, consistency, and quality. Benchmark results of 21 closed- and open-source multimodal models reveal that geoscience reasoning remains a critical bottleneck. The highest-performing model achieves 42.7% overall strict accuracy, while the best open-source models only get 10.3%. Notably, the visual consistency and image quality of the outputs frequently surpass their scientific accuracy. Ultimately, these findings indicate that current models generate superficially plausible results but fail to capture underlying earth science processes.

Primary area:
datasets and benchmarks
Keywords:
Datasets and benchmarks, Multimodal model evaluation, Geoscience, Visual reasoning, Reasoning-guided image editing, Scientific reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mgWyOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GeoR-Bench, a benchmark for evaluating the geoscience visual reasoning capabilities of MLLMs through visual editing and generation tasks. The benchmark consists of 440 samples across 6 geoscience categories and 24 task types, covering diverse data sources and geoscience visual representations. The authors systematically evaluate 21 closed- and open-source model configurations along four dimensions: reasoning, consistency, quality, and strict accuracy. They further analyze performance bottlenecks, category-level differences, and the reasoning capabilities of open-source models.

Critical strengths

This paper extends the visual editing and generation evaluation to remote sensing and geoscience, assessing the geoscience visual reasoning capabilities of multimodal models. GeoR-Bench covers diverse task types, data sources, and scientific visual representations, offering broad evaluation coverage. Furthermore, the paper adopts 4 evaluation protocols to systematically assess 21 representative closed- and open-source model configurations, with additional analyses of instruction explicitness and judge alignment to provide further insights into model performance and evaluation reliability.

Critical weaknesses
  1. The Reasoning dimension evaluates whether the generated images satisfy the task-specific judging rubrics. However, failure to satisfy certain criteria does not necessarily indicate a lack of scientific reasoning, as it may also result from limitations in image generation, geometric preservation, or accurate labeling. Without independent experiments distinguishing reasoning failures from generation failures, the conclusion that geoscience reasoning is the primary performance bottleneck remains insufficiently supported.

  2. The evaluation relies on Gemini-3-Flash and Q-Align, whose scores may be influenced by judging prompts, model capabilities, and domain knowledge. Although the authors assess judge alignment with human ratings, discrepancies remain, and each output is evaluated by only one trained geography undergraduate, without overlapping ratings or inter-rater agreement analysis. For specialized tasks involving seismology, structural geology, and atmospheric dynamics, the current validation may be insufficient to establish the reliability and scientific validity of the automated evaluation.

Questions for authors
  1. In Figure 3, do Bad, Fair, and Good represent formal rating levels? What are the criteria for each level, and how do they correspond to the quantitative scores on a 0–100 scale reported in Tables 1 and 2?
  2. How are the ground-truth output images and task-specific judge.md rubrics constructed and validated? What proportion is generated or assisted by AI, and what expert review procedures are applied? Providing a complete judge.md example would improve transparency.
  3. How exactly are (Ri=1R_i=1) and (Ci=1C_i=1) determined in Strict Accuracy? How are task-specific scoring criteria converted into full satisfaction? If discrete ratings are used for Consistency, how are they mapped to the binary pass condition?
  4. How are the evaluation prompts and task-specific judging rubrics for Reasoning and Consistency designed, and how is Quality evaluation configured? Could the authors provide the complete evaluation prompts, parameter settings, and scoring procedures for reproducibility?
  5. How are MAE and STD in Table 3 calculated, and what exactly does STD measure? To what extent do these metrics establish alignment between automated evaluation and human ratings?
Reviewer_bJkaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper introduces GeoR-Bench, which is a benchmark designed to evaluate geoscience visual reasoning through reasoning-informed image editing. The benchmark is composed from 440 questions across 6 geoscience categories and 24 tasks on Earth-observation image and structured scientific representations (maps, geologic diagrams). For each question, the model is required to interpret the image and generate an output reflecting the expected geoscience process, spatial transformation, or domain-specific constraint. The outputs are evaluated against a rubric that assesses the quality of the generated image from a reasoning, consistency, and image quality perspective. Based on the results shown by closed-source and open-source models, the paper concludes that models struggle with scientifically grounded transformations despite often producing outputs that look visually plausible.

Critical strengths
  1. Paper covers a meaningful range of geospatial intelligence domains, including Geomorphology, Crustal science, Cryosphere, GIS & Spatial Geometry, Atmosphere, Hydrology. The benchmark also uses both Earth-observation imagery and scientific representations (maps, diagrams) which is a complementary work from prior general-purpose image editing benchmarks.

  2. The evaluation design, which uses three dimensions (reasoning, consistency, quality) along with strict accuracy, helps distinguish scientific reasoning correctness from the other two factors, which makes the evaluation result interesting. Also, the benchmark is evaluated on a broad range of models to compare performance and limitations across geoscience tasks.

Critical weaknesses
  1. While one of the evaluation metrics is image generation quality, it is unclear to what extent the performance on the task reflects the model's understanding of underlying geoscience principles rather than the general image-generation capability of the model. For example, the model may have an understanding of the geoscience principle but may not be able to create a plausible image due to its lack of image editing or generation capability. On the other hand, the model may not have an understanding of these principles, yet has seen these data during pretraining, therefore, it applies the edit without understanding the geoscience principle.

  2. While the choice of using the three dimensions for evaluation is interesting, the paper does not provide jusitification on why these three dimensions are selected and whether these dimensions along with the strict accuracy is actually sufficient to measure geoscience intelligence of the models.

  3. The paper motivates the benchmark through potential application in fields such as disaster response and climate adaption, but it is unclear how models that are capable of generating high scoring images can support such fields. A clearer justification would strengthen the contribution.

Questions for authors
  1. How are cases where multiple plausible outcomes are possible handled? How do the current evaluation criteria ensure that they penalize results that are scientifically invalid (as there can exist cases in the real world that cannot be sufficiently described via text or theoretical changes)? How are the reference outputs validated by domain experts?

  2. Please provide justification for the criteria for selecting the three dimensions for evaluation and how they qualify for measuring the geospatial intelligence of models. What evidence supports these evaluation metrics (are they used in the domain, etc.)?

  3. How would the mentioned downstream tasks benefit from a model with high performance on the GeoR-Bench? Is there evidence or justification that can support that higher GeoR-Bench performance corresponds to better performance on practical geoscience analysis or decision-support tasks?

  4. Beyond being an application to the geoscience domain, what additional insights do outputs from GeoR-Bench provide regarding the current capabilities and limitations of image-generation models that cannot be obtained from prior benchmarks such as GRADE?

Reviewer_GLPKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces GeoR-Bench, a benchmark for evaluating geoscience visual reasoning through image editing. It covers diverse geoscience tasks using Earth observation imagery and scientific diagrams. Model outputs are evaluated for reasoning, consistency, and image quality. Experiments across various image generation and editing models reveal substantial room for improvement, with reasoning identified as the main bottleneck.

Critical strengths
  • Evaluating whether generative models produce scientifically valid geoscience outputs, rather than merely plausible images, is a timely problem.
  • The benchmark covers both Earth observation imagery and scientific diagrams, and the sources and licenses are documented.
Critical weaknesses
  • The judge rubrics are generated by an LLM, and it is unclear whether they can assess scientific validity. Satisfying the rubric items does not guarantee that an output is physically correct, especially for forecasting tasks with many plausible outcomes.
  • The human evaluation is insufficient. Eight undergraduate students each rated different outputs once, so neither inter-rater agreement nor agreement with domain experts is available.
  • The paper does not describe how much domain scientists were involved in designing the tasks and rubrics.
  • The judge belongs to the same model family as three of the evaluated models.
  • Geoscience reasoning is conflated with image generation ability. Failures in reasoning and failures in rendering are not separated, e.g., by asking the same questions to VLMs in text form. It is unclear whether the benchmark measures more than geoscience-aware image generation.
  • Some tasks can be solved by deterministic computation (e.g., canyon shadows, snowlines from a DEM) or by recalling well-known places named in the prompt. It is questionable whether such tasks require geoscience reasoning.
Questions for authors
  • How were the ground-truth images of AI-generated samples produced, and with which model?
  • What was the role of domain experts in designing and verifying the tasks, ground truths, and rubrics?
  • Does the ranking of the 21 models by the LLM judge agree with a ranking by human experts (e.g., rank correlation)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.