- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces GeoR-Bench, a benchmark for evaluating the geoscience visual reasoning capabilities of MLLMs through visual editing and generation tasks. The benchmark consists of 440 samples across 6 geoscience categories and 24 task types, covering diverse data sources and geoscience visual representations. The authors systematically evaluate 21 closed- and open-source model configurations along four dimensions: reasoning, consistency, quality, and strict accuracy. They further analyze performance bottlenecks, category-level differences, and the reasoning capabilities of open-source models.
Critical strengths
This paper extends the visual editing and generation evaluation to remote sensing and geoscience, assessing the geoscience visual reasoning capabilities of multimodal models. GeoR-Bench covers diverse task types, data sources, and scientific visual representations, offering broad evaluation coverage. Furthermore, the paper adopts 4 evaluation protocols to systematically assess 21 representative closed- and open-source model configurations, with additional analyses of instruction explicitness and judge alignment to provide further insights into model performance and evaluation reliability.
Critical weaknesses
-
The Reasoning dimension evaluates whether the generated images satisfy the task-specific judging rubrics. However, failure to satisfy certain criteria does not necessarily indicate a lack of scientific reasoning, as it may also result from limitations in image generation, geometric preservation, or accurate labeling. Without independent experiments distinguishing reasoning failures from generation failures, the conclusion that geoscience reasoning is the primary performance bottleneck remains insufficiently supported.
-
The evaluation relies on Gemini-3-Flash and Q-Align, whose scores may be influenced by judging prompts, model capabilities, and domain knowledge. Although the authors assess judge alignment with human ratings, discrepancies remain, and each output is evaluated by only one trained geography undergraduate, without overlapping ratings or inter-rater agreement analysis. For specialized tasks involving seismology, structural geology, and atmospheric dynamics, the current validation may be insufficient to establish the reliability and scientific validity of the automated evaluation.
Questions for authors
- In Figure 3, do Bad, Fair, and Good represent formal rating levels? What are the criteria for each level, and how do they correspond to the quantitative scores on a 0–100 scale reported in Tables 1 and 2?
- How are the ground-truth output images and task-specific judge.md rubrics constructed and validated? What proportion is generated or assisted by AI, and what expert review procedures are applied? Providing a complete judge.md example would improve transparency.
- How exactly are () and () determined in Strict Accuracy? How are task-specific scoring criteria converted into full satisfaction? If discrete ratings are used for Consistency, how are they mapped to the binary pass condition?
- How are the evaluation prompts and task-specific judging rubrics for Reasoning and Consistency designed, and how is Quality evaluation configured? Could the authors provide the complete evaluation prompts, parameter settings, and scoring procedures for reproducibility?
- How are MAE and STD in Table 3 calculated, and what exactly does STD measure? To what extent do these metrics establish alignment between automated evaluation and human ratings?