- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors introduce a benchmark termed UltraVR for ultra-resolution visual reasoning across CCTV, remote sensing, whole-slide pathology, and industrial anomaly detection. The benchmark comprises of questions including structured intermediate questions, verified answers and operation labels to help distinguish grounding, perception, quantification, integration, and inference errors. Evaluations are conducted over 13 VLMs including relatively recent models such as Qwen3.5, GPT5.5, Kimi-K2.6, Seed 2.0, with the best direct baseline reaching 44.9% macro accuracy. Further experiments and analyses study local evidence access, tool-assisted inference and different reasoning protocols.
Critical strengths
-
Ultra-resolution visual reasoning is a relevant and interesting problem to study, with the proposed benchmark providing a useful evaluation setup and provides visual annotations beyond final-answer labels. Specifically, the structured intermediate answers and five operation categories support controlled evaluation of where models fail, which helps distinguish difficulty acquiring visual evidence from difficulty using supplied evidence, within the annotated reasoning paths.
-
The evaluation considers several factors that can influence performance. Besides multiple model families and domains, the authors study crop-and-zoom tools, different visual-input conditions, reasoning prompts, and predicted versus ground-truth intermediate answers. For example, GPT-5.5 improves from 44.9% with full images to 50.1% with local evidence, while adding both does not help. These comparisons provide useful insights beyond demonstrating ultra-resolution difficulty.
Critical weaknesses
-
The GT-Prefix experiment does not fully isolate the contribution of correcting visual evidence. S5 supplies all preceding ground-truth answers, potentially including quantification, integration, and inference steps, beyond just visual observations. Thus, the improvement from 32.4% in S4 to 82.1% in S5 can also reflect supplying intermediate solutions. While the operation-level results are useful, a control correcting only grounding/perception answers and leaving subsequent reasoning to the model would better support the proposed diagnosis.
-
The text-only evaluation does not sufficiently establish the absence of language-prior shortcuts. The reported mean text-only accuracy is 7.4%, but the mean abstention rate is 74.1%. Low accuracy can thus reflect models declining to answer rather than being unable to exploit question or option biases. A forced-choice text-only evaluation, reported separately by domain against the appropriate chance baseline, would provide stronger evidence of visual dependence.
-
Some baseline results differ across tables without a clear explanation raising doubts on evaluation consistency. For example, MiniCPM-V 4.5 achieves 28.9% macro accuracy in table 1 but 27.3% under the corresponding full-image/direct-QA settings in tables 2–3. The authors should clarify whether these comparisons use different subsets, repeated runs, or aggregation procedures.
Questions for authors
Please see weaknesses section.
6 comments
Response Part II
2. Need forced-choice evaluation without images (W2)
Thank you for suggesting this control. We evaluated 12 models on the complete UltraVR. Models received no images and were required to select one option or grid coordinate without abstaining. We report accuracy separately by domain and use an equal-domain macro average. The uniform-random references are 25% for CCTV, RS and WSI, and 1% for the 10-by-10 AD grid, giving a 19.0% Macro reference.
Table 2. Forced-choice text-only accuracy on UltraVR.
| Model | CCTV | RS | AD | WSI | Macro |
|---|---|---|---|---|---|
| Uniform random choice | 25.0 | 25.0 | 1.0 | 25.0 | 19.0 |
| GPT-5.5 | 24.8 | 17.5 | 1.9 | 22.9 | 16.8 |
| GPT-5.4 | 25.8 | 16.0 | 1.4 | 21.5 | 16.2 |
| Gemini-3.1-Pro | 27.5 | 27.5 | 0.0 | 23.6 | 19.7 |
| Seed2.0-Lite | 26.7 | 24.8 | 1.9 | 24.8 | 19.6 |
| Kimi-K2.6 | 23.2 | 24.3 | 0.0 | 27.3 | 18.7 |
| Qwen3.5-27B-FP8 | 28.3 | 4.2 | 0.0 | 32.7 | 16.3 |
| InternVL3.5-38B | 27.5 | 15.8 | 0.7 | 23.2 | 16.8 |
| Gemma-4-26B-A4B-IT | 25.8 | 19.2 | 0.0 | 25.4 | 17.6 |
| DeepSeek-VL2 | 25.0 | 30.8 | 1.4 | 25.5 | 20.7 |
| MiniCPM-V4.5 | 25.8 | 17.5 | 0.0 | 20.9 | 16.1 |
| Phi-4-Reasoning-Vision-15B | 20.0 | 21.7 | 0.0 | 23.5 | 16.3 |
| Kimi-VL-A3B-Thinking-2506 | 23.3 | 24.2 | 0.0 | 22.7 | 17.6 |
| Mean | 25.3 | 20.3 | 0.6 | 24.5 | 17.7 |
Our question and option design included checks for answer-position bias and recurring semantic answers within each question type. We examined whether the correct option could be inferred from the most common attributes across the choices, or identified because it was longer, more specific, or more fluently written. Distractors were checked for relevance and logical plausibility, including whether particular alternatives could be ruled out without viewing the image or were systematically unlikely to be correct. We also checked question wording for assumptions that revealed a target’s existence, a relationship, or the intended conclusion. The appendix describes our question construction and quality-control procedures. We will make these checks more explicit in the main text of the final version.
Forced-choice accuracy was close to the random baseline on average, suggesting that question and option text alone offered limited predictive value. We will include this experiment in the final manuscript
3. Aggregation across the main tables (W3)
Thank you for highlighting the need to clarify aggregation across tables. These aggregation choices serve different reporting purposes. Table 1 reports domain-macro accuracy to compare model capabilities across four equally weighted application domains, preventing domains with more questions from dominating the overall comparison. Tables 2 and 3 focus on performance under different visual-evidence and prompting conditions. Overall question-level accuracy summarizes the fraction of evaluated questions answered correctly under each condition, giving each question equal weight. For the cited MiniCPM baseline, the reported 28.9% and 27.3% are both correct under these respective aggregation rules and are computed from the same baseline predictions. We have cross-checked the aggregation of the corresponding baseline results. In the revised manuscript, we will ensure that diagnostic gaps and table means are computed using the same convention, and make the metric definitions explicit in the evaluation section and table captions.
Dear Reviewer, thank you again for your thoughtful feedback. We would appreciate your thoughts on whether our response addresses your concerns and whether any remaining issues need clarification before the discussion closes.
Response Part I
Thank you for recognizing UltraVR’s value in identifying where models fail and the insights provided by our evaluation across model families, visual-input conditions, reasoning protocols, and tool use. We address your concerns below through additional controls and clarification of the reported results.
1. Isolate the contribution of correcting visual evidence (W1)
Thank you for suggesting this targeted control. We added GT-GP, which uses verified answers only for the annotated grounding/perception steps when constructing the history for subsequent questions. Answers to quantification, integration, and inference questions remain the model's own predictions, including any errors, and are carried forward without correction. The final answer is never supplied. The model must therefore perform the remaining reasoning using the corrected visual observations, even when visual and reasoning steps are interleaved.
We compared GT-GP with the S4 and S5 reference scores from Table 3, reporting accuracy across all UltraVR QAs for 12 models. S4 retains model-predicted history, whereas S5 supplies verified answers to all preceding steps.
Table 1. Performance comparison across S4, GT-GP, and S5. Accuracy (%) is computed across all UltraVR QAs.
Mean final-answer accuracy was 33.0% in S4, 75.6% in GT-GP, and 84.5% in S5. The average gain from S4 to GT-GP was 42.5 percentage points, compared with 8.9 points from GT-GP to S5. The first gain was larger for every evaluated model. Most of the observed recovery was therefore associated with correcting grounding/perception outputs, while verified higher-level reasoning results provided a smaller additional benefit. This pattern strengthens the operation-level evidence that visual fact acquisition is a major bottleneck. We will include this control in the revised manuscript and make explicit which intermediate answers are corrected and which remain model-generated.