acceptodds
Under review as a conference paper at ICLR 2027

UltraVR: A Diagnostic Ultra-Resolution Image–VQA Benchmark for Evidence-Grounded Reasoning

Abstract

Vision-language models (VLMs) have advanced rapidly on visual question answering and multimodal reasoning benchmarks. Yet it remains unclear whether they can reason over ultra-resolution images, where answer-critical evidence may be tiny, subtle, spatially distant, or distributed across regions. Existing evaluations largely report final-answer accuracy, offering limited insight into whether models acquire and integrate the necessary visual evidence. We introduce UltraVR, a diagnostic benchmark for evidence-grounded visual reasoning over ultra-resolution images. UltraVR spans four high-value scenarios: Closed-Circuit Television (CCTV) surveillance, remote sensing (RS), whole-slide image (WSI) pathology, and industrial anomaly detection (AD). These domains pose complementary challenges, including fine-grained object grounding in crowded CCTV scenes, long-range spatial comparison in RS, multi-scale evidence navigation in WSI, and subtle irregularity detection in repetitive industrial layouts. Beyond image-question-answer triples, each UltraVR instance includes a structured ground-truth chain of thought with step-level questions, intermediate answers, and reasoning operation labels. These labels decompose reasoning into evidence grounding, local perception, quantification, evidence integration, and decision inference, enabling process-level diagnosis rather than black-box answer scoring. Using UltraVR, we evaluate frontier proprietary and open-weight VLMs and show that current models remain far from reliable on ultra-resolution reasoning. The structured annotations further localize persistent difficulties in grounding and local perception, even with correct preceding context, while downstream inference often recovers when intermediate visual facts are supplied. These findings establish UltraVR as a diagnostic testbed for measuring not only task success, but where models struggle to establish and use visual evidence.

Primary area:
datasets and benchmarks
Keywords:
ultra-resolution image, visual reasoning, evidence-grounded reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_J6rSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce a benchmark termed UltraVR for ultra-resolution visual reasoning across CCTV, remote sensing, whole-slide pathology, and industrial anomaly detection. The benchmark comprises of questions including structured intermediate questions, verified answers and operation labels to help distinguish grounding, perception, quantification, integration, and inference errors. Evaluations are conducted over 13 VLMs including relatively recent models such as Qwen3.5, GPT5.5, Kimi-K2.6, Seed 2.0, with the best direct baseline reaching 44.9% macro accuracy. Further experiments and analyses study local evidence access, tool-assisted inference and different reasoning protocols.

Critical strengths
  1. Ultra-resolution visual reasoning is a relevant and interesting problem to study, with the proposed benchmark providing a useful evaluation setup and provides visual annotations beyond final-answer labels. Specifically, the structured intermediate answers and five operation categories support controlled evaluation of where models fail, which helps distinguish difficulty acquiring visual evidence from difficulty using supplied evidence, within the annotated reasoning paths.

  2. The evaluation considers several factors that can influence performance. Besides multiple model families and domains, the authors study crop-and-zoom tools, different visual-input conditions, reasoning prompts, and predicted versus ground-truth intermediate answers. For example, GPT-5.5 improves from 44.9% with full images to 50.1% with local evidence, while adding both does not help. These comparisons provide useful insights beyond demonstrating ultra-resolution difficulty.

Critical weaknesses
  1. The GT-Prefix experiment does not fully isolate the contribution of correcting visual evidence. S5 supplies all preceding ground-truth answers, potentially including quantification, integration, and inference steps, beyond just visual observations. Thus, the improvement from 32.4% in S4 to 82.1% in S5 can also reflect supplying intermediate solutions. While the operation-level results are useful, a control correcting only grounding/perception answers and leaving subsequent reasoning to the model would better support the proposed diagnosis.

  2. The text-only evaluation does not sufficiently establish the absence of language-prior shortcuts. The reported mean text-only accuracy is 7.4%, but the mean abstention rate is 74.1%. Low accuracy can thus reflect models declining to answer rather than being unable to exploit question or option biases. A forced-choice text-only evaluation, reported separately by domain against the appropriate chance baseline, would provide stronger evidence of visual dependence.

  3. Some baseline results differ across tables without a clear explanation raising doubts on evaluation consistency. For example, MiniCPM-V 4.5 achieves 28.9% macro accuracy in table 1 but 27.3% under the corresponding full-image/direct-QA settings in tables 2–3. The authors should clarify whether these comparisons use different subsets, repeated runs, or aggregation procedures.

Questions for authors

Please see weaknesses section.

6 comments
AuthorsOpenReview ↗

Response Part I

  Thank you for recognizing UltraVR’s value in identifying where models fail and the insights provided by our evaluation across model families, visual-input conditions, reasoning protocols, and tool use. We address your concerns below through additional controls and clarification of the reported results.


1. Isolate the contribution of correcting visual evidence (W1)

  Thank you for suggesting this targeted control. We added GT-GP, which uses verified answers only for the annotated grounding/perception steps when constructing the history for subsequent questions. Answers to quantification, integration, and inference questions remain the model's own predictions, including any errors, and are carried forward without correction. The final answer is never supplied. The model must therefore perform the remaining reasoning using the corrected visual observations, even when visual and reasoning steps are interleaved.

  We compared GT-GP with the S4 and S5 reference scores from Table 3, reporting accuracy across all UltraVR QAs for 12 models. S4 retains model-predicted history, whereas S5 supplies verified answers to all preceding steps.

Table 1. Performance comparison across S4, GT-GP, and S5. Accuracy (%) is computed across all UltraVR QAs.

ModelS4GT-GPS5GT-GP − S4S5 − GT-GP
GPT-5.538.882.189.2+43.3+7.1
GPT-5.428.480.386.0+51.9+5.7
Gemini-3.1-Pro41.586.389.6+44.8+3.3
Seed2.0-Lite38.978.887.7+39.9+8.9
Kimi-K2.635.075.387.8+40.3+12.5
Qwen3.5-27B-FP841.485.386.1+43.9+0.8
InternVL3.5-38B27.266.780.8+39.5+14.1
Gemma-4-26B-A4B-IT27.275.788.1+48.5+12.4
DeepSeek-VL225.054.172.3+29.1+18.2
MiniCPM-V-4.527.566.778.7+39.2+12.0
Phi-4-Reasoning-Vision-15B29.379.287.4+49.9+8.2
Kimi-VL-A3B-Thinking-250636.376.380.2+40.0+3.9
Mean33.075.684.5+42.5+8.9

  Mean final-answer accuracy was 33.0% in S4, 75.6% in GT-GP, and 84.5% in S5. The average gain from S4 to GT-GP was 42.5 percentage points, compared with 8.9 points from GT-GP to S5. The first gain was larger for every evaluated model. Most of the observed recovery was therefore associated with correcting grounding/perception outputs, while verified higher-level reasoning results provided a smaller additional benefit. This pattern strengthens the operation-level evidence that visual fact acquisition is a major bottleneck. We will include this control in the revised manuscript and make explicit which intermediate answers are corrected and which remain model-generated.

AuthorsOpenReview ↗

Response Part II

2. Need forced-choice evaluation without images (W2)

  Thank you for suggesting this control. We evaluated 12 models on the complete UltraVR. Models received no images and were required to select one option or grid coordinate without abstaining. We report accuracy separately by domain and use an equal-domain macro average. The uniform-random references are 25% for CCTV, RS and WSI, and 1% for the 10-by-10 AD grid, giving a 19.0% Macro reference.

Table 2. Forced-choice text-only accuracy on UltraVR.

ModelCCTVRSADWSIMacro
Uniform random choice25.025.01.025.019.0
GPT-5.524.817.51.922.916.8
GPT-5.425.816.01.421.516.2
Gemini-3.1-Pro27.527.50.023.619.7
Seed2.0-Lite26.724.81.924.819.6
Kimi-K2.623.224.30.027.318.7
Qwen3.5-27B-FP828.34.20.032.716.3
InternVL3.5-38B27.515.80.723.216.8
Gemma-4-26B-A4B-IT25.819.20.025.417.6
DeepSeek-VL225.030.81.425.520.7
MiniCPM-V4.525.817.50.020.916.1
Phi-4-Reasoning-Vision-15B20.021.70.023.516.3
Kimi-VL-A3B-Thinking-250623.324.20.022.717.6
Mean25.320.30.624.517.7

  Our question and option design included checks for answer-position bias and recurring semantic answers within each question type. We examined whether the correct option could be inferred from the most common attributes across the choices, or identified because it was longer, more specific, or more fluently written. Distractors were checked for relevance and logical plausibility, including whether particular alternatives could be ruled out without viewing the image or were systematically unlikely to be correct. We also checked question wording for assumptions that revealed a target’s existence, a relationship, or the intended conclusion. The appendix describes our question construction and quality-control procedures. We will make these checks more explicit in the main text of the final version.

  Forced-choice accuracy was close to the random baseline on average, suggesting that question and option text alone offered limited predictive value. We will include this experiment in the final manuscript


3. Aggregation across the main tables (W3)

  Thank you for highlighting the need to clarify aggregation across tables. These aggregation choices serve different reporting purposes. Table 1 reports domain-macro accuracy to compare model capabilities across four equally weighted application domains, preventing domains with more questions from dominating the overall comparison. Tables 2 and 3 focus on performance under different visual-evidence and prompting conditions. Overall question-level accuracy summarizes the fraction of evaluated questions answered correctly under each condition, giving each question equal weight. For the cited MiniCPM baseline, the reported 28.9% and 27.3% are both correct under these respective aggregation rules and are computed from the same baseline predictions. We have cross-checked the aggregation of the corresponding baseline results. In the revised manuscript, we will ensure that diagnostic gaps and table means are computed using the same convention, and make the metric definitions explicit in the evaluation section and table captions.

Dear Reviewer, thank you again for your thoughtful feedback. We would appreciate your thoughts on whether our response addresses your concerns and whether any remaining issues need clarification before the discussion closes.

Reviewer_v8ZnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes UltraVR benchmark, which tests visual question answering on very large images from CCTV, remote sensing, pathology slides, and industrial anomaly detection. Each question includes intermediate questions and verified answers for grounding, local perception, quantification, integration, and final decision.

The paper found in the process of evaluation, giving correct previous step answers raises average final accuracy from 32.9% to 82.1%. Grounding and local perception remain difficult.

Critical strengths

The intermediate questions make the benchmark useful for studying where errors occur, beyond whether the final answer is correct. Comparing predicted previous answers with verified ones is a useful diagnostic. The paper has few interesting observations, The full-image-plus-crop result is also useful. Local evidence raises mean accuracy to 37.2%, but adding the full image reduces it to 32.4%. More visual input does not automatically improve performance.

Critical weaknesses

It is unclear how the reasoning chains are checked to ensure they support the final answers. It is unclear whether the reported perception errors come from missing visual details or VLMs lacking the knowledge to understand them.

Questions for authors

What fraction of AI-generated questions and intermediate answers were corrected or removed during human review? How good is the AI-generated questions and intermediate answers is not very clear.

4 comments
AuthorsOpenReview ↗

Reponse Part I

  Thank you for recognizing the diagnostic value of our intermediate questions and the useful findings from comparing predicted versus verified answers and different visual inputs. Below, we clarify annotation quality and examine the sources of perception errors.


1. Construction, validation, and human corrections (W1 + Q)

  Thank you for asking us to clarify how the reasoning chains are constructed and checked. We start from the final question and identify the visual evidence needed to answer it. Intermediate questions are organized using task templates or a domain-specific workflow. Each retained chain is reviewed to check its intermediate answers against the annotations and image evidence, and to verify that they jointly support the final answer. This construction and validation process is outlined in Section 3.3 and Appendices A and B.3.

  CCTV uses PANDA's person boxes and available attribute annotations. AI combines the annotated attributes into appearance descriptions. Facing directions are proposed by AI and checked in the original image. Counts and geometric relations are computed from the verified evidence. In RS, object boxes and categories come from DOTA annotations. AI-proposed landmark conditions are checked against the image, and object-based answers are derived from the annotations. AD uses fixed question templates based on the normality rules. A model proposes answers about whether each source image satisfies those conditions, and annotators check and correct the answers. The reviewed labels and known grid position determine the intermediate answers. A violated condition returns the anomalous-cell coordinate, and a satisfied condition returns N/A. Grid-size answers come from the recorded layout. WSI follows a clinical workflow. Pathologists annotate ROI selections and morphology answers from the slide and check that these findings support the final interpretation.

  Human review corrected 11 of 80 CCTV facing-direction answers (13.75%) and 17 of 80 RS landmark conditions (21.25%). Answers derived directly from the object annotations required no corrections. In AD, human review corrected 132 of 1,251 AI-proposed condition answers (10.55%). The reviewed labels were used to construct the final AD step answers.

  We expanded these sections to clarify the sources of intermediate answers and the role of AI assistance, and report the human-review correction statistics with their denominators in the revised manuscript.


2. Controlled tests of visual detail and concept support (W2)

  Thank you for raising this distinction. To examine whether the perception difficulties in UltraVR reflected missing visual detail or insufficient task-relevant knowledge, we conducted two complementary experiments based on the visual sub-tasks in our verified annotations. The first varied image detail and knowledge support while models answered visual questions. The second independently tested their understanding of the concepts and rules involved in those questions.

  For the first experiment, we selected 20 verified sub-questions from distinct source images or slides in each domain, without replacement and with domain-specific stratification. We evaluated the same 12 models in four paired conditions. The higher-detail input retained the complete image within each model's configured image budget. To construct the lower-detail input, we downsampled this image by a factor of four along each axis and then restored the same input dimensions. This reduced spatial detail without changing the field of view or candidate search area. Knowledge support consisted of a recognition reference, compared against format- and approximately length-matched control text. CCTV references explained facing-direction labels and the associated head and body cues. RS references described aerial-object appearances and the spatial and landmark relations queried in the original sub-questions. AD references clarified component identity and integrity, while the original normal conditions remained explicit in every condition. WSI references defined all candidate morphological features rather than identifying the findings present on the current slide. None supplied the current answer, target location or case-specific visual findings. We averaged accuracy equally across the four domains.

AuthorsOpenReview ↗

Response Part II

Table 1. Visual sub-question accuracy under paired resolution and knowledge-support conditions. Detail effect = [(Higher - Lower) + (Higher + reference - Lower + reference)] / 2. Reference effect = [(Lower + reference - Lower) + (Higher + reference - Higher)] / 2.

ModelLower detailHigher detailLower + referenceHigher + referenceDetail effectReference effect
GPT-5.512.5017.2513.0017.75+4.75+0.50
GPT-5.411.2516.2512.0012.50+2.75-1.50
Gemini-3.1-Pro16.2518.7517.5018.25+1.63+0.38
Seed2.0-Lite13.7515.7513.7515.00+1.63-0.38
Kimi-K2.611.2513.7511.2513.00+2.13-0.38
Qwen3.5-27B-FP817.5021.2518.7520.00+2.50+0.00
InternVL3.5-38B18.7521.2519.0021.75+2.63+0.38
Gemma-4-26B-A4B-IT8.7513.7510.0012.50+3.75+0.00
DeepSeek-VL216.2518.7515.7518.00+2.38-0.63
MiniCPM-V-4.58.7513.709.2514.25+4.98+0.53
Phi-4-Reasoning-Vision-15B8.7511.258.7511.75+2.75+0.25
Kimi-VL-A3B-Thinking-250615.0016.2515.0016.00+1.13-0.13
Mean13.2316.5013.6715.90+2.75-0.08

  For the second experiment, we constructed 160 supplementary four-option questions, with 20 concept-identification questions and 20 hypothetical-application questions per domain. Correct-answer positions were balanced within each domain and question type. Models received no images, case-specific findings or additional teaching references. CCTV questions tested facing, attribution, and image-coordinate conventions, including the distinction between facing direction, clothes color, position and movement. RS questions tested aerial-object categories and the spatial rules used to identify targets. AD questions tested component identity and integrity under the original explicitly stated normal conditions. WSI questions tested the 19 morphological concepts used in the expert-annotated sub-tasks through terminology matching and hypothetical descriptions. We computed domain accuracy over all 40 questions and macro accuracy as the equal-weight domain mean.

Table 2. Independent concept and rule comprehension. Macro is the arithmetic mean of the four domain accuracy.

ModelCCTVRSADWSIMacro
GPT-5.598.5099.00100.00100.0099.38
GPT-5.496.0098.00100.00100.0098.50
Gemini-3.1-Pro95.00100.00100.00100.0098.75
Seed2.0-Lite97.5097.50100.00100.0098.75
Kimi-K2.6100.0098.50100.00100.0099.63
Qwen3.5-27B-FP8100.0097.50100.00100.0099.38
InternVL3.5-38B95.0093.0092.50100.0095.13
Gemma-4-26B-A4B-IT95.0092.5096.50100.0096.00
DeepSeek-VL295.0093.5092.0093.5093.50
MiniCPM-V-4.597.5093.0097.50100.0097.00
Phi-4-Reasoning-Vision-15B95.0093.5095.0097.5095.25
Kimi-VL-A3B-Thinking-250692.5091.0094.5097.5093.88
Mean96.4295.5897.3399.0497.09

  The pattern presented in Tables 1 and 2 is consistent with a distinction between understanding task-relevant concepts and recovering their visual evidence. Table 1 shows positive detail effects across all twelve models, averaging +2.75 percentage points, whereas additional recognition references yield a mean effect of −0.08 points. Table 2 reports 97.09% mean macro accuracy on concept and rule comprehension. Taken together, these results suggest that limited access to fine-grained visual evidence contributes more to the visual perception errors than gaps in understanding background knowledge.

Thank you for the detailed rebuttal. My concerns have been addressed, and I will maintain my rating of 3 (Weak acceptance).

Thanks for acknowledging that we addressed your concerns and for maintaining your positive rating. We truly appreciate your feedback!

Reviewer_nyGqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

To address the limitations of existing ultra-resolution visual question answering benchmarks, which primarily focus on final-answer accuracy and lack fine-grained diagnosis of visual reasoning failures, this paper introduces UltraVR, a diagnostic benchmark spanning four domains and featuring structured ground-truth chains of thought (GT-CoT) with five reasoning operation labels. Experimental results show that current vision-language models still struggle with ultra-resolution visual reasoning, achieving an average accuracy of only 33.6% across 13 models. Providing ground-truth intermediate reasoning results improves average accuracy from 32.4% to 82.1%, revealing a substantial gap between visual evidence acquisition and downstream reasoning capabilities.

Critical strengths
  1. Important research problem with clear motivation. The paper addresses an important and well-motivated research problem by distinguishing conventional VQA evaluation based on final-answer accuracy from the process-level reliability of reasoning over ultra-resolution images. This distinction is particularly meaningful for applications that require identifying and integrating fine-grained visual evidence.

  2. Structured GT-CoT annotations enabling fine-grained diagnostics. The paper provides structured intermediate questions, reference answers, and operation labels. This lays the foundation for fine-grained diagnosis and may also support future model training, evaluation, and error analysis research.

Critical weaknesses
  1. Confounding factors in the central diagnostic conclusion. The paper emphasizes the substantial accuracy improvement from S4 (Pred-Step, 32.4%) to S5 (GT-Prefix, 82.1%) and interprets this result as evidence that reliable visual fact acquisition is a major bottleneck. However, this comparison does not fully isolate visual evidence acquisition from downstream reasoning. Ground-truth prefixes may contain not only correct visual observations but also completed counting or relational inferences and semantic information that is highly predictive of subsequent answers. Consequently, S5 may substantially reduce the complexity of the remaining reasoning task, rather than merely prevent the propagation of visual errors. This confounding factor weakens the causal interpretation of one of the paper's central diagnostic findings.
  2. Different visual evidence conditions are not strictly comparable. Table 2 reports higher mean accuracy under V3 (Local Evidence, 37.2%) than under V2 (Full Image, 32.9%), which the paper interprets as evidence of limitations in visual evidence access. However, the construction of V3 differs substantially across domains: CCTV and RS use quadrant-level crops, WSI uses cropped regions of interest, and AD uses a 5×5 cell sheet cropped from the original 10×10 grid, with the anomalous cell randomly placed within the crop. These transformations may change not only the accessibility of fine-grained visual evidence but also the search space, number of candidate targets, and task difficulty. Therefore, the observed improvement under V3 may partly result from reduced visual search complexity rather than improved access to local evidence alone. Additional controlled comparisons would help disentangle these effects and strengthen the paper's conclusions.
Questions for authors
  1. Can the authors provide controlled interventions that separately supply ground-truth visual observations and higher-level reasoning results? This would help establish whether the S4–S5 gap primarily reflects visual evidence acquisition failures.
  2. How many alternative reasoning paths are evaluated per question in Appendix D.8?
  3. What are the effective numbers of independent source images, and how stable are model rankings under image-level resampling?
6 comments
AuthorsOpenReview ↗

Response Part I

  Thank you for recognizing the importance of the problem and the value of our structured GT-CoT annotations for diagnosing visual reasoning failures beyond final-answer accuracy. We address your related weaknesses and questions jointly below.


1. Separating observation correction from supplied reasoning results (W1 + Q1)

  Thank you for highlighting this distinction. Our original diagnosis drew on both the operation-level analyses in Figure 2 and the protocol comparison in Table 3, rather than treating the S4–S5 accuracy gap alone as a measure of visual acquisition errors. To address the concern that S5 may primarily benefit from supplied higher-level solutions, we added GT-GP, which replaces only the annotated grounding/perception outputs with their verified values while leaving subsequent reasoning to the model.

  GT-GP follows the original decomposition. Previously completed GND/PER answers are replaced by their verified values when constructing subsequent inputs. Answers to QUA/INT/INF questions remain the model's own predictions, including any errors, and are carried forward without correction. The final answer is never supplied. The model must therefore use the verified grounding/perception outputs to perform the remaining reasoning and select its final answer, even when visual and reasoning questions are interleaved.

Table1. Performance comparison across S4, GT-GP, and S5. Accuracy (%) is computed across all UltraVR QAs.

ModelS4GT-GPS5GT-GP − S4S5 − GT-GP
GPT-5.538.882.189.2+43.3+7.1
GPT-5.428.480.386.0+51.9+5.7
Gemini-3.1-Pro41.586.389.6+44.8+3.3
Seed2.0-Lite38.978.887.7+39.9+8.9
Kimi-K2.635.075.387.8+40.3+12.5
Qwen3.5-27B-FP841.485.386.1+43.9+0.8
InternVL3.5-38B27.266.780.8+39.5+14.1
Gemma-4-26B-A4B-IT27.275.788.1+48.5+12.4
DeepSeek-VL225.054.172.3+29.1+18.2
MiniCPM-V-4.527.566.778.7+39.2+12.0
Phi-4-Reasoning-Vision-15B29.379.287.4+49.9+8.2
Kimi-VL-A3B-Thinking-250636.376.380.2+40.0+3.9
Mean33.075.684.5+42.5+8.9

  Across all 12 models, mean accuracy increased from 33.0% (S4) to 75.6% (GT-GP), with a smaller further increase to 84.5% (S5), an additional 8.9 percentage points. Most of the observed performance recovery was therefore associated with correcting grounding/perception outputs, while further correcting higher-level reasoning provided a comparatively smaller benefit. This comparison strengthens the evidence for visual fact acquisition as a major bottleneck, beyond what the combined S4–S5 gap alone can establish. We will incorporate this comparison and clarify the respective contributions of these two stages of correction in the revised manuscript.

AuthorsOpenReview ↗

Response Part II

2. Controlled comparison of visual evidence access (W2)

  Thank you for highlighting the distinction between preserving visual detail and reducing the search space. V3 was designed to make answer-critical local evidence more accessible and examine whether models could use it more effectively than under full-image inference. Section 5.2 discussed its gains in terms of limitations in evidence access, compression, or retrieval. However, We agree V3 changes the visual search space differently across domains. To control for this factor, we introduced two full-coverage conditions across all four UltraVR domains, holding the search space fixed while varying the available visual detail.

  In both conditions, each source image was divided into four non-overlapping tiles covering the entire image. CCTV and RS retained all four quadrants. WSI retained the complete benchmark slide image without selecting annotated ROIs. AD retained all four 5×5 blocks, preserving all 100 cells and their original positions. The added prompt instructions specified the tile arrangement without identifying target locations.V5-H (fine-detail) encoded each tile separately using the same per-image resolution setting as V2. V5-L (coarse-detail) used the same tiles, downsampled by 2 times along each dimension and then resized back to the V5-H dimensions. Both conditions therefore had the same search space, tile arrangement, and input dimensions, but differed in the visual detail preserved. Questions, options, image-processing settings, reasoning configuration, and decoding parameters were identical between V5-H and V5-L.

Table 2. Full-coverage visual-evidence comparison. Accuracy is reported in percentages and differences in percentage points.

ModelV2 Full imageV5-L Coarse-detailV5-H Fine-detailV5-H − V2V5-H − V5-L
GPT-5.544.9046.3348.16+3.27+1.84
GPT-5.432.8634.2937.76+4.90+3.47
Gemini-3.1-Pro41.4342.2443.67+2.24+1.43
Seed2.0-Lite39.8043.2745.92+6.12+2.65
Kimi-K2.638.5737.5538.98+0.41+1.43
Qwen3.5-27B-FP833.8835.9240.82+6.94+4.90
InternVL3.5-38B28.3729.5930.61+2.24+1.02
Gemma-4-26B-A4B-IT30.2030.4133.27+3.06+2.86
DeepSeek-VL226.3324.0826.73+0.41+2.65
MiniCPM-V-4.527.3526.3330.20+2.86+3.88
Phi-4-Reasoning-Vision-15B29.5927.9631.84+2.24+3.88
Kimi-VL-A3B-Thinking-250627.5525.1028.57+1.02+3.47
Mean33.4033.5936.38+2.98+2.79

  V5-H improved over V2 by 2.98 percentage points on average when the complete image was presented as four separately encoded tiles rather than a single input. With the search space and tile arrangement held fixed, V5-H also outperformed V5-L across all twelve models, with an average gain of 2.79 points. These controlled experiments shows that preserving finer visual detail can improve performance without reducing the search space, providing additional evidence for the role of visual-evidence access. We will incorporate these results into Table 2 and revise Section 5.2 to clarify what each comparison tests and distinguish the effects of visual detail from changes in search space.

AuthorsOpenReview ↗

Response Part III

3. Alternative paths per question in Appendix D.8 (Q3)

  Each question in Appendix D.8 was evaluated with one alternative reasoning path in addition to the original path. This one-to-one design provides a balanced paired comparison, allowing us to assess sensitivity to the choice of reasoning path on the same questions, with one decomposition per question in each condition.


4. Effective numbers of independent source images and ranking stability (Q4)

  UltraVR contains 248 source units: 40 CCTV frames, 40 RS images, 28 WSI slides, and 140 AD grids, supporting 490 questions, as reported in Table D.3.

  Appendix D.7 reports 10,000 domain-stratified, image-clustered bootstrap replicates. All questions associated with a sampled source are kept together, and the same source draws are used for every model. The ranking analysis uses equal-domain macro accuracy on a common cohort of 490 questions with 248 source units.

Table 3. Model-ranking stability under image-level bootstrap resampling.

ModelMean bootstrap rank95% rank interval
GPT-5.51.17[1, 2]
GPT-5.46.12[5, 9]
Gemini-3.1-Pro2.20[1, 4]
Seed2.03.11[2, 4]
Kimi-K2.63.57[2, 5]
Qwen3.5-27B5.45[4, 8]
InternVL3.5-38B9.08[6, 11]
DeepSeek-VL29.95[7, 11]
MiniCPM-V-4.58.97[6, 11]
Phi-4-Reasoning-Vision-15B7.48[5, 10]
Kimi-VL-A3B-Thinking8.91[6, 11]

  We also conducted 1,000 subsampling trials per fraction, retaining 50% or 75% of the source units within each domain without replacement. Each subset ranking was compared with the ranking on the full common cohort using Kendall's tau and top-three overlap.

Table 4. Ranking consistency under repeated source-level subsampling.

Retained source unitsMean Kendall's tauMean top-three overlap
50%0.81688.0%
75%0.89292.4%

  GPT-5.5 maintained a mean bootstrap rank of 1.17, with a 95% rank interval of [1, 2]. Even after removing half the source units, the resulting rankings remained strongly correlated with the reference ranking, achieving a mean Kendall’s tau of 0.816 and a mean top-three overlap of 88.0%. Retaining 75% of the sources increased these values to 0.892 and 92.4%, respectively. These results show that the leading-model group and broad ranking are consistently recovered across different source-image compositions. We will highlight these existing stability results and clarify the source-level sampling protocol in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.