acceptodds
Withdrawn from ICLR 2027

Evaluating the Gap Between VLMs and Specialized Vision Models at Visual Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Despite impressive improvements in general visual question-answering capabilities, Vision-Language Models (VLMs) continue to struggle at complex spatial reasoning tasks (e.g., comparing the relative positions of objects in a scene or identifying the direction of camera motion given two successive frames in a video) compared to specialized computer vision models (e.g., object detection/segmentation models). We perform a rigorous empirical evaluation to study this performance gap, focusing on two kinds of structural representations: (1) involving a single image or multiple images of unrelated scenes (e.g., bounding boxes), and (2) those involving multiple images of the same or similar scenes (e.g., correspondence points). The key challenge is that specialized models cannot directly solve realistic visual reasoning tasks. We propose to incorporate structural representations predicted by specialized models into the VLM's Chains-of-Thought (CoT), compared to incorporating the VLM's own predictions of the same structure. We find that using specialized models substantially outperforms using the VLM's own predictions at downstream reasoning tasks, rigorously establishing the limited spatial understanding of VLMs. Furthermore, we consider using supervised finetuning or reinforcement learning to improve the VLM's visual understanding capabilities; we find that in some cases these strategies can improve performance, but the gap compared to specialized models persists. Our experiments also provide insights into which kinds of structural representations are useful for improving VLM reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, visual reasoning, VLM reasoning, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PQzoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates VLMs' ability to answer visual questions better when given access to specialize vision model outputs (such as depth, object location) during inference or training. They are motivated to find whether VLMs have the ability to replace these systems; they find that VLMs cannot be effectively trained to be as performant as the specialized model, though they can benefit from their output. They evaluate on BLINK for depth, ERQA for physical scenes, SAT and VST for viewpoint changes and SPAR Bench for distances and matching objects in different images.

Critical strengths
  • They provide a broad swathe of results that are more general than prior work. For example, they test how the expert models can effect QA instead of direct accuracy on the intermediate task (compared to Ramachandran 2025), and multiple settings (SFT, RL, ICL, compared to Hu 2024).
Critical weaknesses
  • There is not enough analysis of their results. They present only a single table (labeled Table 4.2, though it is Table 1) of main results and describe aggregate trends, but provide little diagnostic analysis and do not try to understand what other possible limitations exist.
    • They have some analysis (415-440) but it is very broad and non-specific. Why do depth ordering and camera pose provide the highest mean performance increases? Is it the specific formulation? When the instrumental metrics (depth order accuracy, grid match, etc) does this correspond to better downstream accuracy?
    • Much of the paper is spent on defining exact metrics that are defined elsewhere. For example, Section 3.3-3.4 reads like an Appendix, and most of its details should be moved to there because they do not concern their novel methodology. They define specific rewards without motivation or analysis of them (e.g., L186-198, Eq12,14,15,17) I do not think contributes much. It would be more valuable to investigate why, or to what extent, their findings hold.
    • Also, they do not compare their results to prior papers which are set up very similarly (Ramachandran 2025, "How well does GPT-4o understand vision?" or "Visual Sketchpad", HU 2024)
Questions for authors
  • The abstract is an overclaim: "We find that using specialized models substantially outperforms using the VLM’s own predictions " but the gains are mixed on tests across different views (378-405). Did I misinterpret something here?
  • Why does training fail? What are the failure patterns? (L435-440)
  • Did you compare to different/finer grids and representations (depth, locations, etc)? Also how much does each of the composing fields matter? I think these ablations would provide valuable insight into what is helpful and what information the VLMs can use.
  • There are some results that are weird in the table, like the 00.00% for Qwen 3 VL-4B. This suggests that some models were run on sub-optimal fine tuning settings. Did you do hyperparameter sweeps?
  • Can you distinguish more carefully and analytically the difference in your work than prior work? Do your findings disagree with it, or mostly agree, and if they agree what is novel?
  • L349 says 410 questions, but BLINK is 120, ERQA is 150 (L352) and SAT is 150 (L353). Is this a typo or is there another subset?
1 comment

Thank you for the detailed review and constructive feedback. We have decided to withdraw our submission and revise the paper to further strengthen it. We deeply appreciate your time and effort in reviewing our paper.

Reviewer_9m2COpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates the impact of incorporating structured image-level and scene-level visual representations into vision-language models (VLMs) across multiple models and benchmarks. It compares different approaches, including direct prompting with specialist-generated representations and fine-tuning, to assess their effects on visual understanding tasks. The results highlight how different representation types and integration methods influence model performance across tasks.

Critical strengths
  1. Despite the discrepancy between the paper’s framing and its empirical investigation, the results offer valuable insights into the visual representations that VLMs may fail to encode effectively. These findings suggest that improving VLM performance across a range of tasks may require incorporating such general visual features into their internal representations.

  2. The experiments are conducted systematically across a reasonable number of benchmarks and models, providing a broad evaluation of the proposed approaches.

Critical weaknesses
  1. My primary concern is the mismatch between the paper’s general framing and its empirical investigation. The title and introduction position the work as an investigation into whether contemporary VLMs are on par with specialized vision models across various core vision tasks. However, the empirical investigation primarily examines which types of visual representations, and which methods of incorporating them, can improve VLM performance. These are related but distinct research questions, and the current framing does not adequately reflect the scope of the experiments.

  2. I would like to see the performance of the specialized vision models listed in Table 1 on the tasks evaluated in this paper. In particular, I am interested in how their performance compares with that of the base frontier VLMs and their fine-tuned counterparts.

  3. I am not fully convinced of the novelty of this work, given the existing literature on methods for improving VLM performance. A more precise comparison with related work, or a clearer statement of the paper’s contributions relative to prior studies, would help establish its novelty.

  4. The sections describing the rewards associated with different representations, particularly Sections 3.3 and 3.4, are dense and difficult to follow. I recommend providing additional details, either in the main paper or in dedicated appendix sections. Code snippets or a link to the code repository would also help readers better understand and reproduce the methods.

  5. There are a few presentation issues that should be addressed:

    • Lines 36–38: Please provide a citation to support this claim.
    • Line 371: I believe this should refer to Table 1 rather than Table 4.2.
    • The main text would benefit from additional intuition behind the choice of image-level and scene-level representations. It would also be helpful to discuss how the authors can establish that these representations adequately capture the general visual features responsible for the superior performance of specialized vision models, rather than overlooking other potentially important features or representations.
Questions for authors
  1. What motivated the selection of the image-level and scene-level representations investigated in this work? What criteria were used to determine which representations to include?

  2. Why were the frontier VLMs not fine-tuned as part of the evaluation? Including these experiments could help clarify how the proposed approaches compare with fine-tuning when applied to stronger base models.

  3. How might the findings of this work inform the development of VLMs that inherently encode the visual features or representations studied here, rather than relying on external representations at inference time or through additional fine-tuning?

  4. Why do the authors believe that the direct teacher (prompting-based) approach outperforms the fine-tuning-based approaches? What factors might explain this difference in performance?

1 comment

Thank you for the detailed review and constructive feedback. We have decided to withdraw our submission and revise the paper to further strengthen it. We deeply appreciate your time and effort in reviewing our paper.

Reviewer_nZrzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines the gap between large vision-language models (LVLMs) and specialized computer vision models in visual reasoning. The authors study six intermediate visual representations in two categories: per-image (bounding boxes, coarse 3D pose, object depth ordering) and relational scene-level (correspondence grids, camera pose, and semantic correspondence). They also compare several methods for employing these intermediate representations, including direct prompting with teacher-produced representations, supervised fine-tuning, QA-only GRPO, representation-reward distillation, and mixed-reward GRPO. They report that per-image representations provided by the teacher often improve downstream QA performance. Results for scene-level representations are inconsistent.

Critical strengths
  • The paper investigates the important research question of the extent to which VLMs can use different image representations, both during training and at test time, to improve visual reasoning.

  • The paper examines a diverse range of image representations. This breadth allows the authors to generalize their claims rather than focusing on only one representation.

  • The paper examines a wide range of ways to employ these representations, ranging from fine-tuning variants to test-time optimization.

Critical weaknesses
  • The paper has limited methodological novelty. Using auxiliary learning signals and rewards for RL in VLMs is not new and has been studied before. Visual-RFT [1] uses geometric rewards such as IoU, and Grounded Reinforcement Learning for Visual Reasoning [2] investigates visual grounding during RL-based VLM training. Therefore, using auxiliary visual rewards is not, by itself, a novel training strategy. Beyond RL, VISER [3] shows that simple visual cues, such as horizontal lines, combined with a prompt encouraging sequential scanning can improve LVLM visual reasoning without additional training or specialized vision tools. Moreover, the paper lacks comparisons with other methods from the related literature. This makes it difficult to assess whether the proposed method is actually an improvement over existing methods in the literature.

  • The Direct Teacher comparison has confounding factors. Appendix A.2 explicitly includes teacher outputs in the bounding-box, coarse-3D-pose, and depth-ordering prompts, but its camera-pose and semantic-correspondence prompts do not show corresponding teacher outputs. It is unclear whether these conditions actually supply the proposed teacher information. Base prompting and Direct Teacher also differ in task-specific instructions, without an instruction-only control. For bounding boxes, GPT-5.4 selects question-relevant objects before GroundingDINO predicts boxes, making it difficult to isolate geometric information from object selection.

  • Downstream QA does establish the claimed representation gap. Table 1 reports QA results without measuring the quality of the intermediate representations predicted after SFT or GRPO. Poor fine-tuning results could therefore be due to inaccurate representation prediction, difficulty using a correct representation, or degradation of general QA ability. The claim that VLMs cannot internalize specialized visual capabilities needs direct representation-quality evaluation.

  • The fine-tuning recipe is not fully specified and remains ambiguous. The manuscript does not include essential details such as learning rates, batch sizes, training steps, GRPO group sizes, and random seeds. The mixed reward assigns a weight of 0.65 to representation agreement and 0.25 to answer correctness, without an ablation of the weights. Given the large drops in some trained settings (e.g., Qwen3VL-4B with depth-ordering mixed reward: 52.68% Base versus 26.34%), these results do not isolate model limitations from optimization or reward-design choices.

  • There are several inaccurate statements. For example, Direct Teacher improves several per-image results, but not scene-level results consistently: all three scene-level Direct Teacher conditions reduce GPT-6 Astra's score. The statement that mixed-reward semantic correspondence maintains comparable performance also has an important exception: SmolVLM2-2.2B falls from 23.33% to 10.67%. Furthermore, Table 1 reports point estimates without standard errors or confidence intervals, making it difficult to assess the statistical reliability of the observed improvements and degradations.

  • Evaluation counts and scoring are unclear. Section 4.1 states that the per-image test set contains 410 questions, whereas its component counts (120 BLINK, 150 ERQA, 150 SAT) sum to 420. Section 3.1 specifies a GPT-5 semantic judge for MIMIC-IT training, but Section 4.1 does not clearly define the metric or aggregation used to produce Table 1's percentages on the held-out benchmarks. These details are needed for interpretation and reproduction.

[1] Liu et al., "Visual-RFT: Visual Reinforcement Fine-Tuning", ICCV 2025.

[2] Sarch et al., "Grounded Reinforcement Learning for Visual Reasoning", 2025.

[3] Izadi et al., "Visual Structures Help Visual Reasoning: Addressing the Binding Problem in LVLMs", NeurIPS 2025.

Questions for authors

1- For every Direct Teacher condition, what precise data are inserted into the evaluation input? In particular, where are the camera-pose and semantic-correspondence outputs supplied in the prompts shown in Appendix A.2? Can the authors add instruction-only controls and isolate the contribution of GPT-5.4's question-conditioned object selection?

2- Can the authors report held-out intermediate-representation quality for SFT and GRPO, along with complete optimization hyperparameters, reward-weight ablations, and variability across random seeds? Also, what exact scoring metric and sample counts underlie Table 1?

1 comment

Thank you for the detailed review and constructive feedback. We have decided to withdraw our submission and revise the paper to further strengthen it. We deeply appreciate your time and effort in reviewing our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.