- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper evaluates VLMs' ability to answer visual questions better when given access to specialize vision model outputs (such as depth, object location) during inference or training. They are motivated to find whether VLMs have the ability to replace these systems; they find that VLMs cannot be effectively trained to be as performant as the specialized model, though they can benefit from their output. They evaluate on BLINK for depth, ERQA for physical scenes, SAT and VST for viewpoint changes and SPAR Bench for distances and matching objects in different images.
Critical strengths
- They provide a broad swathe of results that are more general than prior work. For example, they test how the expert models can effect QA instead of direct accuracy on the intermediate task (compared to Ramachandran 2025), and multiple settings (SFT, RL, ICL, compared to Hu 2024).
Critical weaknesses
- There is not enough analysis of their results. They present only a single table (labeled Table 4.2, though it is Table 1) of main results and describe aggregate trends, but provide little diagnostic analysis and do not try to understand what other possible limitations exist.
- They have some analysis (415-440) but it is very broad and non-specific. Why do depth ordering and camera pose provide the highest mean performance increases? Is it the specific formulation? When the instrumental metrics (depth order accuracy, grid match, etc) does this correspond to better downstream accuracy?
- Much of the paper is spent on defining exact metrics that are defined elsewhere. For example, Section 3.3-3.4 reads like an Appendix, and most of its details should be moved to there because they do not concern their novel methodology. They define specific rewards without motivation or analysis of them (e.g., L186-198, Eq12,14,15,17) I do not think contributes much. It would be more valuable to investigate why, or to what extent, their findings hold.
- Also, they do not compare their results to prior papers which are set up very similarly (Ramachandran 2025, "How well does GPT-4o understand vision?" or "Visual Sketchpad", HU 2024)
Questions for authors
- The abstract is an overclaim: "We find that using specialized models substantially outperforms using the VLM’s own predictions " but the gains are mixed on tests across different views (378-405). Did I misinterpret something here?
- Why does training fail? What are the failure patterns? (L435-440)
- Did you compare to different/finer grids and representations (depth, locations, etc)? Also how much does each of the composing fields matter? I think these ablations would provide valuable insight into what is helpful and what information the VLMs can use.
- There are some results that are weird in the table, like the 00.00% for Qwen 3 VL-4B. This suggests that some models were run on sub-optimal fine tuning settings. Did you do hyperparameter sweeps?
- Can you distinguish more carefully and analytically the difference in your work than prior work? Do your findings disagree with it, or mostly agree, and if they agree what is novel?
- L349 says 410 questions, but BLINK is 120, ERQA is 150 (L352) and SAT is 150 (L353). Is this a typo or is there another subset?
Thank you for the detailed review and constructive feedback. We have decided to withdraw our submission and revise the paper to further strengthen it. We deeply appreciate your time and effort in reviewing our paper.