- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces VisualNeedle, a 300-question benchmark for active visual search in information-dense images, including street views, documents, shelves, and maps. Its construction targets three shortcuts: answering from language priors, relying only on a global image view, and ignoring evidence returned by tools. Models can crop regions of an image before answering. To examine whether they use the cropped evidence, the authors introduce a crop-black test that replaces the tool's returned images with black patches. The evaluation covers nine MLLMs and human participants, and includes comparisons with V* Bench and HR-Bench.
Critical strengths
-
The crop-black ablation directly tests whether models use the visual information returned by tools. The comparison with V* Bench and HR-Bench also reveals a limitation in how existing active visual search benchmarks measure the use of intermediate evidence.
-
The benchmark construction follows clearly defined objectives. Each of the three targeted shortcuts is addressed through a corresponding construction constraint, and the evaluation includes tests of these constraints.
-
The paper provides detailed experimental documentation, including the human evaluation protocol, an audit of the automatic judge, stability analysis with confidence intervals, and the evaluation prompts. The judge audit also quantifies errors in the automatic evaluation.
Critical weaknesses
-
The automatic judge uses the same model as the strongest evaluated system. Some answers are evaluated by Gemini 3.1 Pro, and the human audit shows a systematic tendency toward accepting incorrect responses: 61 responses accepted by the judge were marked incorrect by humans, compared with only three cases in the opposite direction. This inflates scores by an estimated 3 to 4.7 percentage points. An independent judge from a different model family would provide a more reliable comparison.
-
The comparison with V* Bench and HR-Bench does not report how often models call Crop. If models rarely use Crop on these benchmarks, replacing its outputs with black images would have little effect regardless of whether the benchmarks measure visual search effectively. Tool-call statistics are necessary to interpret this comparison.
-
The localization success metric does not require crops to be sufficiently precise. Any crop that fully contains the target bounding box counts as successful, even if it covers a large portion of the image. This can overestimate the reported 77% localization success rate and complicates the analysis of localization versus recognition errors.
-
The benchmark contains only 300 questions, resulting in wide confidence intervals for some comparisons and limiting finer-grained analysis. The evaluation also lacks an oracle-crop baseline, which would help separate failures in locating relevant regions from failures in recognizing their content.
Questions for authors
-
How frequently do models call Crop on V* Bench and HR-Bench in Table 3? The fraction of questions with at least one call would be useful.
-
What motivated the choice of Gemini 3.1 Pro as the judge, given that it is also evaluated? An independent model family could help check for judge bias.
-
An oracle-crop baseline would help separate search errors from recognition errors. How well do the models perform when given the ground-truth target region?
-
Could localization accuracy also be reported under an area constraint, for example with crops smaller than 25% of the original image?
4 comments
Response to Reviewer nzAB (2/2)
W3 & Q4. Thank you for this suggestion; we report the area-constrained localization results in Table R3. Note that the ~77% figure you cite refers to the localization success rates of Gemini 3.1 Pro (77.78%), GPT-5.4 (77.00%), and GPT-5.2 (77.00%) in Table 11.
Table R3: Unconstrained localization success (Table 11), area-constrained localization success on the same 900 trajectories per model, and answer accuracy conditioned on whether a crop covering ≤25% of the image contains the target.
| Model | Localization success (%) | Localization, crop ≤ 25% (%) | Localization, crop ≤ 10% (%) | Acc. when a crop ≤ 25% contains the target (%) | Acc. otherwise (%) |
|---|---|---|---|---|---|
| Gemini 3.1 Pro | 77.78 | 77.11 | 71.00 | 65.3 | 24.8 |
| Gemini 3 Flash | 67.22 | 65.67 | 57.11 | 58.4 | 27.5 |
| Doubao-Seed-2.0-Pro | 44.67 | 44.56 | 44.11 | 51.4 | 24.6 |
| Doubao-Seed-1.8 | 33.78 | 33.78 | 33.33 | 52.0 | 21.3 |
| GPT-5.4 | 77.00 | 75.22 | 65.00 | 38.3 | 8.1 |
| GPT-5.2 | 77.00 | 75.44 | 66.22 | 34.6 | 6.8 |
| Qwen3.5-Plus | 43.44 | 41.22 | 35.67 | 39.1 | 10.2 |
| Kimi K2.5 | 51.78 | 51.33 | 45.67 | 29.9 | 8.2 |
| Qwen3-VL-235B-A22B-Thinking | 20.33 | 19.78 | 17.33 | 27.0 | 7.6 |
Requiring the crop to cover at most 25% of the image lowers localization success by at most 2.22 pp for any model, showing that the reported localization rates are not inflated by overly large crops. Moreover, runs in which a ≤25% crop contains the target are answered correctly 2.1–5.1× as often as the remaining runs, yet their accuracy still ranges from only 27.0% to 65.3%, confirming that post-localization recognition and reasoning remain a distinct bottleneck.
W4 & Q3. Thank you for suggesting the oracle-crop baseline, which we have now conducted: the model is given the full image together with a single crop centered on the ground-truth target box (3× its width and height, minimum 64 px per side, resized identically to standard tool outputs) and answers in one turn without tools (Table R4).
Table R4: Oracle-crop baseline, with Table 2 w/ tools scores shown for reference (accuracy %).
| Model | w/ tools (Table 2; reference) | Oracle crop |
|---|---|---|
| Gemini 3.1 Pro | 56.00 | 68.00 |
| Doubao-Seed-2.0-Pro | 36.78 | 60.33 |
| Qwen3.5-Plus | 22.11 | 58.00 |
Each oracle score comes from a single 300-question run. For a controlled comparison, we also ran Gemini 3.1 Pro w/ tools under the same model and judge configuration as the oracle baseline; the oracle crop improves accuracy by 12.3 pp (68.00% vs. 55.67%; paired McNemar p = 6.0e-5).
This controlled comparison on Gemini 3.1 Pro shows clear headroom when the target region is supplied directly. At the same time, even with the oracle crop, its accuracy is 68.00% overall and 55.9% on occluded-object questions. Table R3 reveals the same pattern on the paper's multi-turn trajectories: even when a crop covering at most 25% of the image contains the target, accuracy reaches only 27.0–65.3%. Thus, both visual search and post-localization recognition or reasoning contribute substantially to the remaining errors.
Regarding benchmark size, please see point 1 of our General Response.
Thank the authors for the detailed response and the additional experiments. hope to see those results and discussions in the final paper. I have no more questions and I'm keeping my positive score.
Thank you for your positive feedback. We sincerely appreciate your recognition of the value of our benchmark. We will carefully incorporate all the promised revisions into the final version.
Response to Reviewer nzAB (1/2)
Thank you for your positive assessment and valuable suggestions. We address each point below.
W1 & Q2. Thank you for raising this concern. Following your suggestion, we evaluated a second judge from a different model family (reported below). In our pipeline, the VLM judge is invoked only when normalized exact string matching fails, and outputs a binary correctness decision given the image, question, and reference answer (Appendices B and N). We originally selected Gemini 3.1 Pro because it was the strongest vision model available to us at the time for handling mixed Chinese–English responses.
Table R1: Paired accuracy change when the Table 2 w/ tools runs are re-scored with Seed 2.1 Pro.
Even under normalized exact matching alone—without any VLM judge—Gemini 3.1 Pro still ranks first by a wide margin (39.56% vs. 27.67% for second-place Gemini 3 Flash).
W2 & Q1. Thank you for this helpful question. Under the crop-black setting, models still invoke the Crop tool on the vast majority of questions, averaging 1.76–5.74 calls per question on V* Bench and 2.19–4.84 on HR-Bench (Table R2).
Table R2: Crop tool usage under the crop-black setting on V* Bench and HR-Bench.
On V* Bench, models call Crop on 83.2–100% of questions yet still achieve 75.92–87.43% accuracy under crop-black (Table 3); on HR-Bench, call rates are 71.5–99.5%, and several strong models also retain high crop-black accuracy (Section 4.3). Thus, high accuracy under crop-black cannot be explained by models simply abstaining from tool use. On VisualNeedle, by contrast, crop-black accuracy remains close to text-only accuracy across all models (Table 3), confirming that tool gains on VisualNeedle depend on the returned visual evidence.