acceptodds
Under review as a conference paper at ICLR 2027

VisualNeedle: Benchmarking Active Visual Search in Information-Dense Scenes

Abstract

Frontier multimodal large language models (MLLMs) have been reported to achieve over 90% accuracy on fine-grained perception benchmarks. However, such scores do not necessarily imply faithful use of visual evidence. Prior studies have identified three shortcuts that inflate benchmark performance. First, linguistic priors and lexical cues in questions often enable models to infer plausible answers without seeing the image. Second, coarse global semantics from the visual encoder can bypass fine-grained local details. Third, in some “think-with-images” benchmarks, corrupting the intermediate images returned by visual tools barely affects the final answer. These findings suggest that higher input resolution or larger question pools alone do not elicit genuine active visual search. To address this, we introduce VisualNeedle, a challenging, information-dense, and fine-grained benchmark for scenes where critical evidence is spatially constrained to minute regions and not discernible at a glance. We further propose a counterfactual crop-black setting, which replaces crops returned by tools with black images of the same size, to test whether tool-enabled performance truly relies on intermediate visual evidence. We evaluate 9 prominent MLLMs across four settings: text-only, no-tool, standard tool-enabled, and crop-black. Text-only accuracy stays below 10%, while no-tool accuracy remains below 20%. The best tool-enabled model reaches only 56.00%, still trailing the 63.00% human majority-vote accuracy. These results reveal persistent limitations in fine-grained visual search, while the crop-black ablation confirms that success on VisualNeedle hinges on genuine intermediate visual evidence.

Primary area:
datasets and benchmarks
Keywords:
MLLMs, visual benchmark, think-with-images, active visual search

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nzABOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces VisualNeedle, a 300-question benchmark for active visual search in information-dense images, including street views, documents, shelves, and maps. Its construction targets three shortcuts: answering from language priors, relying only on a global image view, and ignoring evidence returned by tools. Models can crop regions of an image before answering. To examine whether they use the cropped evidence, the authors introduce a crop-black test that replaces the tool's returned images with black patches. The evaluation covers nine MLLMs and human participants, and includes comparisons with V* Bench and HR-Bench.

Critical strengths
  1. The crop-black ablation directly tests whether models use the visual information returned by tools. The comparison with V* Bench and HR-Bench also reveals a limitation in how existing active visual search benchmarks measure the use of intermediate evidence.

  2. The benchmark construction follows clearly defined objectives. Each of the three targeted shortcuts is addressed through a corresponding construction constraint, and the evaluation includes tests of these constraints.

  3. The paper provides detailed experimental documentation, including the human evaluation protocol, an audit of the automatic judge, stability analysis with confidence intervals, and the evaluation prompts. The judge audit also quantifies errors in the automatic evaluation.

Critical weaknesses
  1. The automatic judge uses the same model as the strongest evaluated system. Some answers are evaluated by Gemini 3.1 Pro, and the human audit shows a systematic tendency toward accepting incorrect responses: 61 responses accepted by the judge were marked incorrect by humans, compared with only three cases in the opposite direction. This inflates scores by an estimated 3 to 4.7 percentage points. An independent judge from a different model family would provide a more reliable comparison.

  2. The comparison with V* Bench and HR-Bench does not report how often models call Crop. If models rarely use Crop on these benchmarks, replacing its outputs with black images would have little effect regardless of whether the benchmarks measure visual search effectively. Tool-call statistics are necessary to interpret this comparison.

  3. The localization success metric does not require crops to be sufficiently precise. Any crop that fully contains the target bounding box counts as successful, even if it covers a large portion of the image. This can overestimate the reported 77% localization success rate and complicates the analysis of localization versus recognition errors.

  4. The benchmark contains only 300 questions, resulting in wide confidence intervals for some comparisons and limiting finer-grained analysis. The evaluation also lacks an oracle-crop baseline, which would help separate failures in locating relevant regions from failures in recognizing their content.

Questions for authors
  1. How frequently do models call Crop on V* Bench and HR-Bench in Table 3? The fraction of questions with at least one call would be useful.

  2. What motivated the choice of Gemini 3.1 Pro as the judge, given that it is also evaluated? An independent model family could help check for judge bias.

  3. An oracle-crop baseline would help separate search errors from recognition errors. How well do the models perform when given the ground-truth target region?

  4. Could localization accuracy also be reported under an area constraint, for example with crops smaller than 25% of the original image?

4 comments
AuthorsOpenReview ↗

Response to Reviewer nzAB (1/2)

Thank you for your positive assessment and valuable suggestions. We address each point below.

W1 & Q2. Thank you for raising this concern. Following your suggestion, we evaluated a second judge from a different model family (reported below). In our pipeline, the VLM judge is invoked only when normalized exact string matching fails, and outputs a binary correctness decision given the image, question, and reference answer (Appendices B and N). We originally selected Gemini 3.1 Pro because it was the strongest vision model available to us at the time for handling mixed Chinese–English responses.

  • Human audit (Appendix E). Across 1,800 audited responses, automatic grading agrees with human reviewers on 96.44% of cases (κ = 0.9284). On Gemini 3.1 Pro's own two audited runs, agreement reaches 97.00% and 98.00%—at least as high as on the other four runs—with no clear evidence of systematic self-favoring.
  • Second judge. We re-scored all 27 w/ tools runs from Table 2 plus the two additional audited runs (8,700 responses in total) using Seed 2.1 Pro (ByteDance) [1], a newer and more capable model released after our main evaluation, under the identical judging prompt and rules. On the responses requiring VLM judgment, Seed 2.1 Pro agrees with the original judge on 97.18% of cases (κ = 0.902); in 187 of the 202 disagreements, Seed rejects a response that the original judge accepted. Across the six human-audited runs, Seed's mean absolute accuracy difference from the human audit is 0.44 pp (max 1.33 pp), compared with 3.22 pp for the original judge.
  • Unchanged model ranking (Table R1). Under Seed 2.1 Pro, all 9 models decrease by 0.78–3.44 pp while their relative ranking remains completely unchanged; notably, Doubao-Seed-2.0-Pro—from the same vendor as Seed—shows the largest drop.

Table R1: Paired accuracy change when the Table 2 w/ tools runs are re-scored with Seed 2.1 Pro.

ModelChange under Seed (pp)
Gemini 3.1 Pro−2.44
Gemini 3 Flash−2.33
Doubao-Seed-2.0-Pro−3.44
Doubao-Seed-1.8−2.11
GPT-5.4−1.67
GPT-5.2−1.67
Qwen3.5-Plus−1.56
Kimi K2.5−2.11
Qwen3-VL-235B-A22B-Thinking−0.78

Even under normalized exact matching alone—without any VLM judge—Gemini 3.1 Pro still ranks first by a wide margin (39.56% vs. 27.67% for second-place Gemini 3 Flash).

W2 & Q1. Thank you for this helpful question. Under the crop-black setting, models still invoke the Crop tool on the vast majority of questions, averaging 1.76–5.74 calls per question on V* Bench and 2.19–4.84 on HR-Bench (Table R2).

Table R2: Crop tool usage under the crop-black setting on V* Bench and HR-Bench.

ModelV* ≥1 call (%)V* avg. callsHR-Bench ≥1 call (%)HR-Bench avg. calls
Gemini 3.1 Pro92.12.3171.52.32
Gemini 3 Flash99.03.6686.03.35
Doubao-Seed-2.0-Pro100.04.0599.04.41
Doubao-Seed-1.8100.04.1799.54.38
GPT-5.4100.03.1794.52.99
GPT-5.2100.04.7998.04.46
Qwen3.5-Plus99.55.7499.04.84
Kimi K2.597.42.1795.52.81
Qwen3-VL-235B-A22B-Thinking83.21.7681.52.19

On V* Bench, models call Crop on 83.2–100% of questions yet still achieve 75.92–87.43% accuracy under crop-black (Table 3); on HR-Bench, call rates are 71.5–99.5%, and several strong models also retain high crop-black accuracy (Section 4.3). Thus, high accuracy under crop-black cannot be explained by models simply abstaining from tool use. On VisualNeedle, by contrast, crop-black accuracy remains close to text-only accuracy across all models (Table 3), confirming that tool gains on VisualNeedle depend on the returned visual evidence.

AuthorsOpenReview ↗

Response to Reviewer nzAB (2/2)

W3 & Q4. Thank you for this suggestion; we report the area-constrained localization results in Table R3. Note that the ~77% figure you cite refers to the localization success rates of Gemini 3.1 Pro (77.78%), GPT-5.4 (77.00%), and GPT-5.2 (77.00%) in Table 11.

Table R3: Unconstrained localization success (Table 11), area-constrained localization success on the same 900 trajectories per model, and answer accuracy conditioned on whether a crop covering ≤25% of the image contains the target.

ModelLocalization success (%)Localization, crop ≤ 25% (%)Localization, crop ≤ 10% (%)Acc. when a crop ≤ 25% contains the target (%)Acc. otherwise (%)
Gemini 3.1 Pro77.7877.1171.0065.324.8
Gemini 3 Flash67.2265.6757.1158.427.5
Doubao-Seed-2.0-Pro44.6744.5644.1151.424.6
Doubao-Seed-1.833.7833.7833.3352.021.3
GPT-5.477.0075.2265.0038.38.1
GPT-5.277.0075.4466.2234.66.8
Qwen3.5-Plus43.4441.2235.6739.110.2
Kimi K2.551.7851.3345.6729.98.2
Qwen3-VL-235B-A22B-Thinking20.3319.7817.3327.07.6

Requiring the crop to cover at most 25% of the image lowers localization success by at most 2.22 pp for any model, showing that the reported localization rates are not inflated by overly large crops. Moreover, runs in which a ≤25% crop contains the target are answered correctly 2.1–5.1× as often as the remaining runs, yet their accuracy still ranges from only 27.0% to 65.3%, confirming that post-localization recognition and reasoning remain a distinct bottleneck.

W4 & Q3. Thank you for suggesting the oracle-crop baseline, which we have now conducted: the model is given the full image together with a single crop centered on the ground-truth target box (3× its width and height, minimum 64 px per side, resized identically to standard tool outputs) and answers in one turn without tools (Table R4).

Table R4: Oracle-crop baseline, with Table 2 w/ tools scores shown for reference (accuracy %).

Modelw/ tools (Table 2; reference)Oracle crop
Gemini 3.1 Pro56.0068.00
Doubao-Seed-2.0-Pro36.7860.33
Qwen3.5-Plus22.1158.00

Each oracle score comes from a single 300-question run. For a controlled comparison, we also ran Gemini 3.1 Pro w/ tools under the same model and judge configuration as the oracle baseline; the oracle crop improves accuracy by 12.3 pp (68.00% vs. 55.67%; paired McNemar p = 6.0e-5).

This controlled comparison on Gemini 3.1 Pro shows clear headroom when the target region is supplied directly. At the same time, even with the oracle crop, its accuracy is 68.00% overall and 55.9% on occluded-object questions. Table R3 reveals the same pattern on the paper's multi-turn trajectories: even when a crop covering at most 25% of the image contains the target, accuracy reaches only 27.0–65.3%. Thus, both visual search and post-localization recognition or reasoning contribute substantially to the remaining errors.

Regarding benchmark size, please see point 1 of our General Response.

Thank the authors for the detailed response and the additional experiments. hope to see those results and discussions in the final paper. I have no more questions and I'm keeping my positive score.

Thank you for your positive feedback. We sincerely appreciate your recognition of the value of our benchmark. We will carefully incorporate all the promised revisions into the final version.

Reviewer_JiuPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces VisualNeedle, a 300-question benchmark focused on fine-grained visual evidence in information-dense scenes. It evaluates nine multimodal large language models (MLLMs) under four conditions: text-only, without tools, with tools, and crop-black, where returned crops are replaced with black images. Text-only accuracy remains below 10%, and accuracy without tools remains below 20%. Access to cropping tools improves performance, while replacing the returned crops with black images reduces accuracy, supporting the importance of intermediate visual evidence. The main contribution is the benchmark and diagnostic protocol, which distinguish tool invocation from reliance on the visual information returned by tools. The strongest model achieves 56% accuracy with tools, compared with 63% for the human majority-vote baseline using three annotators per question.

Critical strengths

1- The paper is clearly motivated by an important problem: even if a model gives an accurate final answer, this does not necessarily mean that it successfully searches for and uses local visual evidence to answer correctly. The curated questions and annotated target regions offer a useful basis for investigating this distinction.

2- The experiment between all different MLLM models shows that all models struggle to achieve good accuracy on VisualNeedle. And the additional analysis and specialized-model evaluations provide useful information beyond a leaderboard. Like, the tool-use analyses show that more crop calls do not necessarily lead to higher accuracy.

Critical weaknesses
  1. My main concern is the size of the benchmark. With only 300 questions, it seems too small to support broad conclusions across different scene types and visual abilities. Also, I find it odd that 76 questions were answered incorrectly by all three annotators under the stated protocol. Can the authors explain why this happened? Were these questions particularly difficult, did they require specific language knowledge, or was the wording unclear?

  2. The comparison between evaluation settings does not clearly separate the benefit of adaptive search from the benefit of additional visual information. In the with-tools setting, models receive additional high-resolution observations, can process the image repeatedly, and can select regions adaptively. Consequently, the improvement from 16.33% to 56.00% for Gemini 3.1 Pro establishes the utility of the crop-enabled system, but does not quantify how much of this improvement comes from adaptive search.

  3. The paper’s related-work discussion overlooks relevant work on small-object image retrieval, such as Find your Needle [1]. Although its retrieval setting differs from VisualNeedle’s question-answering setting, both papers address the challenge of capturing small objects in cluttered scenes. Discussing this connection would help clarify VisualNeedle’s contribution.

[1] Green, Michael, et al. "Find your needle: Small object image retrieval via multi-object attention optimization." Advances in Neural Information Processing Systems 38 (2026): 121621-121652.

Questions for authors

Can you please address the weakness section?

7 comments
AuthorsOpenReview ↗

Response to Reviewer JiuP

Thank you for your careful review and helpful suggestions. As requested in your question, we address each weakness below.

W1. Thank you for raising both points. Regarding benchmark size, please see point 1 of our General Response.

Regarding the 76 questions missed by all three annotators, these items reflect the difficulty of the task rather than annotation flaws. The decisive visual evidence is extremely small (99% of target boxes occupy at most 1% of the image area; Appendix C), and annotators worked under a 5-minute time limit per item (Appendix D). Every question underwent multi-round cross-checking to verify unambiguous wording, a unique answer, and clearly visible evidence upon local inspection (Section 3.2). In a follow-up check, 61 of the 76 questions were answered correctly by at least one evaluated model, and manual re-verification of the remaining 15 confirmed that their labels are accurate and the visual evidence is legible. The annotators answered the same English questions given to the models; all are native Chinese speakers with strong English proficiency, so language was not a barrier for them.

W2. Thank you for this insightful comment. We disentangle active search from additional high-resolution input along two lines: (i) controlled ablations that vary only the content returned by the Crop tool, and (ii) single-turn, tool-free baselines that supply a fixed grid of tiles or an oracle crop.

Content controls. Appendix L (Table 16) reports these controls for Gemini 3.1 Pro and GPT-5.4. Their accuracy reaches 56.00% and 31.22% w/ tools, drops to 25.67% and 23.67% under Random Crop (random image regions returned in place of the requested crops), and falls further to 13.33% and 10.33% under Avoid-GT Random Crop (random regions that exclude the target)—below their w/o tools baselines (16.33% and 14.33%). For Gemini 3.1 Pro, 30.33 pp of its 39.67 pp tool gain (76.5%) therefore exceeds what random crops provide, and crops that miss the target yield no gain at all.

Fixed-grid and oracle-crop baselines (Table R1). In the grid baselines, the model receives the full image together with 4 (2×2) or 9 (3×3) uniform tiles; in the oracle baseline, it receives the full image plus one crop centered on the ground-truth target box (3× its width and height). Every tile and oracle crop is cut from the original full-resolution image and resized exactly like a Crop tool output, so these baselines supply the same kind of high-resolution views as tool use, only without adaptive selection. All three baselines are evaluated in a single turn without tools; for each model, they share the same model and judge configuration. The w/ tools column from Table 2 is included for reference.

Table R1: Fixed-grid and oracle-crop baselines, with Table 2 w/ tools scores shown for reference (accuracy %).

Model2×2 grid3×3 gridOracle cropw/ tools (Table 2)
Gemini 3.1 Pro29.6742.3368.0056.00
Doubao-Seed-2.0-Pro29.6736.6760.3336.78
Qwen3.5-Plus15.6718.6758.0022.11

For a direct controlled comparison, we also ran Gemini 3.1 Pro w/ tools under the exact same model and judge configuration as the grid baselines. It achieves 55.67%, outperforming the 2×2 grid by 26.0 pp and the 3×3 grid by 13.3 pp (paired McNemar p = 4.5e-13 and 1.3e-4). Thus, its tool gain cannot be reproduced simply by supplying a fixed set of high-resolution tiles. We discuss the oracle comparison under nzAB W4.

W3. Thank you for pointing us to this relevant work; we will add a discussion of it to Section 2 of the final version. Find your Needle (Green et al., NeurIPS 2025) studies small-object image retrieval: given a query object, it retrieves images containing that object in cluttered scenes from a large corpus, improving single-vector image descriptors via multi-object pre-training and mask-guided attention refinement. VisualNeedle studies the complementary problem at inference time—whether an MLLM can actively locate a small decisive clue within a single dense image and answer a question from it, with crop-black verifying whether the returned crop is actually used. The two works thus address small targets in cluttered scenes at complementary levels: corpus-level retrieval representations versus within-image active visual search.

Dear Reviewer,

As the discussion period is coming to an end, we wanted to kindly follow up on our rebuttal. We hope our response has addressed your concerns, and we would be grateful for any further feedback. If our clarifications have resolved your questions, we would appreciate it if you could consider updating your evaluation accordingly.

Dear Reviewer JiuP,

Thank you for your valuable comments. We understand that you may be busy, as we have not yet received a response during the rebuttal period. If your schedule permits, we would sincerely appreciate the opportunity to discuss our work with you. Thank you again.

Reviewer_cLpROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a new benchmark, VisualNeedle, to evaluate fine-grained perception in cluttered, information-dense scenes. It addresses the shortcut-driven mismeasurement in existing MLLM benchmarks, where high accuracy does not guarantee the actual use of visual evidence. Instead, models often rely on linguistic priors or coarse global semantics, and frequently fail to utilize intermediate visual outputs.

VisualNeedle comprises 300 questions based on high-resolution images, spanning OCR, color recognition, entity recognition, spatial relationships, and occluded object recognition. The dataset is explicitly designed so that linguistic cues and one-shot global views are insufficient and rather, success depends heavily on retrieving local visual evidence. Each question was manually verified by human annotators.

The authors evaluate nine MLLMs across four settings: text-only, without tools, with tools, and a novel "crop-black" diagnostic. Text-only and without-tools accuracies remain below 10% and 20%, respectively. Even with interactive cropping tools, the best model (Gemini 3.1 Pro) achieves only 56.00%, trailing the 63.00% human majority vote. Furthermore, the crop-black setting, which returns black patches instead of real crops and drops accuracy back to text-only levels. This diagnostic, alongside an insightful gain/harm analysis of tool-use behavior, definitively proves that success on VisualNeedle strictly hinges on a model's ability to localize and integrate intermediate visual evidence.

Critical strengths
  1. The benchmark consists of challenging problems that are hard for even humans.
  2. The paper shares valuable insights into the use of tool calls for different models.
  3. The crop-black ablation is a simple and elegant way to evaluate the tool-enable gains.
Critical weaknesses
  1. While the Ethics Statement notes that images were obtained via a "compliant data procurement process," the exact source of the images remain unknown. This lack of transparency makes it difficult to assess potential cultural, linguistic, or geographic biases in the dataset.
  2. The finding that highly capable models like GPT-5.4 experience significant "Harm" and over-explore (averaging 11.07 tool calls) raises questions about prompt/tool-schema alignment. Since all models are forced to use a unified image_zoom_in_tool_reason API, the poor performance of certain models might reflect a mismatch between the benchmark's rigid API schema and the models' native tool-use training, rather than a fundamental flaw in their active visual search capabilities.
  3. At only 300 images, the benchmark is extremely small. This severely limits the statistical power of the category-wise evaluations.
  4. (Minor) The main paper lacks details about human evaluators. It is important to highlight the evaluation settings, such as access to zooming and cropping.
Questions for authors
  1. Could you provide more concrete details regarding the specific sources, geographic distribution, and licensing of the procured images?
  2. How did you verify that this is a reasoning failure rather than a formatting or API-alignment mismatch? Were the evaluated models provided with any few-shot examples of successful Crop(bbox) trajectories in the system prompt to align them to your specific tool schema?
5 comments
AuthorsOpenReview ↗

Response to Reviewer cLpR

Thank you for your thoughtful review and constructive questions. We address each point below.

W1 & Q1. Thank you for raising this point; we agree that transparency regarding data sources is important for assessing potential bias. As noted in the Ethics Statement, all images were obtained through a compliant data procurement process, with the data provider responsible for securing the permissions required for collection, use, and public release. To allow direct inspection, the complete VisualNeedle dataset—including every image, question, reference answer, and target box—is provided in the supplementary material. In the released images, visible scene text is primarily in Chinese.

We find no evidence that this composition introduces a cultural, linguistic, or geographic shortcut. The images depict everyday scenes (street views, documents, shelves, bookcases, and maps), and each question is grounded in a small local visual clue with an annotated target box. Answering correctly therefore requires locating and reading the visual evidence rather than relying on culture-specific priors; consistently, text-only accuracy remains below 10% across all models (Table 3). Moreover, the two Gemini models rank first and second overall, ahead of all models from Chinese developers (Table 2), indicating that the prevalence of Chinese scene text does not confer an advantage on Chinese-developed models.

W2 & Q2. We understand the concern that a unified tool interface might disadvantage certain models, and we appreciate the opportunity to clarify. Although no few-shot examples are provided (all models are evaluated zero-shot, Appendix B; the prompts include no example trajectories, Appendix N), the tool setup follows the providers' official guidance. Every model is given the same Crop tool, image_zoom_in_tool_reason, with required parameters reason, img_idx, and bbox_2d in 0–1000 normalized coordinates (Appendix N). Its schema follows the official Qwen3-VL zoom-in tool in Qwen-Agent (same bbox_2d and img_idx fields), with an added reason field [3]. OpenAI's GPT-5.4 vision guide likewise recommends letting the model crop or zoom into regions before answering [1], and the documentation of these four providers specifies boxes in coordinates normalized to a 1000-unit grid [1–4]; for Gemini, we use its native [y_min, x_min, y_max, x_max] order [2].

Table 11 (Appendix H) shows that GPT-5.4 localizes the target with this tool as often as Gemini 3.1 Pro (77.00% vs. 77.78%). Among successfully localized runs, GPT-5.4 needs 2.20 steps on average to produce its first crop that fully contains the target—comparable to Gemini 3.1 Pro's 1.85 steps—and then performs an average of 9.75 additional crops; as noted in Appendix H, its lower final accuracy cannot be attributed to slow initial localization. Even when a crop covering at most 25% of the image contains the target, GPT-5.4 answers correctly on only 38.3% of trajectories, compared with 65.3% for Gemini 3.1 Pro (Table R3 in our response to nzAB). Overall, tool use still substantially benefits GPT-5.4 (75 gains vs. 20 harms in Section 4.4; p = 1.2e-8).

W3. Thank you; we address benchmark size, including the statistical power of category-wise evaluations, in point 1 of the General Response.

W4. Thank you for the suggestion. The human evaluation protocol is detailed in Appendix D (Table 6): 12 external annotators with no overlap with the data-construction team and no access to ground-truth answers or target boxes; three independent annotators per item; unrestricted zoom and pan in the image viewer; and a time limit of 5 minutes per item (overall mean approximately 3 min 17 s). Section 3.2 of the main text also states that these annotators did not participate in data construction and answered independently.

Dear Reviewer,

As the discussion period is coming to an end, we wanted to kindly follow up on our rebuttal. We hope our response has addressed your concerns, and we would be grateful for any further feedback. If our clarifications have resolved your questions, we would appreciate it if you could consider updating your evaluation accordingly.

Dear Reviewer cLpR,

Thank you for your valuable comments. We understand that you may be busy, as we have not yet received a response during the rebuttal period. If your schedule permits, we would sincerely appreciate the opportunity to discuss our work with you. Thank you again.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all reviewers for their time and constructive feedback. We respond to each review individually, and briefly address the three shared concerns below.

1. Benchmark size (cLpR W3, JiuP W1, nzAB W4). We agree that benchmark size is an important consideration. VisualNeedle's 300 questions are comparable to widely used diagnostic benchmarks: V* Bench has 191 questions, MMVP has 300, and Winoground has 400 examples [1], whose authors likewise note that they "prioritize clean, expert annotations over mere size." For further context, the widely used AIME 2025 benchmark contains only 30 problems [2]. At this scale, exact McNemar tests on the paired w/ and w/o tools runs underlying the Gain/Harm analysis in Section 4.4 (Figure 3) show statistically significant tool gains for 7 of the 9 models at p < 0.01 (Table G1), and the worst-case 95% binomial margin of error for an accuracy measured on 300 questions is ±5.7 pp (run-to-run variation is reported separately in Appendix F). On the same paired runs, the tool gains of both Gemini models also remain significant within each of the five ability categories (34–73 questions each; p < 0.01). We also maintain a non-public 250-question set constructed with the same protocol, on which model rankings are consistent with those on the released 300 questions. Overall, the benchmark size is sufficient to support the paper's main conclusions.

Table G1: Paired significance of tool-use gains, computed from the Gain/Harm counts in Section 4.4 (Figure 3).

ModelΔ (pp)Gain / HarmExact p
Gemini 3.1 Pro+38.0129 / 158.6e-24
Gemini 3 Flash+33.3111 / 115.8e-22
Doubao-Seed-2.0-Pro+23.791 / 205.3e-12
GPT-5.4+18.375 / 201.2e-8
GPT-5.2+15.763 / 169.4e-8
Doubao-Seed-1.8+15.762 / 156.1e-8
Kimi K2.5+6.334 / 159.4e-3
Qwen3.5-Plus+5.341 / 250.064
Qwen3-VL-235B-A22B-Thinking+3.026 / 170.22

2. What the tool gains measure (JiuP W2, nzAB W4). The tool gains on VisualNeedle come from finding and using the target evidence, not from extra views or pixels alone. Appendix L (Table 16) shows that for Gemini 3.1 Pro, 30.33 pp of its 39.67 pp tool gain (76.5%) exceeds a Random Crop control that returns random image regions in place of the requested crops, whereas random crops that avoid the target drop below w/o tools accuracy. We further evaluated fixed-grid and oracle-crop baselines. When Gemini 3.1 Pro w/ tools is evaluated under the same configuration as these baselines, it outperforms the 3×3 grid by 13.3 pp (p = 1.3e-4), so adaptive search adds clear value beyond fixed high-resolution views; the oracle crop outperforms it by a further 12.3 pp (p = 6.0e-5), leaving substantial headroom for better search (see JiuP W2 and nzAB W4).

3. Reliability of the evaluation (nzAB W1, JiuP W1). Following nzAB's suggestion, we re-scored the w/ tools runs using Seed 2.1 Pro, a judge from a different vendor than the original Gemini judge: the ranking of all 9 models remains unchanged, and across the six audited runs in Appendix E, its mean absolute accuracy difference from the human audit is 0.44 pp, compared with 3.22 pp for the original judge (see nzAB W1). In addition, a follow-up check shows that 61 of the 76 questions missed by all three human annotators were answered correctly by at least one evaluated model (see JiuP W1). Thus, our conclusions are robust to the choice of judge, and the questions missed by all annotators reflect task difficulty rather than label errors.

We hope these clarifications are helpful and look forward to the discussion.

Dear Area Chair,

Thank you very much for your time handling our submission. As the discussion deadline approaches, we wanted to kindly ask if you might help remind Reviewers cLpR and JiuP to check our rebuttal when convenient.

Both reviewers noted minor/fixable soundness concerns and raised concrete empirical questions. In our Nov 8 responses, we ran dedicated new experiments and analyses to address each of them directly—specifically:

For Reviewer cLpR: trajectory-level and area-constrained (≤25\le 25%) localization analyses showing that GPT-5.4 localizes targets just as fast and often as Gemini 3.1 Pro under the shared tool schema, alongside checks ruling out language/source shortcuts. For Reviewer JiuP: new 2×22\times 2 / 3×33\times 3 fixed-grid and oracle-crop baselines disentangling adaptive visual search from extra high-resolution views, plus an item-level audit of the 76 questions missed by human annotators. Reviewer nzAB has already reviewed our new experiments (including an independent cross-vendor judge audit), confirmed they have no further questions, and kept their positive rating. We would be very grateful if Reviewers cLpR and JiuP could also let us know whether our responses resolve their concerns, or if any clarification is still needed.

Thank you again for your support!

Best regards, Authors of Submission 29580

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.